Baseline profile macrobenchmarks failing in CI [192084204]

Fixed

Bug

Status Update

No update yet.

Description

cc...@google.com

created issue #1

Jun 25, 2021 06:04PM

See https://blackbox.googleplex.com/dashboard/androidx/target/aosp-androidx-main/androidx_device_tests-sailfish-userdebug-29/6/MetricTests/2

Baseline profile benchmarks in androidx.compose.integration.macrobenchmark package are failing (on Sailfish API 29), with an exception like:

java.lang.RuntimeException: The baseline profile install broadcast was not received. This most likely means that the profileinstaller library is not in the target APK. It must be in order to use CompilationMode.BaselineProfile.
java.lang.RuntimeException: The baseline profile install broadcast was not received. This most likely means that the profileinstaller library is not in the target APK. It must be in order to use CompilationMode.BaselineProfile.
	at androidx.benchmark.macro.CompilationModeKt.compile(CompilationMode.kt:121)
	at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:129)
	at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:241)
	at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:68)
	at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated$default(MacrobenchmarkRule.kt:58)
	at androidx.testutils.MacrobenchUtilsKt.measureStartup(MacrobenchUtils.kt:34)
	at androidx.testutils.MacrobenchUtilsKt.measureStartup$default(MacrobenchUtils.kt:28)
	at androidx.compose.integration.macrobenchmark.IoSettingsStartupBenchmark.startup(IoSettingsStartupBenchmark.kt:40)

Sample failure: https://android-build.googleplex.com/builds/tests/view?invocationId=I66500009645614737&redirect=http%3A%2F%2Fab%2FI66500009645614737&testResultId=TR79926289235033187

Comments

cc...@google.com <cc...@google.com> #2Jun 29, 2021 11:59PM

Leland/Rahul can you take a look at this? We'd like to release another benchmark alpha, but I'll need to hide the Baseline profiles API if it's not working in benchmarks.

(Additionally, I'd love to try out adding some profiles to various AndroidX libraries :D)

le...@google.com <le...@google.com> Jun 30, 2021 07:45PM

Reassigned to ra...@google.com.

ra...@google.com <ra...@google.com> #3Jun 30, 2021 08:23PM

Accepted by ra...@google.com.

This is happening because we call MacrobenchmarkScope.killProcess() which triggers a force-stop on the package. This causes the broadcasts to not get deliverabled, until a user interacts with the target app.

ap...@google.com <ap...@google.com> #4Jul 2, 2021 10:11PM

Marked as fixed.

Project: platform/frameworks/support
Branch: androidx-main

commit a5e7913de40ac67088647fa80ed20bb027857fc7
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Fri Jul 02 10:13:23 2021

Uses explicit broadcasts to deliver intents to receivers.

* This is because the app is going to be force-stopped when running the
Macrobenchmark.
* Also add Jetifier migration configuration for Profile installer.

Fixes:

b/192084204

Test: Compose macrobenchmarks pass successfully.
Relnote: Use an `EmptyActivity` to bring the target app out of a force-stopped state to better support `CompilationMode.BaselineProfile`.

06/30 13:48:51: Launching 'IoSettingsStartupBen...' on Google Pixel 5.
Install successfully finished in 2 s 119 ms.
Running tests

$ adb shell am instrument -w -m -e androidx.benchmark.output.enable true --no-window-animation -e debug false -e class 'androidx.compose.integration.macrobenchmark.IoSettingsStartupBenchmark' androidx.compose.integration.macrobenchmark.test/androidx.test.runner.AndroidJUnitRunner
Connected to process 32016 on device 'google-pixel_5-localhost:33237'.
IoSettingsStartupBenchmark_startup[startup=HOT,compilation=None]
startupMs min 60, median 63, max 71
Traces: Iteration 0 1 2

IoSettingsStartupBenchmark_startup[startup=HOT,compilation=SpeedProfile(iterations=3)]
startupMs min 42, median 60, max 62
Traces: Iteration 0 1 2

IoSettingsStartupBenchmark_startup[startup=HOT,compilation=BaselineProfile]
startupMs min 51, median 56, max 66
Traces: Iteration 0 1 2

IoSettingsStartupBenchmark_startup[startup=WARM,compilation=None]
startupMs min 60, median 69, max 71
Traces: Iteration 0 1 2

IoSettingsStartupBenchmark_startup[startup=WARM,compilation=SpeedProfile(iterations=3)]
startupMs min 56, median 58, max 66
Traces: Iteration 0 1 2

IoSettingsStartupBenchmark_startup[startup=WARM,compilation=BaselineProfile]
startupMs min 52, median 55, max 58
Traces: Iteration 0 1 2

IoSettingsStartupBenchmark_startup[startup=COLD,compilation=None]
startupMs min 53, median 55, max 58
Traces: Iteration 0 1 2

IoSettingsStartupBenchmark_startup[startup=COLD,compilation=SpeedProfile(iterations=3)]
startupMs min 54, median 62, max 63
Traces: Iteration 0 1 2

Change-Id: Id7cac284a5a42f5aeaf92ba6522061f794afcc96

M benchmark/macro/src/main/java/androidx/benchmark/macro/CompilationMode.kt
M jetifier/jetifier/migration.config

https://android-review.googlesource.com/1752966

Issue 192084204

Description

Issue summary

Comments

cc...@google.com <cc...@google.com> #2Jun 29, 2021 11:59PM

le...@google.com <le...@google.com> Jun 30, 2021 07:45PM

ra...@google.com <ra...@google.com> #3Jun 30, 2021 08:23PM

ap...@google.com <ap...@google.com> #4Jul 2, 2021 10:11PM

Add comment

Issue metadata