Status Update
No update yet.
Comments
cc...@google.com
Branch: androidx-main
commit adc821747c2a7f56b6708344dcb0dcc1c5707c90
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Tue Apr 26 13:55:46 2022
Workaround compilation reset unavailable on user builds
* cmd package compile --reset does not work on user builds.
* To work around this, we copy the apk to the tmp directory, and then kick off an un-install and a re-install.
* A side-effect of uninstalling / re-installing is that ART profiles end up in vfy, which is the state we expect.
* Added `compilation.enabled` as an instrumentation test argument, to disable this behavior if the benchmark wants
full control on CompilationModes.
Test: SmallListStartupBenchmark
Fixes:
Bug:
Relnote: "Fixed an issue where compiled methods were not correctly being reset between each macrobenchmark on unrooted builds. This unfortunately requires reinstalling the apk each iteration, which will clear application data for each macrobenchmark."
Before:
SmallListStartupBenchmark_startup[startup=COLD,compilation=None]
startupMs min 198.5, median 210.5, max 222.6
timeToInitialDisplayMs min 196.6, median 208.0, max 220.5
SmallListStartupBenchmark_startup[startup=COLD,compilation=BaselineProfile]
startupMs min 190.7, median 205.3, max 212.2
timeToInitialDisplayMs min 188.8, median 202.2, max 210.3
After the change:
SmallListStartupBenchmark_startup[startup=COLD,compilation=None]
startupMs min 223.9, median 230.1, max 243.8
timeToInitialDisplayMs min 222.1, median 228.3, max 242.0
SmallListStartupBenchmark_startup[startup=COLD,compilation=BaselineProfile]
startupMs min 205.2, median 213.5, max 228.7
timeToInitialDisplayMs min 203.3, median 211.7, max 227.0
Change-Id: I31c740019640da0d0913066bb9ac59ab5c2a9627
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/MacrobenchmarkScope.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/CompilationMode.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Arguments.kt
an...@google.com
Should we fix --reset? I haven't tested, but it looks like it's just a matter of changing enforceSystemOrRoot to enforceSystemOrRootOrShell on
As Nicolas argues in
an...@google.com
Yes, we should still fix --reset such that it works on user builds. Maybe allow the shell to clear caches, and not the app's themselves ? The goal is to be able to make benchmarks easier.
di...@google.com
le...@google.com
cc...@google.com
This bug is for the behavior in macrobenchmark and the workaround.
The issue of opening up that behavior is being tracked at
le...@google.com
di...@google.com
am...@google.com
This fix made all compilation=none and compilation=warmup measurements slower in CI
E.g. androidx.compose.integration.macrobenchmark.SmallListStartupBenchmark_startup_startup_COLD_compilation_None_ went from 340 -> 380ms.
As these were rooted devices, this indicates --reset (done in our rooted CI) does not put the device into the same state as fully reinstallation (after CL landed).
cc...@google.com
Filed
am...@google.com
Note from perf gap investigation in
am...@google.com
am...@google.com
Unfortunately it's not possible to just clear ART profiles prior to Android T on user builds. This is why we need to re-install the APK.
On the next iteration of Macrobenchmark (1.2.0-alpha01), we will try and clear profile caches if the session is rooted.
Description
Component used: androidx.benchmark.macro Version used: Version 1.4.0-alpha04
Calling
PerfettoTraceProcessor.Session#querywithSELECT slice.name, ...shows the following error in the IDE:id, name or spec expected, got 'SELECT s...'