IssueTracker

Project: platform/frameworks/support
Branch: androidx-main

commit 782378852514e76eac3dfd115f7aca6c6ac6eec2
Author: Marcello Albano <maralb@google.com>
Date:   Fri May 17 10:53:04 2024

    Added enabling aot compilation prop in benchmark plugin
   
    Test: ./gradlew benchmark:benchmark-gradle-plugin:test
    Bug: 325512900
    Change-Id: Ie049719ecb7e17ae46abf7c858120f03a35d9222

M       benchmark/gradle-plugin/src/main/kotlin/androidx/benchmark/gradle/BenchmarkPlugin.kt
M       benchmark/gradle-plugin/src/test/kotlin/androidx/benchmark/gradle/BenchmarkPluginTest.kt
M       buildSrc/private/src/main/kotlin/androidx/build/AndroidXImplPlugin.kt

https://android-review.googlesource.com/3094561

Project: platform/frameworks/support
Branch: androidx-main

commit 5ba0e94ed48d43e30c51c999b247497e1ad675b0
Author: Marcello Albano <maralb@google.com>
Date:   Sat May 18 11:58:28 2024

    Added compilation mode into benchmark context
   
    Bug: 325512900
    Change-Id: If5612552a718b459e95cd96dc721e92a88e3fa8a
    Test: ShellTest
    Relnote: Added compilation mode to benchmark context

M       benchmark/benchmark-common/src/androidTest/assets/macro-legacy-trivialscrollbench.json
M       benchmark/benchmark-common/src/androidTest/assets/micro-legacy-trivialbench.json
M       benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/ResultWriterTest.kt
M       benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/ShellTest.kt
A       benchmark/benchmark-common/src/main/java/androidx/benchmark/PackageInfo.kt
M       benchmark/benchmark-common/src/main/java/androidx/benchmark/Shell.kt
M       benchmark/benchmark-common/src/main/java/androidx/benchmark/json/BenchmarkData.kt
M       benchmark/integration-tests/macrobenchmark/src/main/java/androidx/benchmark/integration/macrobenchmark/CompilationModeTest.kt

https://android-review.googlesource.com/3094294

Screenshot above shows synchronizedIncrementBenchmark finishing warmup *before* JIT of the relevant benchmark code is done. Result numbers started at e.g. ~1000ns, dropping to ~200ns part way through the resultset.

+ngeoffray as an FYI

Project: platform/frameworks/support
Branch: androidx-master-dev

commit 345f86d34f7e373f464c0d8185e392b067a2de4a
Author: Chris Craik <ccraik@google.com>
Date:   Wed Oct 09 17:37:17 2019

    Bump thread priority of benchmarks and JIT during benchmarks
   
    The JIT thread is so low priority that other parallel tasks can starve
    it, especially for the first few benchmarks when a process runs.
   
    The system can spin up significant background work right after install
    and/or instrumentation start, and on locked devices with only two big
    cores, there aren't enough CPUs to go around - warmup and benchmark
    both complete before relevant JIT is complete.
   
    Now, we bump the priority of both the benchmark and JIT thread.
    Tracing benchmarks show that the JIT thread goes much faster, which
    should significantly reduce the chance we capture results on unjitted
    code.
   
    This may also motivate us to use CPU affinity + locked small cores in
    the future, we can keep monitoring.
   
    Test: ./gradlew benchmark:b-c:cC
    Test: ./gradlew benchmark:b-b:cC
    Test: ./gradlew recyclerview:r-b:cC
   
    This CL also adds more logging, and unifies all logging under
    "benchmark" tag. This logging was very useful in discovering and
    diagnosing the priority problem, since it showed the edge cases where
    jit finished *during* the measure pass.
   
    Bug: 140773023
    Bug: 142058671
   
    Change-Id: If542e3cb8867165cf7b4688090ee534e68a23562

M       benchmark/common/src/androidTest/java/androidx/benchmark/BenchmarkStateTest.kt
M       benchmark/common/src/main/java/androidx/benchmark/BenchmarkState.kt
A       benchmark/common/src/main/java/androidx/benchmark/ThreadPriority.kt
M       benchmark/common/src/main/java/androidx/benchmark/WarmupManager.kt
M       benchmark/junit4/src/main/java/androidx/benchmark/junit4/BenchmarkRule.kt

https://android-review.googlesource.com/1138018
https://goto.google.com/android-sha1/345f86d34f7e373f464c0d8185e392b067a2de4a

After 15+ runs, recyclerview-benchmark and benchmark-benchmark are *significantly* more stable, see attached screenshots.

There may still be more opportunity here on big/little devices - enabling small cores, and using affinity to keep JIT there, with the main benchmark thread on a big core.