Benchmark gradle plugin is not Gradle project isolation safe [363325823]

Fixed

Bug

Status Update

No update yet.

Description

au...@google.com

created issue #1

Aug 30, 2024 09:32PM

Benchmark gradle plugin is not Gradle project isolation safe as it accesses root project to create lockClocks and unlockClocks tasks which requires root project to be configured before the project applying the benchmark plugin.

Comments

cc...@google.com <cc...@google.com> #2Aug 30, 2024 09:37PM

Reassigned to ma...@google.com.

Saw this as well on Mokey running recent tot master build.

First benchmark only shows jit happening during the allocation phase

Benchmark in the middle of the suite shows it happening during measurement as well, before method tracing and allocation counting.

I verified during the run that the platform still considers the app compiled:

> adb shell dumpsys package dexopt | grep -A1 androidx.compose.ui.benchmark.test
  [androidx.compose.ui.benchmark.test]
    path: /data/app/~~E9h-5MtFh1GXjXCq3kQo8g==/androidx.compose.ui.benchmark.test-jOZKJEy6zBL_SpFBJqlisA==/base.apk
      arm64: [status=speed] [reason=cmdline] [primary-abi]
        [location is /data/app/~~E9h-5MtFh1GXjXCq3kQo8g==/androidx.compose.ui.benchmark.test-jOZKJEy6zBL_SpFBJqlisA==/oat/arm64/base.odex]

Disabling method tracing works around this issue, but we don't seem to see it in CI, on older platform versions (wembley 30, sargo 31): https://screenshot.googleplex.com/57EEmDd8MBNvdwE

My guess is that when turning off method tracing, something fails to be cleaned up, and we don't get to keep using the odex/art files from the speed compilation.

ap...@google.com <ap...@google.com> #3Sep 16, 2024 05:26PM

Appears to be an ART regression starting in API 34

Bramble, API 33

google/bramble/bramble:13/TQ3A.230705.001/10216780:userdebug/dev-keys

112,994   ns    Trace    Method Trace    ModifiersBenchmark.baseReuse[Modifier_10x]
129,934   ns    Trace    Method Trace    ModifiersBenchmark.fullReuse[Modifier_10x]

Bramble, API 34

google/bramble/bramble:14/UP1A.231005.007/10754064:userdebug/dev-keys

 77,300   ns    Trace    Method Trace    ModifiersBenchmark.baseReuse[Modifier_10x]
395,151   ns    Trace    Method Trace    ModifiersBenchmark.fullReuse[Modifier_10x]

Will disable method tracing by default on API 34, and file a bug to ART.

ma...@google.com <ma...@google.com> Sep 18, 2024 06:04PM

Marked as fixed.

au...@google.com <au...@google.com> #4Oct 2, 2024 08:02AM

Filed art bug here: b/303686344

cc...@google.com <cc...@google.com> #5Oct 2, 2024 04:44PM

Project: platform/frameworks/support
Branch: androidx-main

commit 274ff7860d569477ba45d2d48c73bce4a62241ad
Author: Chris Craik <ccraik@google.com>
Date: Thu Oct 05 13:59:01 2023

Disable method tracing by default on API 34+

Bug:303660864
Bug:303686344

Test: ./gradlew bench:b-c:cC
Test: ModifiersBenchmark # on API 33, 34

Method tracing resets compilation state, which doesn't recover on API 34.

Changing method tracing to be off by default while we investigate this.

Change-Id: I568fc7a8a1029250df183bd65baa6b360cd105d5

M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/BenchmarkStateTest.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Arguments.kt

https://android-review.googlesource.com/2776435

pr...@google.com <pr...@google.com> #6Oct 2, 2024 05:37PM

Thanks for the quick response, Chris!