IssueTracker

Saw this as well on Mokey running recent tot master build.

First benchmark only shows jit happening during the allocation phase

Benchmark in the middle of the suite shows it happening during measurement as well, before method tracing and allocation counting.

I verified during the run that the platform still considers the app compiled:

> adb shell dumpsys package dexopt | grep -A1 androidx.compose.ui.benchmark.test
  [androidx.compose.ui.benchmark.test]
    path: /data/app/~~E9h-5MtFh1GXjXCq3kQo8g==/androidx.compose.ui.benchmark.test-jOZKJEy6zBL_SpFBJqlisA==/base.apk
      arm64: [status=speed] [reason=cmdline] [primary-abi]
        [location is /data/app/~~E9h-5MtFh1GXjXCq3kQo8g==/androidx.compose.ui.benchmark.test-jOZKJEy6zBL_SpFBJqlisA==/oat/arm64/base.odex]

Disabling method tracing works around this issue, but we don't seem to see it in CI, on older platform versions (wembley 30, sargo 31): https://screenshot.googleplex.com/57EEmDd8MBNvdwE

My guess is that when turning off method tracing, something fails to be cleaned up, and we don't get to keep using the odex/art files from the speed compilation.

Appears to be an ART regression starting in API 34

Bramble, API 33

google/bramble/bramble:13/TQ3A.230705.001/10216780:userdebug/dev-keys

112,994   ns    Trace    Method Trace    ModifiersBenchmark.baseReuse[Modifier_10x]
129,934   ns    Trace    Method Trace    ModifiersBenchmark.fullReuse[Modifier_10x]

Bramble, API 34

google/bramble/bramble:14/UP1A.231005.007/10754064:userdebug/dev-keys

 77,300   ns    Trace    Method Trace    ModifiersBenchmark.baseReuse[Modifier_10x]
395,151   ns    Trace    Method Trace    ModifiersBenchmark.fullReuse[Modifier_10x]

Will disable method tracing by default on API 34, and file a bug to ART.

Filed art bug here: b/303686344

Thanks for the quick response, Chris!

Chuck reported this same issue today, where another new benchmark being added was performing far worse on runs after the first, this time on Sargo API 31.

Inspecting the trace showed that the same issue was occurring, and after being unable to reproduce the issue on the same OS build, I realized this was because the breakage had been backported by mainline updates to older devices.

Chuck's repro device:

google/sargo/sargo:12/SQ1A.220205.002/8010174:userdebug/dev-keys
package:com.google.android.art versionCode:341010052

My device that didn't repro:

google/bramble/bramble:12/SQ1A.220205.002/8010174:userdebug/dev-keys
package:com.google.android.art versionCode:310924000

The reason I'd not been reproducing this bug generally was because I test on devices not connected to wifi. Chuck had a connected device (for adb over wifi), and had received a mainline update I hadn't been seeing in my previous experiments with the bug, where I tested on both api 33 and api 34. API 34 exhibits the bug without a mainline update, but anything 31+ can exhibit the bug.

For now, I'll turn off method tracing by default on API 31+ to be safe.

In the future, we'll look at something based more specifically on mainline versions, and bundling mainline updates with the fix (once available) into the library or gradle plugin, but we can't make it transparent locally, as mainline updates require a reboot.

@chris is this bug fixed?

Marking as fixed since the default is now working correctly, filed b/316174880 to make further improvements and reenable tracing when it's safe, warning when it's not.

The following release(s) address this bug.It is possible this bug has only been partially addressed:

• androidx.benchmark:benchmark-common:1.3.0-alpha01

This isn't fully addressed - we noticed that the error for measurement after method tracing isn't on by default, working on a CL for that.

The following release(s) address this bug.It is possible this bug has only been partially addressed:

• androidx.benchmark:benchmark-common:1.3.0-beta01

Romain saw this on an API 35 device, reopened the ART bug, and marking this blocked on that.

Project: platform/frameworks/support
Branch: androidx-main
Author: Chris Craik <ccraik@google.com>
Link:      https://android-review.googlesource.com/3460462

Fix method tracing affecting measurements when art built from source on API 34

Fix method tracing affecting measurements when art built from source on API 34 
 
Fixes: 303660864 
Test: TrivialKotlinBenchmark (with duplicated benchmark on VP1A.230706.001) 
Test: DeviceInfoTest 
 
Change-Id: Ic7cad122150c515779365d6565f1c62c69213f82

The following release(s) address this bug.It is possible this bug has only been partially addressed:

• androidx.benchmark:benchmark-common:1.4.0-alpha07
• androidx.benchmark:benchmark-macro:1.4.0-alpha07

and as long as it's not, be mentioned on the docs with an explanation of what should be done about it.

I have at least added a warning to the docs for now.

I'll look into the bug you linked.

You can see the warning I added in your editor as a documentation comment on the EncryptedFile and EncryptedSharedPreferences classes. Those comments get copied into the reference info on the jetpack website:

https://developer.android.com/reference/androidx/security/crypto/EncryptedFile

https://developer.android.com/reference/androidx/security/crypto/EncryptedSharedPreferences

But I think it would be much better for us to provide an Android backup agent implementation that does the filtering for you.

You can definitely accomplish this with XML configuration: https://developer.android.com/guide/topics/data/autobackup#IncludingFiles

But you can also write a custom backup agent class to do the work at runtime. I will update the documentation later to include a reference XML configuration.

Thanks for your feedback!

The issue still persists.