IssueTracker

Component used: androidx.benchmark:benchmark-macro-junit4 Version used: 1.2.0-alpha04

Devices/Android versions reproduced on:

pixel2Api31(ManagedVirtualDevice) {
  device = "Pixel 2"
  apiLevel = 31
  systemImageSource = "aosp"
}

If this is a bug in the library, we would appreciate if you could attach:

• Sample project to trigger the issue:

  https://github.com/roliveiravictor/Automatic-Baseline-Profiles/tree/feature/baseline-profile-generator

• Reproduction step: Run ./gradlew :benchmark:pixel2Api31BenchmarkAndroidTest

> Task :benchmark:pixel2Api31BenchmarkAndroidTest
Starting 2 tests on pixel2Api31

com.example.benchmark.BaselineProfileGenerator > startup[pixel2Api31] FAILED 
        java.lang.IllegalStateException: The profile is empty.
                at androidx.benchmark.macro.BaselineProfilesKt.profile(BaselineProfiles.kt:164)
Tests on pixel2Api31 failed: There was 1 failure(s).

11:29:58 I/XmlResultReporter: XML test result file generated at /Users/vrocha/AndroidStudioProjects/AutomaticBaselineProfiles/benchmark/build/outputs/androidTest-results/managedDevice/pixel2Api31/TEST-pixel2Api31-_benchmark-.xml. Total tests 2, failure 1, passed 1, 

Context:

I would like to generate Baseline Profile rules with GitHub actions. As known, Macrobenchmark can't run on emulators, and Baseline Profile to be generated through ManagedVirtualDevice requires the execution of all instrumented tests (including benchmarks, whenever available.)

By enabling dryRunMode, I can avoid benchmark failures due to emulators' absence. Still, no Baseline Profile gets to be generated.

Question:

Considering this is a bug, could it be fixed so that we can get to run :benchmark:pixel2Api31BenchmarkAndroidTest and generate the Baseline Profile without annotating ExampleStartupBenchmark to be ignored and setting dryRunMode.enable=false?

https://github.com/roliveiravictor/Automatic-Baseline-Profiles/pull/6/files