IssueTracker

DESCRIBE THE ISSUE IN DETAIL:

When you attempt to run a specific macrobenchmark from Android Studio, the run configuration automatically sets the androidx.benchmark.enabledRules instrumentation argument to baselineprofile,macrobenchmark,microbenchmark, and somewhere in the Gradle logic it injects an instrumentation argument to select the specific test. baselineprofile,macrobenchmark,microbenchmark

If automatic baseline profile creation is enabled through the baseline profile plugin this those same instrumentation arguments are also passed to that instrumentation run, which results in failure to capture the correct baseline profile.

STEPS TO REPRODUCE:

Project setup

// macrobenchmark/build.gradle.kts
plugins {
  id("com.android.test")
  id("androidx.baselineprofile") // version 1.3.0-beta02 set in the root build.gradle.kts
  id("kotlin-android")
}

android {
  ... other config ...
  testOptions.managedDevices.devices {
    create<ManagedVirtualDevice>("pixel6Api33") {
      device = "Pixel 6"
      apiLevel = 33
      systemImageSource = "aosp"
    }
  }
  targetProjectPath = ":app"
}

baselineProfile {
   managedDevices += "pixel6Api33"
   useConnectedDevices = false
}

// app/build.gradle.kts
plugins {
  id("com.android.application")
  id("androidx.baselineprofile")
  id("kotlin-android")
}

android {
  ... other config ...
}

dependencies {
  baselineProfile(project(":macrobenchmark"))
  implementation("androidx.tracing:tracing-perfetto-binary:1.0.0")
}

baselineProfile {
  automaticGenerationDuringBuild = true
  saveInSrc = true
}

Make sure that the :macrobenchmark module has at least one instrumented test that has the MacrobenchmarkRule and one test that has the BaselineProfileRule.

If you now use click the run button on the macrobenchmark, the gradle task will fail because with the default run configuration it injects -e androidx.benchmark.enabledRules baselineprofile,macrobenchmark,microbenchmark for the instrumentation run that is executed on the managed virtual device and the MacrobenchmarkRule is set up to fail all tests eagerly if the device that they run on is an emulator.

If you now edit the run configuration in Android Studio and remove the androidx.benchmark.enabledRules instrumentation arguments, the instrumentation run for creating baseline profiles no longer fails, but now, because Android Studio injected the specific test class to run, it will only attempt to run that test class, but since macrobenchmark rules are not enabled, those tests will be skipped. The actual test that is supposed to go through the steps to capture a baseline profile is not executed at all.

So far the only proper workaround that I have found is do everything from commandline, with invoking the final instrumentation manually.

./gradlew :app:installBenchmarkRelease :macrobenchmark:installBenchmarkRelease
adb shell am instrument -w -e class com.example.macrobenchmark.StartupBenchmark \
    com.example.benchmark/androidx.test.runner.AndroidJUnitRunner

Studio Build: #AI-241.18034.62.2413.12138059

Version of Gradle Plugin: 8.5.1

Version of Gradle: 8.7

Version of Java: Azul 17

OS: MacOS