AutomaticGenerationDuringBuild and managedDevices used together break benchmark tests [355515798]

Fixed

Bug

Status Update

No update yet.

Description

ma...@google.com

created issue #1

Jul 25, 2024 11:31PM

When automaticGenerationDuringBuild is enabled and managedDevices are used, benchmark tests fail with:

java.lang.AssertionError: ERRORS (not suppressed): EMULATOR
WARNINGS (suppressed): 

ERROR: Running on Emulator
    Benchmark is running on an emulator, which is not representative of
    real user devices. Use a physical device to benchmark. Emulator
    benchmark improvements might not carry over to a real user's
    experience (or even regress real device performance).

Example configuration in consumer module:

baselineProfile {
    variants {
        // This works
        create("release") {
            automaticGenerationDuringBuild = true
            saveInSrc = true
        }
    }
}

Example configuration in producer module:

baselineProfile {
    useConnectedDevices = false
    managedDevices = [...] 
}

Comments

ap...@google.com <ap...@google.com> #2Aug 19, 2024 11:59PM

Project: platform/frameworks/support
Branch: androidx-main

commit 0c2ddcd70f6090ccc660e49c03048e1de85c2073
Author: Marcello Albano <maralb@google.com>
Date: Tue Aug 13 15:44:14 2024

Added benchmark argument to skip tests when running on emulator

Bug:355515798
Test: ./gradlew :benchmark:benchmark-baseline-profile-gradle-plugin:test :benchmark:benchmark-common:test :benchmark:benchmark-macro:test
Relnote: "When automaticGenerationDuring build is enabled, benchmakrs
will also trigger baseline profile generation. This will fail, if
emulators are used. With the new argument we can instead skip the test."

Change-Id: If3f51413074ac8b29bb42740038d285121430b50

M benchmark/baseline-profile-gradle-plugin/src/main/kotlin/androidx/baselineprofile/gradle/producer/BaselineProfileProducerExtension.kt
M benchmark/baseline-profile-gradle-plugin/src/main/kotlin/androidx/baselineprofile/gradle/producer/BaselineProfileProducerPlugin.kt
M benchmark/baseline-profile-gradle-plugin/src/main/kotlin/androidx/baselineprofile/gradle/utils/Constants.kt
M benchmark/baseline-profile-gradle-plugin/src/test/kotlin/androidx/baselineprofile/gradle/producer/BaselineProfileProducerPluginTest.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Arguments.kt
M benchmark/benchmark-junit4/src/main/java/androidx/benchmark/junit4/BenchmarkRule.kt
M benchmark/benchmark-macro/build.gradle
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Macrobenchmark.kt

https://android-review.googlesource.com/3217071

ap...@google.com <ap...@google.com> #3Aug 20, 2024 12:11AM

Project: platform/frameworks/support
Branch: androidx-main

commit 1587de845dc66a5dd7a2fc016f4653d6691a6bf2
Author: Marcello Albano <maralb@google.com>
Date: Mon Aug 19 16:03:15 2024

Improved DeviceInfo.isEmulator to consider `sdk_` prefix

Test: existing
Bug: 355515798
Change-Id: I954a47ca80673b8c7201257eccb94c21dad494bd

M benchmark/benchmark-common/src/main/java/androidx/benchmark/DeviceInfo.kt

https://android-review.googlesource.com/3228989

ma...@google.com <ma...@google.com> #4Aug 20, 2024 04:02PM

Marked as fixed.

We'll try to put this in version 1.3.1 of the plugin.

na...@google.com <na...@google.com> #5Sep 18, 2024 05:37PM

The following release(s) address this bug.It is possible this bug has only been partially addressed:

androidx.benchmark:benchmark-baseline-profile-gradle-plugin:1.3.1
androidx.benchmark:benchmark-common:1.3.1
androidx.benchmark:benchmark-junit4:1.3.1
androidx.benchmark:benchmark-macro:1.3.1

Issue 355515798

Description

Issue summary

Comments

ap...@google.com <ap...@google.com> #2Aug 19, 2024 11:59PM

ap...@google.com <ap...@google.com> #3Aug 20, 2024 12:11AM

ma...@google.com <ma...@google.com> #4Aug 20, 2024 04:02PM

na...@google.com <na...@google.com> #5Sep 18, 2024 05:37PM

Add comment

Issue metadata