IssueTracker

Are you seeing a line like the following? :

ERRORS (not suppressed): ...
WARNINGS (suppressed): ...

Can you try suppressing other errors, does that work? E.g. run the macrobenchmark on an emulator, which should require adding "EMULATOR" to the suppression list, and show it as a suppressed warning.

I tried this out and it worked for me - I just created a trivial test app and macrobench, test passed. Swapped to debuggable = true in the app, it broke with the expected error, and added the testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "DEBUGGABLE" in the macrobenchmark to make it work again (now with a warning at the top).

Sample project attached.

By the way, note that any measurements you make of your debug app won't be representative of your actual release app.

I am guessing that this is not due to Macrobenchmark itself but something in my rather large app build configuration.

Agree, does sound like a gradle configuration issue. My guess is some plugin or custom gradle logic is filtering or resetting the argument list (perhaps when it intends to add to them).

Also I was curious why this was an issue for your am instrument command when that's not running through Gradle, and an experiment shows that order matters, and is causing the same issue in a different way:

# works
adb shell am instrument -e androidx.benchmark.suppressErrors DEBUGGABLE -w com.example.macrobenchmark/androidx.test.runner.AndroidJUnitRunner

But if you pass args with -e after the -w they get ignored by am instrument:

# doesn't work
adb shell am instrument -w com.example.macrobenchmark/androidx.test.runner.AndroidJUnitRunner -e androidx.benchmark.suppressErrors DEBUGGABLE

If you want to test this entirely isolated from macrobenchmark, you can try and pass arguments via testInstrumentationRunnerArguments[] as you're already doing, and read them from any test module with InstrumentationRegistry.getArguments().getString("androidx.benchmark.suppressErrors", "DEFAULT_VALUE") (e.g. to assert you're getting the arg value you expect).

For the Gradle side, suggest looking for Gradle plugin code that modifies testInstrumentationRunnerArguments.

I'm going to close this as not reproducible since this doesn't appear to be a benchmark issue, but I'm subscribed to the bug if you have any follow up questions (or if you want to tell me why you want to benchmark a debug build :D )

2. I see the same problem with your sample app locally. I only updated the Android SDK location, built and installed both apks on my device. When I run adb I get the original error. You do not get the error. This to me means there may still be a general issue, or some issue I have that is different than your setup. Basically, I cannot blame my gradle build anymore because your sample app does the same thing.

adb shell am instrument -w com.example.macrobenchmark/androidx.test.runner.AndroidJUnitRunner by itself isn't expected to work, since that argument being set in the gradle file isn't used when invoking am instrument directly.

With the sample project, all three of the following work for me:

A) ./gradlew macrobenchmark:connectedAndroidTest (uses the value from testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "DEBUGGABLE")

B) running the test from Studio (uses the value from testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "DEBUGGABLE")

C) ./gradlew app:installBenchmark macrobenchmark:installBenchmark && adb shell am instrument -e androidx.benchmark.suppressErrors DEBUGGABLE -w com.example.macrobenchmark/androidx.test.runner.AndroidJUnitRunner

Do any of these work for you?

3. Regarding the test suggestions, I would run the getString() from some unit test I create and run on the device? Not sure how/where to run that code.

If you are setting testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "DEBUGGABLE" in your macrobenchmark's build.gradle and that doesn't work, though it does in the sample (for you), that would suggest it's a gradle issue in your project. You could write a test like the following to test instrumentation arguments are being passed as you expect:

@Test
fun checkArg() {
    assertEquals(
        "DEBUGGABLE",
        InstrumentationRegistry.getArguments().getString("androidx.benchmark.suppressErrors", "DEFAULT_VALUE")
    )
}

From there you could connect a debugger to check InstrumentationRegistry.getArguments().

Did you test on a physical device running Android 15 by chance?

Yes, this is a physical API 35 device. I also tried it on a API 35 emulator, which worked as well (once I changed DEBUGGABLE -> DEBUGGABLE,EMULATOR

Do any of these work for you?

Yes, they all work for me.

[testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "DEBUGGABLE"] being set in the gradle file isn't used when invoking am instrument directly.

I guess the "bug" here is that the test runner emits a fix-up message that does not apply to the form in which the tests were run in my case. Applying the fix does nothing as you say when executing the tests via adb. I confess to not understanding details of how the test runner works nor the interface between adb and the app test code, but perhaps the fix-up message could be updated to indicate that the fix only applies to test runs called from the IDE and gradle "connected" targets if others also have this misunderstanding in the future.

Thanks again for your help!

Also, perhaps this page could mention the same constraints regarding test params w/ adb: https://developer.android.com/topic/performance/benchmarking/macrobenchmark-instrumentation-args

I guess the "bug" here is that the test runner emits a fix-up message that does not apply to the form in which the tests were run in my case.

Fair, we can't detect how the am instrument command was run, but reopening to make the error message more correct/specific by saying something like: "add this instrumentation argument, which can be set in the following way if running via gradle or studio". That will also better account for people not using Gradle.

Doc fix out for review.

Doc update landed, https://developer.android.com/topic/performance/benchmarking/macrobenchmark-instrumentation-args now describes how to pass the argument directly to am instrument.