Assigned
Status Update
Comments
er...@gmail.com <er...@gmail.com> #2
When I run Macrobenchmark via the `connectedCheck` target with info logs on, I can see the arg being set:
```
INFO: Configuring Android Instrumentation driver: android_instrumentation_runtime {
instrumentation_info {
...
args_map {
key: "androidx.benchmark.suppressErrors"
value: "DEBUGGABLE"
}
}
}
```
Attempted to pass the flag via the `-e` option to the `instrument` command of ADB, but no change to error output:
```
$ adb shell am instrument -w com.ring.benchmark/androidx.test.runner.AndroidJUnitRunner -e androidx.benchmark.suppressErrors DEBUGGABLE
...
ERROR: Benchmark Target is Debuggable
```
```
INFO: Configuring Android Instrumentation driver: android_instrumentation_runtime {
instrumentation_info {
...
args_map {
key: "androidx.benchmark.suppressErrors"
value: "DEBUGGABLE"
}
}
}
```
Attempted to pass the flag via the `-e` option to the `instrument` command of ADB, but no change to error output:
```
$ adb shell am instrument -w com.ring.benchmark/androidx.test.runner.AndroidJUnitRunner -e androidx.benchmark.suppressErrors DEBUGGABLE
...
ERROR: Benchmark Target is Debuggable
```
cc...@google.com <cc...@google.com> #3
Are you seeing a line like the following? :
ERRORS (not suppressed): ...
WARNINGS (suppressed): ...
Can you try suppressing other errors, does that work? E.g. run the macrobenchmark on an emulator, which should require adding "EMULATOR" to the suppression list, and show it as a suppressed warning.
er...@gmail.com <er...@gmail.com> #4
I tried this out and it worked for me - I just created a trivial test app and macrobench, test passed. Swapped to debuggable = true
in the app, it broke with the expected error, and added the testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "DEBUGGABLE"
in the macrobenchmark to make it work again (now with a warning at the top).
Sample project attached.
By the way, note that any measurements you make of your debug app won't be representative of your actual release app.
cc...@google.com <cc...@google.com>
yo...@gmail.com <yo...@gmail.com> #5
Thank you for the fast response. Yes, I see the suppressed messages like:
```
java.lang.AssertionError: ERRORS (not suppressed): DEBUGGABLE
WARNINGS (suppressed):
ERROR: Benchmark Target is Debuggable
Target package com.ringapp.debug is running with debuggable=true in its manifest,
which drastically reduces runtime performance in order to support debugging
features. Run benchmarks with debuggable=false. Debuggable affects execution
speed in ways that mean benchmark improvements might not carry over to a
real user's experience (or even regress release performance).
While you can suppress these errors (turning them into warnings)
PLEASE NOTE THAT EACH SUPPRESSED ERROR COMPROMISES ACCURACY
// Sample suppression, in a benchmark module's build.gradle:
android {
defaultConfig {
testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "DEBUGGABLE"
}
}
at androidx.benchmark.ConfigurationErrorKt.checkAndGetSuppressionState(ConfigurationError.kt:102)
at androidx.benchmark.macro.MacrobenchmarkKt.checkErrors(Macrobenchmark.kt:190)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:229)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:401)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:108)
at com.ring.benchmark.AbstractStartupBenchmark.startup(StartupBenchmark.kt:61)
at com.ring.benchmark.AbstractStartupBenchmark.startupPartialCompilation(StartupBenchmark.kt:51)
```
If I set the EMULATOR flag in addition, and run w/ an emulator attached instead of a physical device:
```
testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "EMULATOR,DEBUGGABLE"
```
I get:
```
java.lang.AssertionError: ERRORS (not suppressed): EMULATOR DEBUGGABLE
WARNINGS (suppressed):
ERROR: Running on Emulator
Benchmark is running on an emulator, which is not representative of
real user devices. Use a physical device to benchmark. Emulator
benchmark improvements might not carry over to a real user's
experience (or even regress real device performance).
ERROR: Benchmark Target is Debuggable
Target package com.ringapp.debug is running with debuggable=true in its manifest,
which drastically reduces runtime performance in order to support debugging
features. Run benchmarks with debuggable=false. Debuggable affects execution
speed in ways that mean benchmark improvements might not carry over to a
real user's experience (or even regress release performance).
```
This leads me to conclude that the gradle declaration is having no effect for some reason. I am guessing that this is not due to Macrobenchmark itself but something in my rather large app build configuration. How can I further debug thigs? For example, I assume that the `testInstrumentationRunnerArguments` declaration causes some state change in the resulting benchmark apk. How can I verify this? I tried unzipping the benchmark apk and seems like a string match in dex files but otherwise no obvious way to see where the issue might be.
```
java.lang.AssertionError: ERRORS (not suppressed): DEBUGGABLE
WARNINGS (suppressed):
ERROR: Benchmark Target is Debuggable
Target package com.ringapp.debug is running with debuggable=true in its manifest,
which drastically reduces runtime performance in order to support debugging
features. Run benchmarks with debuggable=false. Debuggable affects execution
speed in ways that mean benchmark improvements might not carry over to a
real user's experience (or even regress release performance).
While you can suppress these errors (turning them into warnings)
PLEASE NOTE THAT EACH SUPPRESSED ERROR COMPROMISES ACCURACY
// Sample suppression, in a benchmark module's build.gradle:
android {
defaultConfig {
testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "DEBUGGABLE"
}
}
at androidx.benchmark.ConfigurationErrorKt.checkAndGetSuppressionState(ConfigurationError.kt:102)
at androidx.benchmark.macro.MacrobenchmarkKt.checkErrors(Macrobenchmark.kt:190)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:229)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:401)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:108)
at com.ring.benchmark.AbstractStartupBenchmark.startup(StartupBenchmark.kt:61)
at com.ring.benchmark.AbstractStartupBenchmark.startupPartialCompilation(StartupBenchmark.kt:51)
```
If I set the EMULATOR flag in addition, and run w/ an emulator attached instead of a physical device:
```
testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "EMULATOR,DEBUGGABLE"
```
I get:
```
java.lang.AssertionError: ERRORS (not suppressed): EMULATOR DEBUGGABLE
WARNINGS (suppressed):
ERROR: Running on Emulator
Benchmark is running on an emulator, which is not representative of
real user devices. Use a physical device to benchmark. Emulator
benchmark improvements might not carry over to a real user's
experience (or even regress real device performance).
ERROR: Benchmark Target is Debuggable
Target package com.ringapp.debug is running with debuggable=true in its manifest,
which drastically reduces runtime performance in order to support debugging
features. Run benchmarks with debuggable=false. Debuggable affects execution
speed in ways that mean benchmark improvements might not carry over to a
real user's experience (or even regress release performance).
```
This leads me to conclude that the gradle declaration is having no effect for some reason. I am guessing that this is not due to Macrobenchmark itself but something in my rather large app build configuration. How can I further debug thigs? For example, I assume that the `testInstrumentationRunnerArguments` declaration causes some state change in the resulting benchmark apk. How can I verify this? I tried unzipping the benchmark apk and seems like a string match in dex files but otherwise no obvious way to see where the issue might be.
Description
Component used: Microbenchmark
Version used: 1.2.0 beta01
Devices/Android versions reproduced on: Xiaomi Mi Note 10 Lite (API 31)
When I run the Microbenchmark test on my phone, the activity does not run and the test fails. I tried to run the activity manually and the exception I encountered was related to the Shell class, so the problem is probably here, also there is no info related to how to benchmark suspend function and what is preferred way to do that.