TraceSectionMetric in only one iteration produces missing results. [315827519]

Assigned

Bug

Status Update

No update yet.

Description

ml...@google.com

created issue #1

Dec 11, 2023 07:32PM

Component used: Macrobench 1.3.0-SNAPSHOT

If I have a TraceSectionMetric("my_trace") and the my_trace occurs only in one (first?) iteration of the macrobenchmark, the standard (non-sampled) benchmark results aren't produced properly -- it only has one item in the runs array and not for each run.

For example:

"timeToInitialDisplayMs": {
                    "minimum": 490.606384,
                    "maximum": 490.606384,
                    "median": 490.606384,
                    "runs": [
                        490.606384
                    ]
                },
..
..
"warmupIterations": 0,
"repeatIterations": 3,
"thermalThrottleSleepSeconds": 0

Attaching JSON results + perfetto traces for 3 iterations.

com_google_compose_integration_macrobenchmark-benchmarkData.json

20 KB

Download

InstalledAppsDebugStartupBenchmark_startup_Verify_app_com_example_jetcaster_jetcaster-debug_apk__iter002_2023-12-11-19-21-03.perfetto-trace

5.5 MB

Download

InstalledAppsDebugStartupBenchmark_startup_Verify_app_com_example_jetcaster_jetcaster-debug_apk__iter001_2023-12-11-19-20-56.perfetto-trace

5.6 MB

Download

InstalledAppsDebugStartupBenchmark_startup_Verify_app_com_example_jetcaster_jetcaster-debug_apk__iter000_2023-12-11-19-20-42.perfetto-trace

25 MB

Download

Comments

ml...@google.com <ml...@google.com> #2Dec 11, 2023 07:51PM

When I run Macrobenchmark via the `connectedCheck` target with info logs on, I can see the arg being set:

```
INFO: Configuring Android Instrumentation driver: android_instrumentation_runtime {
instrumentation_info {
...
args_map {
key: "androidx.benchmark.suppressErrors"
value: "DEBUGGABLE"
}
}
}
```

Attempted to pass the flag via the `-e` option to the `instrument` command of ADB, but no change to error output:

```
$ adb shell am instrument -w com.ring.benchmark/androidx.test.runner.AndroidJUnitRunner -e androidx.benchmark.suppressErrors DEBUGGABLE
...
ERROR: Benchmark Target is Debuggable
```

cc...@google.com <cc...@google.com> #3Apr 11, 2024 04:52PM

Are you seeing a line like the following? :

ERRORS (not suppressed): ...
WARNINGS (suppressed): ...

Can you try suppressing other errors, does that work? E.g. run the macrobenchmark on an emulator, which should require adding "EMULATOR" to the suppression list, and show it as a suppressed warning.

ml...@google.com <ml...@google.com> #4Apr 12, 2024 01:10PM

I tried this out and it worked for me - I just created a trivial test app and macrobench, test passed. Swapped to debuggable = true in the app, it broke with the expected error, and added the testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "DEBUGGABLE" in the macrobenchmark to make it work again (now with a warning at the top).

Sample project attached.

By the way, note that any measurements you make of your debug app won't be representative of your actual release app.

Issue 315827519

Description

Issue summary

Comments

ml...@google.com <ml...@google.com> #2Dec 11, 2023 07:51PM

cc...@google.com <cc...@google.com> #3Apr 11, 2024 04:52PM

ml...@google.com <ml...@google.com> #4Apr 12, 2024 01:10PM

Add comment

Issue metadata