Status Update
No update yet.
Comments
al...@google.com
Branch: androidx-main
commit bfdd111df98ae51af5266573ad915bfc006bb7bf
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Tue Jun 06 15:17:45 2023
Cleanup method tracing implementation in Macrobenchmark.
* No longer requires the use of `:benchmark:benchmark-internal`.
* Relies on `androidx.benchmark.profiling.mode` set to `MethodTracing`.
* Correctly handles process cold starts, given we use `am start --start-profiler <FILE> -W <INTENT>`.
Relnote: "
Provides the ability to generate method traces as a result of running a Macrobenchmark
using the following test instrumentation arguments.
```groovy
android {
defaultConfig {
testInstrumentationRunnerArguments [
'androidx.benchmark.profiling.mode': 'MethodTracing',
// Other arguments
]
}
}
```
"
Test: Ran TrivialStartupBenchmark + added unit tests.
Bug:
Change-Id: I7ad37bee17646cfef1564c9f0c84d20923d63ed9
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Arguments.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Shell.kt
D benchmark/benchmark-internal/build.gradle
D benchmark/benchmark-internal/src/main/AndroidManifest.xml
D benchmark/benchmark-internal/src/main/java/androidx/benchmark/internal/MethodTracingReceiver.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkScopeTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/StartupTimingMetricTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoSdkHandshakeTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/BaselineProfiles.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Macrobenchmark.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/MacrobenchmarkScope.kt
M benchmark/integration-tests/macrobenchmark-target/build.gradle
M settings.gradle
ra...@google.com
Remaining things to do here, copied from comment on CL:
- disable metric collection and reporting during method tracing - is this part done already, via the warning prefix?
- link the trace in the output (we may need a good general improvement to Outputs/Instrumentation stuff to make this easier, maybe we need to generalize ProfilerResult somehow, I can get to this eventually)
- run method tracing as a single iteration, in place of measurement (I think it's better to replace for macro, since runs are so long)
How big are the method traces for a startup vs scrolling macrobench, out of curiousity?
al...@google.com
Method traces are < 10 M for activity startup cases (~10s CUJ).
disable metric collection and reporting during method tracing - is this part done already, via the warning prefix?
We don't disable collection, but the results include a warning. We should add a prefix to our metrics like we do for microbenchmark.
Run method tracing as a single iteration, in place of measurement (I think it's better to replace for macro, since runs are so long)
Rather than do this, I wanted to introduce a phase where all we do is just method tracing after metrics are captured. This way, we don't affect metrics at all; but include a usable method trace in addition to the metrics.
cc...@google.com
ap...@google.com
Method traces are < 10 M for activity startup cases (~10s CUJ).
I expect startup cases we have are pretty trivial. What about scrolling cases?
introduce a phase
I don't know if I like that as the default, for method tracing or for sampling in macrobench, since tests are usually so much longer than micro. Consider a 5 min scrolling test, where you're just not collecting profiling results for minutes on end while the developer waits.
Could always eventually make it an option like androidx.benchmark.profiling.primaryMeasurePhase.enable
na...@google.com
Branch: androidx-main
commit 6a2d90e57ebe9de586b19f195a386229bffb9dc1
Author: Chris Craik <ccraik@google.com>
Date: Wed Jun 14 11:12:30 2023
Single output code for micro/macro studio metric and trace output
Bug: 227205461
Bug: 286306579
Bug: 285912360
Test: ./gradlew bench:b-c:cC
Test: ./gradlew bench:b-m:cC
Test: Micro, macro, and BP tests show correctly in Studio
Relnote: "Fix warnings to always be printed in Benchmark output in
Studio, and workaround leading whitespaces from Benchmark output not
showing up in Studio"
Unify ide output codepath for all benchmark/bp/tracerule code.
This is a necessary prerequisite for outputting additional metrics
from microbench, such as counters or trace metrics, so that when
additional metrics are included, they'll automatically show up.
This is also a necessary prerequisite for linking profiling traces
from macro in Studio.
This also fixed a couple long standing issues:
* Fixes warnings only going to one of the Studio v1/v2 string pair.
* Works around Studio swallowing leading whitespace by left aligning
macro metric names
Change-Id: Ia61d0fdc52ca0de2ece777ea6dd76bc77876fcf2
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/BenchmarkStateTest.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/InstrumentationResultsTest.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/BenchmarkState.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/InstrumentationResults.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Profiler.kt
M benchmark/benchmark-junit4/src/main/java/androidx/benchmark/junit4/PerfettoTraceRule.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/FileLinkingRule.kt
D benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/IdeSummaryStringTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/BaselineProfiles.kt
D benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/IdeSummaryString.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Macrobenchmark.kt
Description
Component used: androidx.benchmark
Version used: built from HEAD (master) on 6th June 2022
Devices/Android versions reproduced on: Google Pixel 5 (redfin-userdebug) running UpsideDownCake built from HEAD (master) on 6th June 2022
If this is a bug in the library, we would appreciate if you could attach:
atest cts/common/device-side/bedstead/nene/src/test/java/com/android/bedstead/nene/benchmarking/NeneBenchmarksTest.java)