Add Systrace support [145598917]

Fixed

Feature Request

Status Update

No update yet.

Description

pa...@google.com

created issue #1

Dec 3, 2019 06:12PM

Add systrace collection and reporting into the benchmarking library .

one-pager:

https://docs.google.com/document/d/1mzVNUV8qwvq6yKadFjQdqTy41t9EoSCQP6l0bqVAV3A

Comments

ap...@google.com <ap...@google.com> #2Oct 19, 2020 11:45PM

Thanks for your detailed post.

However, benchmark build type is under configured: at least isProfileable is not set to true for existing build type, probably there's more.

We should set isProfileable = true by default when overriding an existing benchmark build type.

This issue is not about some specific configuration flag, but the general approach of dealing with external configuration. As a developer adopting baseline profiles, it seems extremely risky to me using a custom configuration due to how it's applied under the hood and the fact it may break default configuration.

I agree that is not great but this is a little tricky to do. For custom baseline profile build types we override all the properties. For benchmark I left it open to configure but it's mostly about these 2 properties:

isMinifyEnabled
isShrinkResources

I don't have a way to see if the user is setting them before overriding, so for this reason, I'd prefer not to. I agree with you that some other properties could be set by default to make this easier, i.e.:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

The reason why I mentioned the release signing config in the beginning is because I want to use debug signing config.

In the specific of your issue, i.e. using a debug certificate can you override the benchmark and baseline profile setting? You should be able to do something like:

android {
    buildTypes {
        release { ... }
        debug { ... }
        benchmarkRelease {
            ...
            signingConfig signingConfigs.debug
        }
        nonMinifiedRelease {
            ...
            signingConfig signingConfigs.debug
        }
    }
}

sj...@google.com <sj...@google.com> Oct 21, 2020 08:13PM

Reassigned to cc...@google.com.

cc...@google.com <cc...@google.com> #3Oct 21, 2020 08:26PM

Project: platform/frameworks/support
Branch: androidx-main
Author: Marcello Albano <maralb@google.com>
Link: https://android-review.googlesource.com/3303459

Added override for debuggable and profileable for benchmark builds in bpgp

Expand for full commit details

Added override for debuggable and profileable for benchmark builds in bpgp

Test: ./gradlew :benchmark:benchmark-baseline-profile-gradle-plugin:test
Bug: 369213505
Relnote: "isProfileable is always overridden in benchmark builds,
  and isDebuggable is also now always overridden in both benchmark and
  nonMinified (baseline profile capture) builds."
Change-Id: I487fa71083921682173f04fcbb477be5baf165f8

Files:

M benchmark/baseline-profile-gradle-plugin/src/main/kotlin/androidx/baselineprofile/gradle/apptarget/BaselineProfileAppTargetPlugin.kt
M benchmark/baseline-profile-gradle-plugin/src/test/kotlin/androidx/baselineprofile/gradle/apptarget/BaselineProfileAppTargetPluginTest.kt

Hash: 1906bbe52ba7ccb9ca0e1c1d6de33e7c91b5c6f0
Date: Fri Oct 11 10:07:06 2024

ap...@google.com <ap...@google.com> #4Oct 29, 2020 07:03PM

I've landed a change that will set the following properties also when the benchmark build type already exists:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

As well as the following for agp 8.0:

isDebuggable = false

I'm going ahead and closing this - if you've further questions please answer here and will reopen. Thanks.

ap...@google.com <ap...@google.com> #5Nov 2, 2020 10:30PM

The following release(s) address this bug.It is possible this bug has only been partially addressed:

androidx.benchmark:benchmark-baseline-profile-gradle-plugin:1.4.0-alpha04

cc...@google.com <cc...@google.com> #6Nov 3, 2020 10:33PM

Reassigned to cc...@google.com.

Since release signing config is used by default instead of debug is not mentioned in release notes - maybe this is a bug?

Cause the last time I found it mentioned in the release notes was in version 1.1

signingConfig.debug is used as the default signing config (b/153583269)

So, if the switch to the release one indeed happened - maybe it's an issue?

cc...@google.com <cc...@google.com> #7Nov 3, 2020 10:51PM

The new build types should inherit the certificates from the release build type, unless none is specified in which case debug is used: code link

This should be updated in the documentation, thanks for mentioning it. Will take care of updating it.

ap...@google.com <ap...@google.com> #8Nov 4, 2020 04:03AM

Project: platform/frameworks/support
Branch: androidx-master-dev

commit 4724b1f55919968f2a486da5f8f8b064927911cf
Author: Chris Craik <ccraik@google.com>
Date: Tue Nov 03 14:45:09 2020

Add requestLegacyExternalStorage to benchmarks

Bug: 145598917
Test: ./gradlew benchmark:b-b:installRelease && adb shell am instrument -w -r -e debug false -e newRunListenerMode true -e class 'androidx.benchmark.benchmark.TrivialJavaBenchmark' -e androidx.benchmark.output.enable true -e listener androidx.benchmark.junit4.InstrumentationResultsRunListener androidx.benchmark.benchmark.test/androidx.benchmark.junit4.AndroidBenchmarkRunner

Turning on benchmark output has caused benchmarks to start crashing on
API 29, due to inability to access output dir.

Note: unable to reproduce this crash on my local Q device, even with
manual am instrument command above

Change-Id: Ib19ec37ec10046b493c8b44207aab0ce93ad24b6

M ads/ads-identifier-benchmark/src/androidTest/AndroidManifest.xml
M appcompat/appcompat-benchmark/src/androidTest/AndroidManifest.xml
M benchmark/benchmark/src/androidTest/AndroidManifest.xml
M collection/collection-benchmark/src/androidTest/AndroidManifest.xml
M navigation/benchmark/src/androidTest/AndroidManifest.xml
M recyclerview/recyclerview-benchmark/src/androidTest/AndroidManifest.xml
M room/benchmark/src/androidTest/AndroidManifest.xml
M slices/benchmark/src/androidTest/AndroidManifest.xml
M work/workmanager-benchmark/src/androidTest/AndroidManifest.xml

https://android-review.googlesource.com/1486358

cc...@google.com <cc...@google.com> #9Nov 9, 2020 08:33PM

We've got initial traces in CI! For example, sample sponge2 results. Note all the entries with "trace" in the name.

Current Constraints:

only captured on API 29: Pixel 2016 (can support 28, with some additional work)
only small subset of benchmarks capture a trace, due to overhead (b/171749803)

Next step is to capture traces for all benchmarks with app tag disabled. This will mean no e.g. compose trace sections will be captured, but it will allow us to see CPU usage, thread scheduling, GCs, etc. We'll keep the app tag enabled on the *WithTracing variants.

ap...@google.com <ap...@google.com> #10Aug 13, 2021 04:21PM

Project: platform/frameworks/support
Branch: androidx-main

commit f5d35eb9ca852dfac791c2dc1d6ce953b2843880
Author: Chris Craik <ccraik@google.com>
Date: Wed Aug 11 17:35:21 2021

Move perfetto capture and protos to benchmark-common

Test: ./gradlew benchmark:benchmark-common:cC benchmark:benchmark-macro:cC benchmark:benchmark-macro-junit4:cC benchmark:benchmark-junit4:cC
Bug: 196115387
Bug: 145598917
Relnote: N/A

PerfettoTraceProcessor and its binary will stay in benchmark-macro.

Moves PerfettoRule to benchmark-junit, so that microbenchmarks don't
need to depend on benchmark-macro-junit4. Eventually, this
functionality can be merged into BenchmarkRule, as long as
in-process/app tag tracing is configurable, and off by
default. UserspaceTracing will enable us to still capture
BenchmarkState lifecycle events, like different modes (warmup /
measure / allocation count).

This unblocks raising the minAPI of Macrobench to the long-term goal
of API 21, which is the lowest that can support shell commands, as
well as perfetto capture.

This change also moves a few supporting classes, such as DeviceInfo
and ConfigurationError, which were always intended to gradually
replace benchmark-common's error / device state querying. This starts
that process.

As testutils-ktx is used by benchmark-common now, add to all project
config flavors.

Change-Id: I6559b6735d0106e38ec5d8eaeddcac9297d0c431

M benchmark/benchmark-common/api/current.txt
M benchmark/benchmark-common/api/public_plus_experimental_current.txt
M benchmark/benchmark-common/api/restricted_current.txt
M benchmark/benchmark-common/build.gradle
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/ConfigurationErrorTest.kt
A benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/Packages.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/ShellBehaviorTest.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/ShellTest.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/UserspaceTracingTest.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/perfetto/AtraceTagTest.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/perfetto/PerfettoConfigTest.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/perfetto/UiStateTest.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/ConfigurationError.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/DeviceInfo.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/UserspaceTracing.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/AtraceTag.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoCapture.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoCaptureWrapper.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoConfig.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoHelper.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/UiState.kt
M benchmark/benchmark-common/src/main/java/perfetto/protos/package-info.java
M benchmark/benchmark-common/src/main/proto/perfetto_config.proto
M benchmark/benchmark-common/src/main/proto/perfetto_trace.proto
M benchmark/benchmark-junit4/api/current.txt
M benchmark/benchmark-junit4/api/public_plus_experimental_current.txt
M benchmark/benchmark-junit4/api/restricted_current.txt
M benchmark/benchmark-junit4/build.gradle
M benchmark/benchmark-junit4/src/androidTest/java/androidx/benchmark/junit4/PerfettoRuleTest.kt
M benchmark/benchmark-junit4/src/main/java/androidx/benchmark/junit4/PerfettoRule.kt
M benchmark/benchmark-macro-junit4/api/current.txt
M benchmark/benchmark-macro-junit4/api/public_plus_experimental_current.txt
M benchmark/benchmark-macro-junit4/api/restricted_current.txt
M benchmark/benchmark-macro/api/current.txt
M benchmark/benchmark-macro/api/public_plus_experimental_current.txt
M benchmark/benchmark-macro/api/restricted_current.txt
M benchmark/benchmark-macro/build.gradle
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/FileLinkingRule.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/StartupTimingMetricTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoCaptureTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessorTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/CompilationMode.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Macrobenchmark.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessor.kt
M benchmark/benchmark/build.gradle
M benchmark/benchmark/src/androidTest/java/androidx/benchmark/benchmark/PerfettoOverheadBenchmark.kt
M buildSrc/private/src/main/kotlin/androidx/build/uptodatedness/TaskUpToDateValidator.kt
M compose/ui/ui-graphics/benchmark/build.gradle
M compose/ui/ui-graphics/benchmark/src/androidTest/java/androidx/compose/ui/graphics/benchmark/VectorBenchmarkWithTracing.kt
M development/build_log_simplifier/messages.ignore
M settings.gradle

https://android-review.googlesource.com/1795168

cc...@google.com <cc...@google.com> #11Oct 13, 2021 01:26AM

I see the next step here being to always enable tracing, but to disable the app tag. Just need to use something (like ForceTracing, which just landed in macrobench) to ensure high level lifecycle events are captured.

ap...@google.com <ap...@google.com> #12Nov 30, 2021 10:06PM

Marked as fixed.

Project: platform/frameworks/support
Branch: androidx-main

commit ec8b0e22b5e2415f1c165d5ff1b9e73fff4b956b
Author: Chris Craik <ccraik@google.com>
Date: Fri Nov 19 14:52:10 2021

Capture Perfetto traces during Microbenchmarks

Bug: 205636583
Fixes: 145598917
Test: ./gradlew benchmark-common
Relnote: "Add low-overhead System Tracing to microbench output on
Android Q (API 29+). Note that this does not currently capture custom
tracing (via android.os.Trace or androidx.tracing Jetpack APIs) to
avoid affecting results. This tracing should be useful in diagnosing
instability, especially from sources outside the benchmark."

Uses same timing information as microbench itself to generate
userspace tracing information, which is appended into a captured
perfetto trace.

Eventually this approach could enable other metrics available in
traces (such as time a thread is running vs not).

This approach has multiple drawbacks/constraints currently:

* Ignores calls to Trace.beginSection/endSection
* These have been found to be too high overhead in common
scenarios to be on by default. Turning it on drastically affected
many compose benchmarks.
* Eventually we can either make app tracing tag configurable,
offer microbenchmark-specific userspace tracing (easier than
general case, since we can append manually), or offer a more
general userspace tracing solution (which would also be much
lower overhead).
* Puts microbench library tracing in separate track, separate from
the app process
* This can be fixed with improvements to the proto generation to
pick the correct track id, given the test TID .
* Only traces with bundled perfetto (API 29+)
* If deemed useful enough, we can use unbundled perfetto as well
(move perfetto tracebox from benchmark-macro to benchmark-common)

Change-Id: I298be3c9f9f7ef9ba1d2c63d79c864e8c8b16e04

M benchmark/benchmark-junit4/api/current.txt
M benchmark/benchmark-junit4/api/restricted_current.txt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/UserspaceTracing.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/MetricCaptureTest.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/InstrumentationResults.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/MetricsContainer.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoCaptureWrapper.kt
M benchmark/benchmark-junit4/src/main/java/androidx/benchmark/junit4/PerfettoRule.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/BenchmarkState.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/InstrumentationResultsTest.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/MetricCapture.kt
M benchmark/benchmark-junit4/src/main/java/androidx/benchmark/junit4/BenchmarkRule.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/StartupTimingMetricTest.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoCapture.kt
M benchmark/benchmark-junit4/api/public_plus_experimental_current.txt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Profiler.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/BenchmarkStateTest.kt

https://android-review.googlesource.com/1898071