Lower minApi level from Q (29) -> L (21) [183129298]

Fixed

Bug

Status Update

No update yet.

Description

cc...@google.com

created issue #1

Mar 18, 2021 07:57PM

Currently min API 30 due to using perfetto trace_processor_shell binary from Q platform build.

Switch to unbundled perfetto (at least on old devices) and trace_processor_shell built from github repo. Currently this is blocked on documentation for binary stripping (b/182513943). Min API for these is API 21, and planned to be maintained (see discussion in perfetto-dev group: https://groups.google.com/a/google.com/g/perfetto-dev/c/vRHKg4Cjpok)

Comments

cc...@google.com <cc...@google.com> Mar 19, 2021 10:50PM

Reassigned to ra...@google.com.

ra...@google.com <ra...@google.com> #2Apr 20, 2021 06:21PM

Thanks for your detailed post.

However, benchmark build type is under configured: at least isProfileable is not set to true for existing build type, probably there's more.

We should set isProfileable = true by default when overriding an existing benchmark build type.

This issue is not about some specific configuration flag, but the general approach of dealing with external configuration. As a developer adopting baseline profiles, it seems extremely risky to me using a custom configuration due to how it's applied under the hood and the fact it may break default configuration.

I agree that is not great but this is a little tricky to do. For custom baseline profile build types we override all the properties. For benchmark I left it open to configure but it's mostly about these 2 properties:

isMinifyEnabled
isShrinkResources

I don't have a way to see if the user is setting them before overriding, so for this reason, I'd prefer not to. I agree with you that some other properties could be set by default to make this easier, i.e.:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

The reason why I mentioned the release signing config in the beginning is because I want to use debug signing config.

In the specific of your issue, i.e. using a debug certificate can you override the benchmark and baseline profile setting? You should be able to do something like:

android {
    buildTypes {
        release { ... }
        debug { ... }
        benchmarkRelease {
            ...
            signingConfig signingConfigs.debug
        }
        nonMinifiedRelease {
            ...
            signingConfig signingConfigs.debug
        }
    }
}

Message last modified on Apr 20, 2021 06:23PM

ap...@google.com <ap...@google.com> #3Apr 20, 2021 06:28PM

Project: platform/frameworks/support
Branch: androidx-main
Author: Marcello Albano <maralb@google.com>
Link: https://android-review.googlesource.com/3303459

Added override for debuggable and profileable for benchmark builds in bpgp

Expand for full commit details

Added override for debuggable and profileable for benchmark builds in bpgp

Test: ./gradlew :benchmark:benchmark-baseline-profile-gradle-plugin:test
Bug: 369213505
Relnote: "isProfileable is always overridden in benchmark builds,
  and isDebuggable is also now always overridden in both benchmark and
  nonMinified (baseline profile capture) builds."
Change-Id: I487fa71083921682173f04fcbb477be5baf165f8

Files:

M benchmark/baseline-profile-gradle-plugin/src/main/kotlin/androidx/baselineprofile/gradle/apptarget/BaselineProfileAppTargetPlugin.kt
M benchmark/baseline-profile-gradle-plugin/src/test/kotlin/androidx/baselineprofile/gradle/apptarget/BaselineProfileAppTargetPluginTest.kt

Hash: 1906bbe52ba7ccb9ca0e1c1d6de33e7c91b5c6f0
Date: Fri Oct 11 10:07:06 2024

ap...@google.com <ap...@google.com> #4Apr 20, 2021 06:28PM

I've landed a change that will set the following properties also when the benchmark build type already exists:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

As well as the following for agp 8.0:

isDebuggable = false

I'm going ahead and closing this - if you've further questions please answer here and will reopen. Thanks.

ap...@google.com <ap...@google.com> #5Apr 21, 2021 01:47AM

The following release(s) address this bug.It is possible this bug has only been partially addressed:

androidx.benchmark:benchmark-baseline-profile-gradle-plugin:1.4.0-alpha04

cc...@google.com <cc...@google.com> Apr 28, 2021 12:03AM

Reassigned to cc...@google.com.

cc...@google.com <cc...@google.com> #6Apr 29, 2021 11:26PM

Since release signing config is used by default instead of debug is not mentioned in release notes - maybe this is a bug?

Cause the last time I found it mentioned in the release notes was in version 1.1

signingConfig.debug is used as the default signing config (b/153583269)

So, if the switch to the release one indeed happened - maybe it's an issue?

ca...@google.com <ca...@google.com> #7May 3, 2021 08:02PM

The new build types should inherit the certificates from the release build type, unless none is specified in which case debug is used: code link

This should be updated in the documentation, thanks for mentioning it. Will take care of updating it.

ap...@google.com <ap...@google.com> #8May 4, 2021 04:44PM

Project: platform/frameworks/support
Branch: androidx-main

commit 7fd221c0c8f7948351293a7fbf75eef0bf698a42
Author: Chris Craik <ccraik@google.com>
Date: Fri Apr 30 17:19:22 2021

Fix output issue on API 21, suppress still-broken tests

Bug: 183129298
Test: ./gradlew benchmark:b-c:cC benchmark:b-m:cC

Shell can't access externalCacheDir (but somehow can access cacheDir)
on API 21.

Also on API 21, UiDevice.executeShellCommand doesn't capture the
output of `echo foo`. executeShellScript still partially works
(e.g. `ls` works), but we'll need more investigation to better
understand shell command limitations on that API level.

Change-Id: I2bafe8218ed50ac54cd990190d2f9a2e2364028d

M benchmark/common/src/main/java/androidx/benchmark/Outputs.kt
A benchmark/macro/src/androidTest/java/androidx/benchmark/macro/OutputsTest.kt
M benchmark/macro/src/androidTest/java/androidx/benchmark/macro/perfetto/ShellUtilsTest.kt
M benchmark/macro/src/main/java/androidx/benchmark/macro/perfetto/ShellUtils.kt

https://android-review.googlesource.com/1692806

cc...@google.com <cc...@google.com> #9Jun 4, 2021 08:42PM

(Coming back to this after I/O) Carmen - filed b/190219056 to track the perfetto support for android_startup below API 29.

ap...@google.com <ap...@google.com> #10Jul 29, 2021 01:10AM

Project: platform/frameworks/support
Branch: androidx-main

commit 3dff7103380e8f0561029da051bd444d8bbcd26b
Author: Chris Craik <ccraik@google.com>
Date: Wed Jul 28 16:58:05 2021

Only use atrace tags supported on the local platform version

Bug: 183129298
Test: PerfettoConfigTest
Test: AtraceTagTest

Codifies list of in-use atrace tags with which platform versions
they're supported on. Validated on API 21 -> 29 emulators.

Currently, the way we invoke perfetto doesn't tell us if starting the
process failed, so this eliminates one source of this failure.

Planned follow up work
- investigate shell wrapper issues on API 21/22.
- improve perfetto launch failures to throw clearly
- add additional config validation
- (low prior) stash shell session rooted status

Change-Id: I99f5db15d1d0ee84d340b6879ef363f41c4327b5

M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkScopeTest.kt
A benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/AtraceTagTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoConfigTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/ShellUtilsTest.kt
A benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/AtraceTag.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoCapture.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoConfig.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/ShellUtils.kt

https://android-review.googlesource.com/1780588

ap...@google.com <ap...@google.com> #11Jul 29, 2021 05:33PM

Project: platform/frameworks/support
Branch: androidx-main

commit 48ae41ec2aada7a167afbd9e03ce67e9114aed23
Author: Chris Craik <ccraik@google.com>
Date: Thu Jul 29 09:43:38 2021

Fix ShellUtils on API 21/22 - chmod +x not supported

Bug: 183129298
Test: ShellUtilsTest # verified on API 21 through 29

Change-Id: I8f235b8044fb54f9f23129a62e8a732536900add

M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/ShellUtilsTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/ShellUtils.kt

https://android-review.googlesource.com/1782028

ap...@google.com <ap...@google.com> #12Jul 30, 2021 12:55AM

Project: platform/frameworks/support
Branch: androidx-main

commit e137997eab441062cc59ddaec31cb6609cd47039
Author: Chris Craik <ccraik@google.com>
Date: Thu Jul 29 10:21:32 2021

Validate PerfettoTraceProcessor back to API 21

Bug: 183129298
Test: PerfettoTraceProcessor # validated on API 21 through 26

Also adds fixed-trace query test.

Change-Id: I1a47f77c218d1918bd2273e3f6b1f43451f6ac76

M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessorTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessor.kt

https://android-review.googlesource.com/1782029

ap...@google.com <ap...@google.com> #13Jul 31, 2021 12:06AM

Project: platform/frameworks/support
Branch: androidx-main

commit c75fd5f903d15795ab43417debd86822c21bcca4
Author: Chris Craik <ccraik@google.com>
Date: Thu Jul 29 17:55:05 2021

Test trace capture back to API 28, macrobench scope to 27

Test: PerfettoCaptureTest
Test: MacrobenchmarkScopeTest
Bug: 183129298

Clarify with comment that perfetto support for O is blocking lowering,
as config doesn't appear to be the issue currently.

Throw if trying to use bundled version before 29, since its behavior
is unpredictable (passes on local Pixel2 API 28, but not Pixel1 API 28
in CI)

Additionally, switched bench:integ-tests:macrobenchmark to minSdk=29,
since 28 won't capture metrics yet.

Change-Id: I087c4b812df3a223ed239062d35a9b731cab39e7

M benchmark/benchmark-macro/build.gradle
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkScopeTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoCaptureTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoHelper.kt
M benchmark/integration-tests/macrobenchmark/build.gradle
M benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/TrivialListScrollBenchmark.kt

https://android-review.googlesource.com/1780831

ap...@google.com <ap...@google.com> #14Aug 2, 2021 09:08PM

Project: platform/frameworks/support
Branch: androidx-main

commit eeb028cd24aee126f6f89a59df531304e57ad937
Author: Chris Craik <ccraik@google.com>
Date: Fri Jul 30 15:30:20 2021

Lower MacrobenchmarkScope min API to 21

Test: MacrobenchmarkScopeTest
Test: ShellUtilsTest
Test: ShellBehaviorTest
Bug: 183129298

Fixes usage of ps -A and pidof.

Started documenting shell version inconsistencies with
ShellBehaviorTest, to be sure we understand which behavior is where,
as testing (local especially) will always be limited, and it's good to
understand these behaviors as best as possible.

Note that low-API level testing has been limited rooted devices so
far, as that's what's available as emulators. Additional fixes likely
needed for user builds of these platform versions.

Change-Id: I7a5243a0570a81b676c4adff2d8267a3b68b61ca

M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkScopeTest.kt
A benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/ShellBehaviorTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoCaptureTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/ShellUtilsTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Metric.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoHelper.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/ShellUtils.kt
M benchmark/integration-tests/macrobenchmark-target/src/main/java/androidx/benchmark/integration/macrobenchmark/target/NotExportedActivity.kt

https://android-review.googlesource.com/1783410

cc...@google.com <cc...@google.com> #15Aug 2, 2021 09:35PM

Removed b/184814381, as we have a workaround (mentioned in that bug).

cc...@google.com <cc...@google.com> Aug 3, 2021 07:54PM

Accepted by cc...@google.com.

ap...@google.com <ap...@google.com> #16Aug 3, 2021 09:47PM

Project: platform/frameworks/support
Branch: androidx-main

commit 872057a24acf988c6d43e8ff066ae3ce4c881768
Author: Chris Craik <ccraik@google.com>
Date: Tue Aug 03 12:17:49 2021

Fix app cmdline wildcard to only be used API 28+

Bug: 183129298
Test: ./gradlew benchmark:benchmark-macro:cC

Related improvements:

* Fixed trace processor query to use "OR" for multi-slice queries

* Expanded PerfettoCaptureTest to also test non-app tag (to enable
easily isolating app-tag tracing issues)

* Fixed crash in FileLinkingRule when run from gradle cmdline

Change-Id: Ia194d886834b7a1c1c24800cb102c6f8bab39684

M benchmark/benchmark-macro-junit4/src/main/java/androidx/benchmark/macro/junit4/PerfettoRule.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/FileLinkingRule.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/StartupTimingMetricTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoCaptureTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoConfigTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessorTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Macrobenchmark.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoCapture.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoCaptureWrapper.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoConfig.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessor.kt

https://android-review.googlesource.com/1785269

ap...@google.com <ap...@google.com> #17Aug 16, 2021 06:12PM

Project: platform/frameworks/support
Branch: androidx-main

commit 6a161d9d5b72a8921a87f728e69267ed476aa180
Author: Chris Craik <ccraik@google.com>
Date: Thu Aug 12 15:20:56 2021

Raise min API of benchmark-macro(-junit4) libraries to 21

Bug: 183129298

Test: ./gradlew benchmark:benchmark-common:cC benchmark:benchmark-macro:cC benchmark:benchmark-macro-junit4:cC benchmark:benchmark-junit4:cC

Relnote: "Raised min API to 21 to reflect intended lowest API level to
be supported in the future. Current min API supported continues to
be conveyed via RequiredApi(), and is currently 29"

Removes all SdkSuppress(21) and RequireApi(21) in macrobench, as
they're no longer needed. Also unifies OutputTest into
benchmark-common, as it should continue to using SdkSuppress(21).

Change-Id: I440d67683d41150fc9dd84d4f0a7100f2c9f32bd

M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/OutputsTest.kt
M benchmark/benchmark-macro-junit4/build.gradle
M benchmark/benchmark-macro/api/current.txt
M benchmark/benchmark-macro/api/public_plus_experimental_current.txt
M benchmark/benchmark-macro/api/restricted_current.txt
M benchmark/benchmark-macro/build.gradle
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/CompilationModeTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkScopeTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkTest.kt
D benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/OutputsTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessorTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/CompilationMode.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Macrobenchmark.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/MacrobenchmarkScope.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Metric.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessor.kt
M testutils/testutils-macrobenchmark/build.gradle

https://android-review.googlesource.com/1795170

ap...@google.com <ap...@google.com> #18Aug 25, 2021 05:23PM

Project: platform/frameworks/support
Branch: androidx-main

commit 4d383e006bca3302eac0729de693ddc1c94d17ac
Author: Chris Craik <ccraik@google.com>
Date: Mon Aug 23 17:45:50 2021

Enable perfetto capture testing back to API 21/23

Bug: 183129298
Test: ./gradlew benchmark:benchmark-common:cC benchmark:benchmark-macro:cC

While API 21 and 22 mostly work, UiState will fail to append to the
trace, due to a file access permission issue. We can investigate this
in a follow-up.

The major fix was generalizing how we detect package/executable names
in ps output to handle old platform behavior.

This change also:
- removes the no-longer-necessary socket naming args

- speculatively fixes a flake discovered in PerfettoCaptureTest
(reproing on older devices), where legit tracepoint usage would
cause the test to flake

Change-Id: Ib637f9ef30bf557f5b03198e0803f5e4fa32278c

M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/PerfettoHelperTest.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Shell.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoHelper.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoCaptureTest.kt

https://android-review.googlesource.com/1806638

ap...@google.com <ap...@google.com> #19Sep 10, 2021 02:42AM

Project: platform/frameworks/support
Branch: androidx-main

commit addb956b048ef75eb639935ce3c5de878456162e
Author: Chris Craik <ccraik@google.com>
Date: Wed Sep 08 15:33:28 2021

Trace-based FrameTimingMetric, Step 1

Bug: 195987329
Bug: 183129298
Test: TrivialListScrollBenchmark, FrameTimingQueryTest
Test: ./gradlew benchmark:benchmark-macro:cC

Initial step of moving FrameTimingMetric to be based on Perfetto trace
data. This has several advantages:

- Improved precision - floating point ms measurements

- Metric customization - e.g. deadline-aware metrics

- Metric clarification - enables labeling different metrics per API
level, as opposed to dumpsys which changes behavior across api
levels.

Currently, new metrics are private, opt-in per benchmark, and tagged
separately from other metrics for comparison in Studio/CI.

Planned next steps:

- Merge (new) frame timing metrics to be merged across
iterations. This will significantly simply output, as we can avoid
min/med/max and rely entirely on percentiles to show distribution.

- Move FrameTimingMetric itself to be trace based, with metric
emphasis depending on current platform version.

- Make macrobench no longer throw on API < 29 (initially, for
FrameTimingMetric benchmarks only)

- Go through this for startup as well

Change-Id: Ib07e51ea2e5822ed75187731530cf1f5a40b8e66

M benchmark/benchmark-common/src/main/java/androidx/benchmark/MetricResult.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoConfig.kt
M benchmark/benchmark-macro/api/restricted_current.txt
A benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/FrameTimingQueryTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessorTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Metric.kt
A benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/FrameTimingQuery.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessor.kt
A benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/Slice.kt
M benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/TrivialListScrollBenchmark.kt

https://android-review.googlesource.com/1822405

ap...@google.com <ap...@google.com> #20Sep 16, 2021 05:22PM

Project: platform/frameworks/support
Branch: androidx-main

commit 47fd60aaf2b811b6d7f1b4d40d2380d6c0f73fc9
Author: Chris Craik <ccraik@google.com>
Date: Wed Sep 15 16:10:00 2021

Trace-based FrameTimingMetric, Step 2

Bug: 195987329
Bug: 183129298
Fixes: 200082324
Test: TrivialListScrollBenchmark # Validated on 29 & 31
Test: ./gradlew benchmark:benchmark-common:cC benchmark:benchmark-macro:cC

Completes migration of FrameTimingMetric to be trace-based, and
surface FrameNegativeSlack when available.

This change also makes several related changes to frame metrics coming
from FrameTimingQuery:

* Deleted FrameTime, as it's always preferable to use something
deadline based

* Renamed BasicFrameTime -> FrameCpuTime for clarity, and to match metric plan

* Made FrameSlack negative, so the *worst* frames show up in 99P,
instead of best. This avoids having to differentate between low=bad
and high=bad metrics.

Adds FrameTimingGfxInfoMetric to existing scroll benchmarks for
temporary continuity in CI tracking.

Change-Id: Ied56278b1f76f37df44181dc736590753aa1cb30

M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Metric.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/FrameTimingQuery.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/FrameTimingQueryTest.kt
M benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/TrivialListScrollBenchmark.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/IdeSummaryString.kt
M compose/integration-tests/macrobenchmark/src/androidTest/java/androidx/compose/integration/macrobenchmark/TrivialListScrollBenchmark.kt
M compose/integration-tests/macrobenchmark/src/androidTest/java/androidx/compose/integration/macrobenchmark/NestedListsScrollBenchmark.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/IdeSummaryStringTest.kt
M benchmark/benchmark-macro/api/restricted_current.txt

https://android-review.googlesource.com/1828135

ap...@google.com <ap...@google.com> #21Sep 17, 2021 05:58PM

Project: platform/frameworks/support
Branch: androidx-main

commit 358637d914f7c6b6cfc3d442aaa4eee1c6bd611a
Author: Chris Craik <ccraik@google.com>
Date: Wed Sep 15 17:25:13 2021

Lower min macrobenchmark to API23 (scrolling benchmarks only)

Bug: 183129298
Test: TrivialListScrollBenchmark # validated on API25
Test: ./gradlew benchmark:benchmark-common:cC benchmark:benchmark-macro:cC # API 25
Relnote: Enable scrolling macrobenchmarks to run back to API 23

Lowers Macrobenchmark @RequiresApi -> 23. StartupTimingMetric still
requires 29, which is under investigation.

Change-Id: If39c203832b9aed765aa05d7573eb9c88bf1dc70

M compose/integration-tests/macrobenchmark/build.gradle
M benchmark/benchmark-macro/api/current.txt
M compose/integration-tests/macrobenchmark/src/androidTest/java/androidx/compose/integration/macrobenchmark/SmallListStartupBenchmark.kt
M compose/integration-tests/macrobenchmark/src/androidTest/java/androidx/compose/integration/macrobenchmark/NestedListsScrollBenchmark.kt
M benchmark/benchmark-macro/api/public_plus_experimental_current.txt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoCaptureWrapper.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Metric.kt
M benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/TrivialStartupBenchmark.kt
M compose/integration-tests/macrobenchmark/src/androidTest/java/androidx/compose/integration/macrobenchmark/IoSettingsStartupBenchmark.kt
M benchmark/integration-tests/macrobenchmark/build.gradle
M benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/SmallListStartupBenchmark.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkTest.kt
M benchmark/benchmark-macro-junit4/src/main/java/androidx/benchmark/macro/junit4/MacrobenchmarkRule.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Macrobenchmark.kt
M compose/integration-tests/macrobenchmark/src/androidTest/java/androidx/compose/integration/macrobenchmark/TrivialStartupBenchmark.kt
M benchmark/benchmark-macro-junit4/api/current.txt
M benchmark/benchmark-macro-junit4/api/restricted_current.txt
M benchmark/benchmark-macro-junit4/api/public_plus_experimental_current.txt
M benchmark/benchmark-macro/api/restricted_current.txt

https://android-review.googlesource.com/1828427

ap...@google.com <ap...@google.com> #22Oct 11, 2021 10:07PM

Project: platform/frameworks/support
Branch: androidx-main

commit c155f6c5c18095c77e02f25e2c35179d225d2cf1
Author: Chris Craik <ccraik@google.com>
Date: Mon Oct 04 09:53:10 2021

Initial impl of StartupTimingQuery, part of new startup metric

Bug: 183129298
Bug: 190219056
Test: ./gradlew benchmark:benchmark-common:cC benchmark:benchmark-macro:cC

Tested only on API 31 so far, as there are issues with capturing
traces (with app trace tags enabled) on older platform versions.

Adds check for atrace args to be too long on older platform versions,
which was causing problems when attempting to test on API 23.

Known compatibility issues for lowering min API (e.g. 'launching'
missing for hot launches on certain old versions) will be handled in
follow up work.

Change-Id: I2eb4700f2c7cbc822ef5a23da7b97192da0ee177

M benchmark/integration-tests/macrobenchmark-target/build.gradle
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/CompilationMode.kt
A benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/TrivialStartupFullyDrawnBenchmark.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoConfig.kt
A benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/StartupTimingQueryTest.kt
M benchmark/benchmark-junit4/src/main/java/androidx/benchmark/junit4/PerfettoRule.kt
A benchmark/integration-tests/macrobenchmark-target/src/main/java/androidx/benchmark/integration/macrobenchmark/target/TrivialStartupFullyDrawnActivity.kt
M benchmark/integration-tests/macrobenchmark-target/src/main/res/values/donottranslate-strings.xml
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Macrobenchmark.kt
M benchmark/integration-tests/macrobenchmark-target/src/main/AndroidManifest.xml
A benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/StartupTimingQuery.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/perfetto/PerfettoConfigTest.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Shell.kt

https://android-review.googlesource.com/1850510

ap...@google.com <ap...@google.com> #23Oct 12, 2021 09:29PM

Project: platform/frameworks/support
Branch: androidx-main

commit b5b2d530418de72c6b8f7420c86e78e6efb19c4f
Author: Chris Craik <ccraik@google.com>
Date: Mon Oct 11 23:02:44 2021

Lower min api tested for StartupTimingQuery to API 24

Bug: 183129298
Bug: 190219056
Test: StartupTimingQueryTest
Relnote: N/A

Adds support for hot launches pre-api 29, which weren't traced. We do
this with a very simple approximation for now, measuring from
activityresume to end of the next DrawFrame.

Additionally, adds tracing wrapper with fallback reflection for the
single startup-critical trace section, as it factors into
StartupTimingQuery.

This accounts for the only known platform trace section content
differences going back to intended min support level of 21.

This does not solve other compatibility issues on these old devices,
such as missing trace content, which needs to be investigated
separately. For creating the sample traces, `Thread.sleep()` and
retrying when missing content were both used.

Change-Id: Ief6dd7eff6ab90607df1088adf456c89921a55bb

M benchmark/benchmark-macro/api/current.txt
M benchmark/benchmark-macro/api/restricted_1.1.0-beta01.txt
A benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/ForceTracing.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoConfig.kt
M benchmark/benchmark-macro/api/public_plus_experimental_1.1.0-beta01.txt
M benchmark/benchmark-macro/api/1.1.0-beta01.txt
M benchmark/benchmark-macro/api/public_plus_experimental_current.txt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/StartupTimingQueryTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/MacrobenchmarkScope.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Macrobenchmark.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/StartupTimingQuery.kt
M benchmark/benchmark-macro/api/restricted_current.txt

https://android-review.googlesource.com/1853079

ap...@google.com <ap...@google.com> #24Oct 14, 2021 07:52PM

Project: platform/frameworks/support
Branch: androidx-main

commit 14225e715dfe78937377f70d2dc786d69a70f35c
Author: Chris Craik <ccraik@google.com>
Date: Wed Oct 13 17:34:52 2021

Switch StartupTimingMetric to custom trace-based implementation

Bug: 183129298
Bug: 190219056
Test: TrivialStartupBenchmark # macrobench
Test: ./gradlew benchmark:benchmark-common:cC benchmark:benchmark-macro:cC

Relnote: """Backport StartupTimingMetric to work back to API 23. This
new implementation also better handles reportFullyDrawn() to wait
until corresponding content has been rendered."""

Switch StartupTimingMetric to custom trace metric. This new version:

- Is validated against the old in a few specific cases on newer devices

- Supports old API levels (where trace section names were different, and sometimes not present)

- Fixes reportFullyDrawn to not define the fully drawn point
immediately, and instead wait until rendering completes.

Change-Id: If3ac9a339bedbb61785c80ea78c0e7ae8f9018e9

M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessorTest.kt
M benchmark/benchmark-macro/api/current.txt
M benchmark/benchmark-macro/api/restricted_1.1.0-beta01.txt
M benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/TrivialStartupFullyDrawnBenchmark.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/ForceTracing.kt
M compose/integration-tests/macrobenchmark/src/androidTest/java/androidx/compose/integration/macrobenchmark/SmallListStartupBenchmark.kt
M testutils/testutils-macrobenchmark/src/main/java/androidx/testutils/MacrobenchUtils.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/perfetto/UiStateTest.kt
M compose/integration-tests/macrobenchmark/src/androidTest/java/androidx/compose/integration/macrobenchmark/NestedListsScrollBenchmark.kt
M benchmark/benchmark-macro/api/public_plus_experimental_1.1.0-beta01.txt
M benchmark/benchmark-macro/api/1.1.0-beta01.txt
M benchmark/benchmark-macro/api/public_plus_experimental_current.txt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/StartupTimingQueryTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Metric.kt
M benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/TrivialStartupBenchmark.kt
M compose/integration-tests/macrobenchmark/src/androidTest/java/androidx/compose/integration/macrobenchmark/IoSettingsStartupBenchmark.kt
M benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/SmallListStartupBenchmark.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Macrobenchmark.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/StartupTimingMetricTest.kt
M compose/integration-tests/macrobenchmark/src/androidTest/java/androidx/compose/integration/macrobenchmark/TrivialStartupBenchmark.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/IterationResult.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/StartupTimingQuery.kt
M benchmark/benchmark-macro/api/restricted_current.txt

https://android-review.googlesource.com/1857375

cc...@google.com <cc...@google.com> #25Oct 28, 2021 07:49PM

Marked as fixed.

Fixed with https://android-review.googlesource.com/c/platform/frameworks/support/+/1861673

As API 21 doesn't have tracing of Choreographer#doFrame, we've decided to only support API 23 as a minimum for now. If there's need for it, we can file a new bug and investigate our options down the line.