IssueTracker

Thanks for your detailed post.

However, benchmark build type is under configured: at least isProfileable is not set to true for existing build type, probably there's more.

We should set isProfileable = true by default when overriding an existing benchmark build type.

This issue is not about some specific configuration flag, but the general approach of dealing with external configuration. As a developer adopting baseline profiles, it seems extremely risky to me using a custom configuration due to how it's applied under the hood and the fact it may break default configuration.

I agree that is not great but this is a little tricky to do. For custom baseline profile build types we override all the properties. For benchmark I left it open to configure but it's mostly about these 2 properties:

isMinifyEnabled
isShrinkResources

I don't have a way to see if the user is setting them before overriding, so for this reason, I'd prefer not to. I agree with you that some other properties could be set by default to make this easier, i.e.:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

The reason why I mentioned the release signing config in the beginning is because I want to use debug signing config.

In the specific of your issue, i.e. using a debug certificate can you override the benchmark and baseline profile setting? You should be able to do something like:

android {
    buildTypes {
        release { ... }
        debug { ... }
        benchmarkRelease {
            ...
            signingConfig signingConfigs.debug
        }
        nonMinifiedRelease {
            ...
            signingConfig signingConfigs.debug
        }
    }
}

Project: platform/frameworks/support
Branch: androidx-main
Author: Marcello Albano <maralb@google.com>
Link:      https://android-review.googlesource.com/3303459

Added override for debuggable and profileable for benchmark builds in bpgp

Added override for debuggable and profileable for benchmark builds in bpgp

Test: ./gradlew :benchmark:benchmark-baseline-profile-gradle-plugin:test
Bug: 369213505
Relnote: "isProfileable is always overridden in benchmark builds,
  and isDebuggable is also now always overridden in both benchmark and
  nonMinified (baseline profile capture) builds."
Change-Id: I487fa71083921682173f04fcbb477be5baf165f8

I've landed a change that will set the following properties also when the benchmark build type already exists:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

As well as the following for agp 8.0:

isDebuggable = false

I'm going ahead and closing this - if you've further questions please answer here and will reopen. Thanks.

The following release(s) address this bug.It is possible this bug has only been partially addressed:

• androidx.benchmark:benchmark-baseline-profile-gradle-plugin:1.4.0-alpha04

Since release signing config is used by default instead of debug is not mentioned in release notes - maybe this is a bug?

Cause the last time I found it mentioned in the release notes was in version 1.1

signingConfig.debug is used as the default signing config (b/153583269)

So, if the switch to the release one indeed happened - maybe it's an issue?

The new build types should inherit the certificates from the release build type, unless none is specified in which case debug is used: code link

This should be updated in the documentation, thanks for mentioning it. Will take care of updating it.

I think I've been able to reproduce the issue locally - on a Pixel 2 API 29, I've seen cold startup benchmarks specifically, fail, though they have a different stack:

02-11 11:37:32.531 13395 13429 E TestRunner: java.util.NoSuchElementException: Collection contains no element matching the predicate.
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.macro.perfetto.StartupTimingQuery.findEndRenderTimeForUiFrame(StartupTimingQuery.kt:202)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.macro.perfetto.StartupTimingQuery.getFrameSubMetrics(StartupTimingQuery.kt:171)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.macro.StartupTimingMetric.getMetrics$benchmark_macro_debug(Metric.kt:275)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:208)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:300)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:102)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated$default(MacrobenchmarkRule.kt:92)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.testutils.MacrobenchUtilsKt.measureStartup(MacrobenchUtils.kt:81)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.testutils.MacrobenchUtilsKt.measureStartup$default(MacrobenchUtils.kt:75)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.integration.macrobenchmark.TrivialStartupBenchmark.startup(TrivialStartupBenchmark.kt:40)
...

Traces attached from the same benchmark run - iter 0 has the process named correctly, iter 1 does not - Perfetto is using the shortened process name. +Lalit - we're using bundled perfetto in this case.

We previously supported truncated process names in our queries, but thought we'd fixed all the cases where they occur as part of b/204789927. In that case though, we hadn't seen them on API 29.

Lalit - is the problem here that we're polling too infrequently? We're polling every 10 seconds, and this is a test that kills the process before tracing is started, and which only lasts a couple seconds (since it's so trivial). (Full logcat from this local run: https://paste.googleplex.com/6709539849109504)

Re #comment7 - try opening in Perfetto Web viewer Canary. Current stable web version is a bit behind, so I expect that's the issue. If that doesn't work, suggest filing a bug to https://github.com/google/perfetto/issues with the trace.

Running TrivialStartupBenchmark, COLD only. Pixel 2, 4 parameterized tests, 10 iters each.

API 29 Bundled perfetto:
            proc_stats_poll_ms = 10000, // 4 failures
            proc_stats_poll_ms = 1000,  // 4 failures
            proc_stats_poll_ms = 500,   // 3 failures
            proc_stats_poll_ms = 100,   // 0 failures
API 29 Unbundled perfetto:
            proc_stats_poll_ms = 10000, // 0 failures
API 30 Bundled perfetto:
            proc_stats_poll_ms = 10000, // 0 failures

Looks like proc_stats_poll_ms is the problem when using bundled perfetto, and that scan_all_processes_on_start = true isn't supported on the bundled copy until API 30.

Will put up a CL to use unbundled perfetto on API 29, and avoid throwing in this case.

The problem is not actually that scan_all_processes_on_start = true is not supported - scan_all_processes_on_start was added in P (API 29) - see aosp/675862.

I looked a bit into the traces you linked Chris and it seems like for some reason, we do not scan the pid for the startup process on demand - I tried to do some git archeology to find which CL between Q and R fixed the issue but didn't get anywhere - since you've found a workaround for this anyway, I didn't look too much further.

This fix should be available in beta04 next week.

Note that the fix assumes that the problem only occurs on Android 10 / API 29. If you have seen this problem on any other os version, please comment with the device, os version, and the trace from the failing iteration.

Version: 1.1.0-beta03

Device: Pixel 6 (API 31)

The same problem hapenned running the startup benchmarks on iosched.

It happens for StartupMode.WARM, for COLD and HOT it succeeds.

The problem in #comment14 is a separate problem from what was fixed above (Cold startups on Android 10). The above fix was about process names not showing up correctly. In these warm startup traces, the process is named correctly, but we don't capture any renderthread work, or frames at all.

Tomas, can you try working around by adding a Thread.sleep(500) and seeing if that solves the problem? Either the frames are being produced, and am start -W isn't waiting for them (my guess), or the frame isn't getting rendered (much less likely).

Thread.sleep(500) helps.

Tried 5x 3iterations

• without sleep -> fails every time
• with sleep -> success every time

Can you try and repro on API 30, and attach a few success traces?

I'll put up a workaround, and pass to Rahul to file a platform bug.

It's super weird this only affects warm startup, and not hot/cold.

Passing over to Rahul.

I started a WIP CL for this, but found I wasn't able to repro the behavior in #comment14 on my API 32 device - the opposite actually :|

Left some ideas for next investigation steps in that CL:

https://android-review.googlesource.com/c/platform/frameworks/support/+/1991372

If am start -w is still unreliable, we can consider something more complex to detect end of frame after startup, like polling dumpsys gfxinfo <pkg> report: https://developer.android.com/studio/command-line/dumpsys#ui

The WIP CL is: https://android-review.googlesource.com/c/platform/frameworks/support/+/1991372/1/benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/MacrobenchmarkScope.kt#140 for reference.

I can't reproduce this on API 32 for iosched.

StartupBenchmark_startup[mode=COLD]
  timeToInitialDisplayMs   min 285.0,   median 288.6,   max 296.9
    Traces: Iteration 0 1 2

StartupBenchmark_startup[mode=WARM]
  timeToInitialDisplayMs   min 84.2,   median 85.0,   max 89.2
    Traces: Iteration 0 1 2

StartupBenchmark_startup[mode=HOT]
  timeToInitialDisplayMs   min 44.7,   median 53.1,   max 70.0
    Traces: Iteration 0 1 2

This is using 1.1.0-beta04.

Hm, so I tested the 1.1.0-beta04 on user builds on:

• SM-F711B (API 30)
• Pixel3 (API 31)
• Pixel6 (API 32)

It always fails for WARM on Pixel3 (API 31).

If I uninstall the target app from the devices (and let the tests install it), it even fails on the Pixel6 (API 32) for HOT.

https://android-review.googlesource.com/c/platform/frameworks/support/+/2012227 is an attempt to fix this.

mlykotom@ can you try reproducing this with this fix ? I can help you create a local maven artifact if that helps.

Can you try using ab/8275593 once its available (on androidx.dev) to see if you can still reproduce this issue ?

The Pixel6 looks like fixed, but Pixel 3 Android 12 with StartupMode.WARM still fails, but I found out it always happens for the last iteration.

E.g. if I specify iterations=7, it will fail when running the 8th time (because one is for warmup).

mlykotom@ can you add a device.waitForIdle() in the measure block and see if that helps ?

The CL as written doesn't do anything for warm startups, since the process is already alive.

Given how flaky this wait has been, I'm just had the thought we can rewrite this await behavior using gfxinfo to see if a frame has been produced. Will need to handle named subprocesses correctly though.

Rahul, can you file a big to the platform with the repro from your CL? am start -w should definitely wait for the process to come alive (and really, for the first frame too).

Filed a platform bug at https://b.corp.google.com/issues/226179160

The remaining problem with startups not being detected should be solved by https://android-review.googlesource.com/c/platform/frameworks/support/+/2037359, just merged, though unfortunately didn't make beta05. This change adds a backup method to confirm launches have completed.

If anyone hitting this problem still sees it in beta05 (with the workaround in #comment24), you can try a snapshot of macrobenchmark via androidx.dev with the overhaul, just use 1.2.0-SNAPSHOT with build 8353149 or later (once a snapshot build catches up).

Last patch fixes startup benchmark, but has caused failures in the multiprocess benchmark, as well as all scrolling benchmarks since the launch pattern isn't consistent with startup mode. Working on these.

Above issues should be fixed with https://android-review.googlesource.com/c/platform/frameworks/support/+/2048353/

Verified test fixes with forrest runs: https://android-build.googleplex.com/builds/abtd/run/L06800000953800226 and https://android-build.googleplex.com/builds/abtd/run/L95100000953796505 (Note API 29 cancellations are b/226977838)