IssueTracker

Can someone hitting this error please attach the .perfetto-trace files for the given failing test? The path is printed in logcat, if they aren't copied to the host already.

Is this only reproducing on beta03?

I think I've been able to reproduce the issue locally - on a Pixel 2 API 29, I've seen cold startup benchmarks specifically, fail, though they have a different stack:

02-11 11:37:32.531 13395 13429 E TestRunner: java.util.NoSuchElementException: Collection contains no element matching the predicate.
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.macro.perfetto.StartupTimingQuery.findEndRenderTimeForUiFrame(StartupTimingQuery.kt:202)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.macro.perfetto.StartupTimingQuery.getFrameSubMetrics(StartupTimingQuery.kt:171)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.macro.StartupTimingMetric.getMetrics$benchmark_macro_debug(Metric.kt:275)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:208)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:300)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:102)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated$default(MacrobenchmarkRule.kt:92)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.testutils.MacrobenchUtilsKt.measureStartup(MacrobenchUtils.kt:81)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.testutils.MacrobenchUtilsKt.measureStartup$default(MacrobenchUtils.kt:75)
02-11 11:37:32.531 13395 13429 E TestRunner: 	at androidx.benchmark.integration.macrobenchmark.TrivialStartupBenchmark.startup(TrivialStartupBenchmark.kt:40)
...

Traces attached from the same benchmark run - iter 0 has the process named correctly, iter 1 does not - Perfetto is using the shortened process name. +Lalit - we're using bundled perfetto in this case.

We previously supported truncated process names in our queries, but thought we'd fixed all the cases where they occur as part of b/204789927. In that case though, we hadn't seen them on API 29.

Lalit - is the problem here that we're polling too infrequently? We're polling every 10 seconds, and this is a test that kills the process before tracing is started, and which only lasts a couple seconds (since it's so trivial). (Full logcat from this local run: https://paste.googleplex.com/6709539849109504)

Re #comment7 - try opening in Perfetto Web viewer Canary. Current stable web version is a bit behind, so I expect that's the issue. If that doesn't work, suggest filing a bug to https://github.com/google/perfetto/issues with the trace.

Running TrivialStartupBenchmark, COLD only. Pixel 2, 4 parameterized tests, 10 iters each.

API 29 Bundled perfetto:
            proc_stats_poll_ms = 10000, // 4 failures
            proc_stats_poll_ms = 1000,  // 4 failures
            proc_stats_poll_ms = 500,   // 3 failures
            proc_stats_poll_ms = 100,   // 0 failures
API 29 Unbundled perfetto:
            proc_stats_poll_ms = 10000, // 0 failures
API 30 Bundled perfetto:
            proc_stats_poll_ms = 10000, // 0 failures

Looks like proc_stats_poll_ms is the problem when using bundled perfetto, and that scan_all_processes_on_start = true isn't supported on the bundled copy until API 30.

Will put up a CL to use unbundled perfetto on API 29, and avoid throwing in this case.

The problem is not actually that scan_all_processes_on_start = true is not supported - scan_all_processes_on_start was added in P (API 29) - see aosp/675862.

I looked a bit into the traces you linked Chris and it seems like for some reason, we do not scan the pid for the startup process on demand - I tried to do some git archeology to find which CL between Q and R fixed the issue but didn't get anywhere - since you've found a workaround for this anyway, I didn't look too much further.

This fix should be available in beta04 next week.

Note that the fix assumes that the problem only occurs on Android 10 / API 29. If you have seen this problem on any other os version, please comment with the device, os version, and the trace from the failing iteration.

Version: 1.1.0-beta03

Device: Pixel 6 (API 31)

The same problem hapenned running the startup benchmarks on iosched.

It happens for StartupMode.WARM, for COLD and HOT it succeeds.

The problem in #comment14 is a separate problem from what was fixed above (Cold startups on Android 10). The above fix was about process names not showing up correctly. In these warm startup traces, the process is named correctly, but we don't capture any renderthread work, or frames at all.

Tomas, can you try working around by adding a Thread.sleep(500) and seeing if that solves the problem? Either the frames are being produced, and am start -W isn't waiting for them (my guess), or the frame isn't getting rendered (much less likely).

Thread.sleep(500) helps.

Tried 5x 3iterations

• without sleep -> fails every time
• with sleep -> success every time

Can you try and repro on API 30, and attach a few success traces?

I'll put up a workaround, and pass to Rahul to file a platform bug.

It's super weird this only affects warm startup, and not hot/cold.

Passing over to Rahul.

I started a WIP CL for this, but found I wasn't able to repro the behavior in #comment14 on my API 32 device - the opposite actually :|

Left some ideas for next investigation steps in that CL:

https://android-review.googlesource.com/c/platform/frameworks/support/+/1991372

If am start -w is still unreliable, we can consider something more complex to detect end of frame after startup, like polling dumpsys gfxinfo <pkg> report: https://developer.android.com/studio/command-line/dumpsys#ui

The WIP CL is: https://android-review.googlesource.com/c/platform/frameworks/support/+/1991372/1/benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/MacrobenchmarkScope.kt#140 for reference.

I can't reproduce this on API 32 for iosched.

StartupBenchmark_startup[mode=COLD]
  timeToInitialDisplayMs   min 285.0,   median 288.6,   max 296.9
    Traces: Iteration 0 1 2

StartupBenchmark_startup[mode=WARM]
  timeToInitialDisplayMs   min 84.2,   median 85.0,   max 89.2
    Traces: Iteration 0 1 2

StartupBenchmark_startup[mode=HOT]
  timeToInitialDisplayMs   min 44.7,   median 53.1,   max 70.0
    Traces: Iteration 0 1 2

This is using 1.1.0-beta04.

Hm, so I tested the 1.1.0-beta04 on user builds on:

• SM-F711B (API 30)
• Pixel3 (API 31)
• Pixel6 (API 32)

It always fails for WARM on Pixel3 (API 31).

If I uninstall the target app from the devices (and let the tests install it), it even fails on the Pixel6 (API 32) for HOT.

https://android-review.googlesource.com/c/platform/frameworks/support/+/2012227 is an attempt to fix this.

mlykotom@ can you try reproducing this with this fix ? I can help you create a local maven artifact if that helps.

Can you try using ab/8275593 once its available (on androidx.dev) to see if you can still reproduce this issue ?

The Pixel6 looks like fixed, but Pixel 3 Android 12 with StartupMode.WARM still fails, but I found out it always happens for the last iteration.

E.g. if I specify iterations=7, it will fail when running the 8th time (because one is for warmup).

mlykotom@ can you add a device.waitForIdle() in the measure block and see if that helps ?

The CL as written doesn't do anything for warm startups, since the process is already alive.

Given how flaky this wait has been, I'm just had the thought we can rewrite this await behavior using gfxinfo to see if a frame has been produced. Will need to handle named subprocesses correctly though.

Rahul, can you file a big to the platform with the repro from your CL? am start -w should definitely wait for the process to come alive (and really, for the first frame too).

Filed a platform bug at https://b.corp.google.com/issues/226179160

The remaining problem with startups not being detected should be solved by https://android-review.googlesource.com/c/platform/frameworks/support/+/2037359, just merged, though unfortunately didn't make beta05. This change adds a backup method to confirm launches have completed.

If anyone hitting this problem still sees it in beta05 (with the workaround in #comment24), you can try a snapshot of macrobenchmark via androidx.dev with the overhaul, just use 1.2.0-SNAPSHOT with build 8353149 or later (once a snapshot build catches up).

Last patch fixes startup benchmark, but has caused failures in the multiprocess benchmark, as well as all scrolling benchmarks since the launch pattern isn't consistent with startup mode. Working on these.

Above issues should be fixed with https://android-review.googlesource.com/c/platform/frameworks/support/+/2048353/

Verified test fixes with forrest runs: https://android-build.googleplex.com/builds/abtd/run/L06800000953800226 and https://android-build.googleplex.com/builds/abtd/run/L95100000953796505 (Note API 29 cancellations are b/226977838)