IssueTracker

Can you experiment to see what a necessary value for the kill process delay is? 10000 is 10 seconds, and is a very long time to wait.

Happy to make a change to poll to wait for the kill to succeed, but I want to know the rough range of time we're expecting here to better understand what's happening. Using a shorter time will also let your macrobenchmarks run faster with the workaround.

Project: platform/frameworks/support
Branch: androidx-main
Author: Chris Craik <ccraik@google.com>
Link:      https://android-review.googlesource.com/3302738

Improve scope.killProcess with validation, and generalize killProcessesAndWait

Improve scope.killProcess with validation, and generalize killProcessesAndWait

Test: ShellTest
Test: MacrobenchmarkScopeTest
Test: TrivialStartupBenchmark # new:24 vs old:23 seconds for 10 iterations, slowdown from exra shell commands not drastic
Bug: 351582215

Relnote: "Speculative fix for errors from `StartupMode.COLD`
    exceptions: `Package <packagename> must not be running prior to cold
    start!`. Now, `MacrobenchmarkScope.killProcess()` (including the one
    run before each iteration, used to implement `StartupMode.COLD`
    behavior) will wait to verify that the app's processes have all
    stopped running."

Change-Id: I60aa6669366286e7275c2debcda7221c78165659

Hello!

Can we reopen this issue to get a proper solution please? We are still getting this exception when running macro benchmark tests. We tried every combination possible of latest benchmark and profile installer versions:

• Benchmark 1.3.1 and 1.4.0-alpha06
• Profile installer 1.3.1 and 1.4.1

We tested with Android 14 and 15 devices while applying both solutions stated here and here with no success.

Reopened. The speculative fix (with the retry until process is confirmed killed) went into 1.4.0-alpha03, so if you're seeing this on 1.4.0-alpha06 you'd have that.

We tested with Android 14 and 15 devices while applying both solutions stated here and here with no success.

1. Do you mean that even if you use testInstrumentationRunnerArguments["androidx.benchmark.killProcessDelayMillis"] = "10000" the problem still occurs? Jist want to confirm that's what you've tried.
2. Is there perhaps anything your app is doing to be resilient to kills or otherwise come back? disreguard, see you answered above
3. Does this reproduce for you with the sample project/app? (if it's app specific and I can repro on a production app, I can try that)
4. If it happens for the github sample, have you tried it on different devices (emulators as well as physical devices)?

I want to narrow down where this occurs so we can try and reproduce. If it happens with a prod app of yours, that also helps us take a look.

Thanks for reopening!

Do you mean that even if you use testInstrumentationRunnerArguments["androidx.benchmark.killProcessDelayMillis"] = "10000" the problem still occurs? Jist want to confirm that's what you've tried.

That is correct. We tried that, didn't seem to work for these cases.

Does this reproduce for you with the sample project/app?

I wasn't able to repro it with the sample app. Seems to be specific to our app/tests. Although I noticed tests are pretty simple there, only pressHome on setup for example... On our setup steps we need to login (only the first time) before proceeding to the actual benchmarking iterations.

That being said... We found a way of making it work (for now) by having the following:

• Benchmark updated to 1.4.0-alpha06
• At the end of the setup part of our COLD start tests both pressHome() and the solution stated here

Are you able to provide a standalone isolated repro project or steps that works with the your production application? Note that you can change the sample on github to specify your packagename - as long as the package is installed when the test runs, that should make it easier to isolate.

If I can reproduce locally, I can dig in further.

Macrobenchmark 1.1.1 didn't check if the process kill failed - it would just measure startup times incorrectly if the app process didn't die. The fix was to kill the process after the setupBlock and check for success which landed in 1.2.0-alpha07.

As a workaround to mimic the old (incorrect!) measurement behavior, you could implement your own cold startup by passing startupMode = null, and adding the following to your setup block to attempt a cold startup, but without the success check:

    dropShaderCache()
    killProcess()
    dropKernelPageCache()

This will bypass the exception, but similar to 1.1.1, doesn't actually give you a correct measurement.

If you can consistently get an app into this state, please let me know if you find a way to workaround it (killing the app in some other way, waiting in some other way) would be happy to hear about it.

One problem we found with killProcess() is that it essentially calls killall $packageName, which doesn't work if the process name is different than the package name (our case).

We ended up with a custom setupBlock { } that calls am force-stop $packageName, which reliably kills all processes of the given package at the start of each iteration:

Runtime.getRuntime().exec("am force-stop $packageName\n").waitFor()

It works like a charm, the tests are now reliable and consistent. However, it requires a couple of protected permissions: android.permission.FORCE_STOP_PACKAGES and android.permission.INTERACT_ACROSS_USERS_FULL, which is likely a deal breaker for many.

I am using Firebase Test Lab to execute performance tests on device bluejay,version=32, lib version 1.4.0-alpha06, and face the same issue. Specifically, I attempted to mitigate this by using the following configuration: testInstrumentationRunnerArguments["androidx.benchmark.killProcessDelayMillis"] = "10000" and noticed that I get a TestRunner exception in 50-100 ms after the process kill is initiated, but I expected it to happen in 10_000 ms.

- 13:57:15.732 26121 26140 D Benchmark: Killing process dif.tech.plata.perf
...
- 13:57:15.874 26121 26140 E TestRunner: java.lang.IllegalStateException: Package dif.tech.plata.perf must not be running prior to cold start!

Could the root cause of this issue be that the arguments weren't passed to this scope correctly, or that Thread.sleep isn't functioning as expected?

1.4.0-alpha07

For your information:

it helped to set pressHome(10_000) in setupBlock