IssueTracker

Thanks for your detailed post.

However, benchmark build type is under configured: at least isProfileable is not set to true for existing build type, probably there's more.

We should set isProfileable = true by default when overriding an existing benchmark build type.

This issue is not about some specific configuration flag, but the general approach of dealing with external configuration. As a developer adopting baseline profiles, it seems extremely risky to me using a custom configuration due to how it's applied under the hood and the fact it may break default configuration.

I agree that is not great but this is a little tricky to do. For custom baseline profile build types we override all the properties. For benchmark I left it open to configure but it's mostly about these 2 properties:

isMinifyEnabled
isShrinkResources

I don't have a way to see if the user is setting them before overriding, so for this reason, I'd prefer not to. I agree with you that some other properties could be set by default to make this easier, i.e.:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

The reason why I mentioned the release signing config in the beginning is because I want to use debug signing config.

In the specific of your issue, i.e. using a debug certificate can you override the benchmark and baseline profile setting? You should be able to do something like:

android {
    buildTypes {
        release { ... }
        debug { ... }
        benchmarkRelease {
            ...
            signingConfig signingConfigs.debug
        }
        nonMinifiedRelease {
            ...
            signingConfig signingConfigs.debug
        }
    }
}

Project: platform/frameworks/support
Branch: androidx-main
Author: Marcello Albano <maralb@google.com>
Link:      https://android-review.googlesource.com/3303459

Added override for debuggable and profileable for benchmark builds in bpgp

Added override for debuggable and profileable for benchmark builds in bpgp

Test: ./gradlew :benchmark:benchmark-baseline-profile-gradle-plugin:test
Bug: 369213505
Relnote: "isProfileable is always overridden in benchmark builds,
  and isDebuggable is also now always overridden in both benchmark and
  nonMinified (baseline profile capture) builds."
Change-Id: I487fa71083921682173f04fcbb477be5baf165f8

I've landed a change that will set the following properties also when the benchmark build type already exists:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

As well as the following for agp 8.0:

isDebuggable = false

I'm going ahead and closing this - if you've further questions please answer here and will reopen. Thanks.

The following release(s) address this bug.It is possible this bug has only been partially addressed:

• androidx.benchmark:benchmark-baseline-profile-gradle-plugin:1.4.0-alpha04

Since release signing config is used by default instead of debug is not mentioned in release notes - maybe this is a bug?

Cause the last time I found it mentioned in the release notes was in version 1.1

signingConfig.debug is used as the default signing config (b/153583269)

So, if the switch to the release one indeed happened - maybe it's an issue?

The new build types should inherit the certificates from the release build type, unless none is specified in which case debug is used: code link

This should be updated in the documentation, thanks for mentioning it. Will take care of updating it.

Reopened. The speculative fix (with the retry until process is confirmed killed) went into 1.4.0-alpha03, so if you're seeing this on 1.4.0-alpha06 you'd have that.

We tested with Android 14 and 15 devices while applying both solutions stated here and here with no success.

1. Do you mean that even if you use testInstrumentationRunnerArguments["androidx.benchmark.killProcessDelayMillis"] = "10000" the problem still occurs? Jist want to confirm that's what you've tried.
2. Is there perhaps anything your app is doing to be resilient to kills or otherwise come back? disreguard, see you answered above
3. Does this reproduce for you with the sample project/app? (if it's app specific and I can repro on a production app, I can try that)
4. If it happens for the github sample, have you tried it on different devices (emulators as well as physical devices)?

I want to narrow down where this occurs so we can try and reproduce. If it happens with a prod app of yours, that also helps us take a look.

Thanks for reopening!

Do you mean that even if you use testInstrumentationRunnerArguments["androidx.benchmark.killProcessDelayMillis"] = "10000" the problem still occurs? Jist want to confirm that's what you've tried.

That is correct. We tried that, didn't seem to work for these cases.

Does this reproduce for you with the sample project/app?

I wasn't able to repro it with the sample app. Seems to be specific to our app/tests. Although I noticed tests are pretty simple there, only pressHome on setup for example... On our setup steps we need to login (only the first time) before proceeding to the actual benchmarking iterations.

That being said... We found a way of making it work (for now) by having the following:

• Benchmark updated to 1.4.0-alpha06
• At the end of the setup part of our COLD start tests both pressHome() and the solution stated here

Are you able to provide a standalone isolated repro project or steps that works with the your production application? Note that you can change the sample on github to specify your packagename - as long as the package is installed when the test runs, that should make it easier to isolate.

If I can reproduce locally, I can dig in further.

Macrobenchmark 1.1.1 didn't check if the process kill failed - it would just measure startup times incorrectly if the app process didn't die. The fix was to kill the process after the setupBlock and check for success which landed in 1.2.0-alpha07.

As a workaround to mimic the old (incorrect!) measurement behavior, you could implement your own cold startup by passing startupMode = null, and adding the following to your setup block to attempt a cold startup, but without the success check:

    dropShaderCache()
    killProcess()
    dropKernelPageCache()

This will bypass the exception, but similar to 1.1.1, doesn't actually give you a correct measurement.

If you can consistently get an app into this state, please let me know if you find a way to workaround it (killing the app in some other way, waiting in some other way) would be happy to hear about it.

One problem we found with killProcess() is that it essentially calls killall $packageName, which doesn't work if the process name is different than the package name (our case).

We ended up with a custom setupBlock { } that calls am force-stop $packageName, which reliably kills all processes of the given package at the start of each iteration:

Runtime.getRuntime().exec("am force-stop $packageName\n").waitFor()

It works like a charm, the tests are now reliable and consistent. However, it requires a couple of protected permissions: android.permission.FORCE_STOP_PACKAGES and android.permission.INTERACT_ACROSS_USERS_FULL, which is likely a deal breaker for many.

I am using Firebase Test Lab to execute performance tests on device bluejay,version=32, lib version 1.4.0-alpha06, and face the same issue. Specifically, I attempted to mitigate this by using the following configuration: testInstrumentationRunnerArguments["androidx.benchmark.killProcessDelayMillis"] = "10000" and noticed that I get a TestRunner exception in 50-100 ms after the process kill is initiated, but I expected it to happen in 10_000 ms.

- 13:57:15.732 26121 26140 D Benchmark: Killing process dif.tech.plata.perf
...
- 13:57:15.874 26121 26140 E TestRunner: java.lang.IllegalStateException: Package dif.tech.plata.perf must not be running prior to cold start!

Could the root cause of this issue be that the arguments weren't passed to this scope correctly, or that Thread.sleep isn't functioning as expected?

1.4.0-alpha07

For your information:

it helped to set pressHome(10_000) in setupBlock