java.lang.IllegalStateException: Package ${packageName} must not be running prior to cold start! [351582215]

Assigned

Bug

Status Update

No update yet.

Description

kl...@naver.com

created issue #1

Jul 8, 2024 05:06AM

Component used: androidx.benchmark.benchmark-macro-junit4

Version used:1.2.4

Devices/Android versions reproduced on:

AGP: 8.2.0

Android Studio: Iguana

Baseline Profile: androidx.baselineprofile -> 1.2.4

An error occurs when I create the Baseline Profile module using the Baseline Profile Generator template and then run the sample code below.

@RunWith(AndroidJUnit4::class)
@LargeTest
class StartupBenchmarks {

    @get:Rule
    val rule = MacrobenchmarkRule()

    @Test
    fun startupCompilationNone() =
        benchmark(CompilationMode.None())

    @Test
    fun startupCompilationBaselineProfiles() =
        benchmark(CompilationMode.Partial(BaselineProfileMode.Require))

    private fun benchmark(compilationMode: CompilationMode) {
        // This example works only with the variant with application id `com.nhn.android.calendar`."
        rule.measureRepeated(
            packageName = "${packageName}",
            metrics = listOf(StartupTimingMetric()),
            compilationMode = compilationMode,
            startupMode = StartupMode.COLD,
            iterations = 5,
            setupBlock = {
                pressHome()
                allowNotifications()
            },
            measureBlock = {
                startActivityAndAllowNotifications()

                // TODO Add interactions to wait for when your app is fully drawn.
                // The app is fully drawn when Activity.reportFullyDrawn is called.
                // For Jetpack Compose, you can use ReportDrawn, ReportDrawnWhen and ReportDrawnAfter
                // from the AndroidX Activity library.

                // Check the UiAutomator documentation for more information on how to
                // interact with the app.
                // https://d.android.com/training/testing/other-components/ui-automator
            }
        )
    }
}

java.lang.IllegalStateException: Package ${packageName} must not be running prior to cold start!
at androidx.benchmark.macro.MacrobenchmarkKt$macrobenchmarkWithStartupMode$1.invoke(Macrobenchmark.kt:463)
at androidx.benchmark.macro.MacrobenchmarkKt$macrobenchmarkWithStartupMode$1.invoke(Macrobenchmark.kt:436)
at androidx.benchmark.macro.MacrobenchmarkKt$macrobenchmark$measurements$1.invoke(Macrobenchmark.kt:265)
at androidx.benchmark.macro.MacrobenchmarkKt$macrobenchmark$measurements$1.invoke(Macrobenchmark.kt:255)
at androidx.benchmark.perfetto.PerfettoTraceProcessor$Companion.runServer(PerfettoTraceProcessor.kt:105)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:255)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:436)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:107)
at com.nhn.android.calendar.baselineprofile.mobile.StartupBenchmarks.benchmark(StartupBenchmarks.kt:52)
at com.nhn.android.calendar.baselineprofile.mobile.StartupBenchmarks.startupCompilationNone(StartupBenchmarks.kt:44)

Is there a solution?

Comments

cc...@google.com <cc...@google.com> #2Jul 17, 2024 05:53PM

Reassigned to cc...@google.com.

Yigit, do you have time to fix it?

reemission of the same liveData is racy

fr...@disneystreaming.com <fr...@disneystreaming.com> #3Oct 1, 2024 03:35PM

yea i'll take it.

jo...@disneystreaming.com <jo...@disneystreaming.com> #4Oct 8, 2024 12:25PM

Thanks for the detailed analysis. This may not be an issue anymore since we've started using Main.immediate there but I' not sure; I'll try to create a test case.

cc...@google.com <cc...@google.com> #5Oct 8, 2024 03:57PM

just emitting same live data reproduces the issue.

@Test
fun raceTest() {
val subLiveData = MutableLiveData(1)
val subject = liveData(testScope.coroutineContext) {
emitSource(subLiveData)
emitSource(subLiveData) //crashes
}
subject.addObserver().apply {
testScope.advanceUntilIdle()
}
}

ap...@google.com <ap...@google.com> #6Oct 9, 2024 10:12PM

With 2.2.0-alpha04 (that use Main.immediate), the issue seems to be still there (I tested it by calling emitSource() twice, like your test case)

cc...@google.com <cc...@google.com> Nov 12, 2024 06:51PM

Marked as fixed.

fr...@disneystreaming.com <fr...@disneystreaming.com> #7Jan 10, 2025 02:43PM

yea sorry immediate does not fix it.
I actually have a WIP fix for it:

https://android-review.googlesource.com/c/platform/frameworks/support/+/1112186

if your case is the one i found (emitting same LiveData multiple times, as shown in #5) you can work around it by adding a dummy transformation.

val subLiveData = MutableLiveData(1)
val subject = liveData(testScope.coroutineContext) {
emitSource(subLiveData.map {it })
emitSource(subLiveData.map {it} )
}

Message last modified on Jan 10, 2025 02:43PM

cc...@google.com <cc...@google.com> #8Jan 10, 2025 10:11PM

Status: Assigned (reopened)

Project: platform/frameworks/support
Branch: androidx-master-dev

commit af12e75e6b4110f48e44ca121466943909de8f06
Author: Yigit Boyar <yboyar@google.com>
Date: Tue Sep 03 12:58:11 2019

Fix coroutine livedata race condition

This CL fixes a bug in liveData builder where emitting same
LiveData source twice would make it crash because the second
emission registry could possibly happen before first one is
removed as source.

We fix it by using a suspending dispose function. It does feel
a bit hacky but we cannot make DisposableHandle.dispose async
and we do not want to block there. This does not mean that there
is a problem if developer disposes it manually since our emit
functions take care of making sure it disposes (and there is
no other way to add source to the underlying MediatorLiveData)

Bug: 140249349
Test: BuildLiveDataTest#raceTest_*

Change-Id: I0b464c242a583da4669af195cf2504e2adc4de40

M lifecycle/lifecycle-livedata-ktx/api/2.2.0-alpha05.txt
M lifecycle/lifecycle-livedata-ktx/api/current.txt
M lifecycle/lifecycle-livedata-ktx/api/public_plus_experimental_2.2.0-alpha05.txt
M lifecycle/lifecycle-livedata-ktx/api/public_plus_experimental_current.txt
M lifecycle/lifecycle-livedata-ktx/api/restricted_2.2.0-alpha05.txt
M lifecycle/lifecycle-livedata-ktx/api/restricted_current.txt
M lifecycle/lifecycle-livedata-ktx/src/main/java/androidx/lifecycle/CoroutineLiveData.kt
M lifecycle/lifecycle-livedata-ktx/src/test/java/androidx/lifecycle/BuildLiveDataTest.kt

https://android-review.googlesource.com/1112186

https://goto.google.com/android-sha1/af12e75e6b4110f48e44ca121466943909de8f06

Message last modified on Jan 10, 2025 10:13PM

fr...@disneystreaming.com <fr...@disneystreaming.com> #9Jan 14, 2025 09:41AM

Thanks for reopening!

Do you mean that even if you use testInstrumentationRunnerArguments["androidx.benchmark.killProcessDelayMillis"] = "10000" the problem still occurs? Jist want to confirm that's what you've tried.

That is correct. We tried that, didn't seem to work for these cases.

Does this reproduce for you with the sample project/app?

I wasn't able to repro it with the sample app. Seems to be specific to our app/tests. Although I noticed tests are pretty simple there, only pressHome on setup for example... On our setup steps we need to login (only the first time) before proceeding to the actual benchmarking iterations.

That being said... We found a way of making it work (for now) by having the following:

Benchmark updated to 1.4.0-alpha06
At the end of the setup part of our COLD start tests both pressHome() and the solution stated here

jo...@disneystreaming.com <jo...@disneystreaming.com> #10Jan 17, 2025 11:19AM

Hi, I am part of the same team than Francisco (previous comment). Sadly we got the same error again, even after the workaround mentioned above.

Something we also found is that sometimes it helps to force stop the app, wipe app data and caches, and uninstall manually in order to remove the error in the following run. We also found that even after we get the error, the process seems to stay alive somehow. If we run the command: `ps | grep com.disney.disneyplus` we can see that the process is still there, which might be related.

Message last modified on Jan 17, 2025 11:22AM

cc...@google.com <cc...@google.com> #11Jan 17, 2025 08:08PM

Are you able to provide a standalone isolated repro project or steps that works with the your production application? Note that you can change the sample on github to specify your packagename - as long as the package is installed when the test runs, that should make it easier to isolate.

If I can reproduce locally, I can dig in further.

on...@gmail.com <on...@gmail.com> #12Jan 20, 2025 08:10PM

I am not sure if this is helpful, but I had this exact same error yesterday with versions 1.2.0-1.3.3 of Macrobenchmark. 1.1.1 was working fine.

Then, one of my coworkers tried restarting adb as root and the error disappeared.

I'd call it a workaround though, since running adb root is not always possible or convenient. What changed in 1.2.0+ that root is now required for Macrobenchmark to run properly?

cc...@google.com <cc...@google.com> #13Jan 22, 2025 10:24PM

Macrobenchmark 1.1.1 didn't check if the process kill failed - it would just measure startup times incorrectly if the app process didn't die. The fix was to kill the process after the setupBlock and check for success which landed in 1.2.0-alpha07.

As a workaround to mimic the old (incorrect!) measurement behavior, you could implement your own cold startup by passing startupMode = null, and adding the following to your setup block to attempt a cold startup, but without the success check:

    dropShaderCache()
    killProcess()
    dropKernelPageCache()

This will bypass the exception, but similar to 1.1.1, doesn't actually give you a correct measurement.

If you can consistently get an app into this state, please let me know if you find a way to workaround it (killing the app in some other way, waiting in some other way) would be happy to hear about it.

on...@gmail.com <on...@gmail.com> #14Jan 24, 2025 04:45AM

One problem we found with killProcess() is that it essentially calls killall $packageName, which doesn't work if the process name is different than the package name (our case).

We ended up with a custom setupBlock { } that calls am force-stop $packageName, which reliably kills all processes of the given package at the start of each iteration:

Runtime.getRuntime().exec("am force-stop $packageName\n").waitFor()

It works like a charm, the tests are now reliable and consistent. However, it requires a couple of protected permissions: android.permission.FORCE_STOP_PACKAGES and android.permission.INTERACT_ACROSS_USERS_FULL, which is likely a deal breaker for many.

op...@gmail.com <op...@gmail.com> #15Jan 28, 2025 12:16PM

I am using Firebase Test Lab to execute performance tests on device bluejay,version=32, lib version 1.4.0-alpha06, and face the same issue. Specifically, I attempted to mitigate this by using the following configuration: testInstrumentationRunnerArguments["androidx.benchmark.killProcessDelayMillis"] = "10000" and noticed that I get a TestRunner exception in 50-100 ms after the process kill is initiated, but I expected it to happen in 10_000 ms.

- 13:57:15.732 26121 26140 D Benchmark: Killing process dif.tech.plata.perf
...
- 13:57:15.874 26121 26140 E TestRunner: java.lang.IllegalStateException: Package dif.tech.plata.perf must not be running prior to cold start!

Could the root cause of this issue be that the arguments weren't passed to this scope correctly, or that Thread.sleep isn't functioning as expected?

Message last modified on Jan 28, 2025 12:56PM

p....@squaregps.com <p....@squaregps.com> #16Feb 3, 2025 02:50PM

1.4.0-alpha07

For your information:

it helped to set pressHome(10_000) in setupBlock