IssueTracker

This is working as intended - you're passing StartupMode.COLD, which dictates that the process is killed before measurement.

For a FrameTimingMetric benchmark, just pass null to StartupMode, or Warm if you want the activity recreated for each iteration (edit: in between setup/measure, which isn't the case here).

Expectation is that for measuring a cold startups, setup block can be used to modify persistent app state (e.g. perform login, or load content that will eventually be displayed), but not app process state.

Change from this:

            startupMode = StartupMode.COLD,

To this:

            startupMode = null,

Summarizing offline convo -

startupMode.cold defines what happens between setup + measure blocks. If you want to setup state that isn't measured, just pass null to startup mode.

There's very few cases where it makes sense to pass a startupMode without measuring startup, but you could have a custom metric, so we don't throw in this case, we just hope people will pass the default (null) in these cases.

If you use startupMode = null (default option), you need to write the benchmark in a way that the app state is the same as when starting the benchmark. E.g. when scrolling a finite list, eventually the benchmark would go to the end of the list and therefore not measuring any frames.

Otherwise you can use StartupMode.WARM that restarts the activity, but it doesn't do it in between setupBlock and measureBlock, so you can prepare the UI in setupBlock and measure it in measureBlock.

I think the core problem is that it's surprising that "startActivity" may not actually start a new activity, when it's not the core action being measured.

Options:

1. Change startupMode = null behavior to always set new instance flag, and reduce this surprise
2. Overhaul activity lifecycle control - Change startActivityAndWait() to take a StartupMode parameter, and recommend the one without a parameter to only be used when you don't care about the type of startup. We'd deprecate the variant of measureRepeated that takes a startupMode.

I'm thinking we go with #1 for now, but we can think about some variant of 2 in the long term.

2 is more complex than it sounds on the surface, since we then have scenarios where someone calls startActivityAndWait(HOT), but the activity/process aren't alive. We could pre-launch the process/activity as needed, but that's not possible to do outside the measure block, so it makes a warm/hot startup benchmark trace very confusing, where there's 2 startups for the first iteration.

I agree that 2) makes probably more sense in the long run and simplifies the API to know when something is started / killed.

For now, I let's go with 1). We should emphasize this behavior change in case anyone was doing something that relied on this behavior.

Is this behavior change scheduled for one of the upcoming alphas?

I'm experiencing the same issue:

I try to use benchmark in the official https://github.com/android/sunflower sample app. After benchmark change from 1.2.0-alpha06 to >= 1.2.0-alpha07, setupBlock doesn't work as expected any more.

    private fun benchmarkPlantDetail(compilationMode: CompilationMode) =
        benchmarkRule.measureRepeated(
            packageName = PACKAGE_NAME,
            metrics = listOf(FrameTimingMetric()),
            compilationMode = compilationMode,
            iterations = 10,
            startupMode = null,
            setupBlock = {
                pressHome()
                startActivityAndWait()
                goToPlantListTab()
                Log.d("Benchmark", "setupBlock iteration=$iteration")
            }
        ) {
            Log.d("Benchmark", "measureBlock iteration=$iteration")
            goToPlantDetail()
        }
}

Log:

Benchmark: setupBlock iteration=0
Benchmark: Dropping shader cache for com.google.samples.apps.sunflower
Benchmark: Profile Installer - Benchmark Operation: DROP_SHADER_CACHE
ProfileInstaller:
Benchmark: Killing process com.google.samples.apps.sunflower
... ...
Benchmark: measureBlock iteration=0

The path to go to the plant detail screen is: start the app -> click on plant list button to go to the plant list screen -> click on the first plant to go to a plant detail screen The intention of PlantDetailBenchmarks is to only benchmark the frame performance in plant detail screen in a cold boot situation.

I tried changing to use "startupMode = null" with benchmark of 1.2.0-alpha13, it will throw exception when running PlantDetailBenchmarks. In the 2nd iteration, it won't go to the "Plant list screen" first, but go to the plant detail screen directly and result in "Attempt to invoke virtual method 'int androidx.test.uiautomator.UiObject2.getChildCount()' on a null object reference" because the plant list button was not shown in the 2nd iteration.

Any suggestions on how to benchmark screen frame performance in a cold boot situation while the screen is not the first startup screen? My branch is https://github.com/android/sunflower/compare/main...YingChen0605:sunflower:0505CustomBaseline_startup_benchmarkTry1.2.0-alpha13?expand=1 if it helps

Thanks!

You can fix this by changing StartupMode from null to WARM. As mentioned in comment #8, startupMode = null doesn't actually start new Activity if it already exists. StartupMode.WARM will always create a new Activity. So this should fix the problem.

Thank you for the response! But with WARM, it won't be a cold start from 2nd iteration? We can not profile a screen performance for a cold start

(I believe BaselineProfile benefits the flow after first-time install. I'm trying to profile the impact of BaselineProfile in this case.)

If you want to profile a subsequent screen for cold start, you can do the following in your setup block:

    startupMode = null // can leave out, this is the default
    setupBlock = {
        killProcess() // force a cold start in the next startActivityAndWait()
        startActivityAndWait()
        goToPlantListTab()
        Log.d("Benchmark", "setupBlock iteration=$iteration")
    }

I agree this is confusing if you're not trying to measure startup. I'm not sure if this is an argument in favor of restoring the old behavior (which I still think is dangerous if you are trying to measure startup), or of deprecating the startupMode argument, and just having a flag in startActivityAndWait which lets you control whether it should kill the process first (cold) or just launch a new activity (warm), as mentioned in #comment8

Given this is causing confusion for users, I'm going to add a warning to DAC.