IssueTracker

Component used: Macrobenchmark
Version used: 1.2.0-alpha06
Devices/Android versions reproduced on: Pixel 6 Android 13

When having a benchmark with multiple @Test functions, it seems that the final benchmarkData.json file only contains one benchmark result in the benchmarks JSON array.

This can be reproduced with the code taken from the Baseline profile documentation.

@RunWith(AndroidJUnit4ClassRunner::class)
class ColdStartupBenchmark {
    @get:Rule
    val benchmarkRule = MacrobenchmarkRule()

    @Test
    fun startupNoCompilation() = startup(CompilationMode.None())

    @Test
    fun startupBaselineProfile() = startup(CompilationMode.Partial())

    @Test
    fun startupFullCompilation() = startup(CompilationMode.Full())

    private fun startup(compilationMode: CompilationMode) = benchmarkRule.measureRepeated(
        packageName = "com.example.macrobenchmark.target",
        metrics = listOf(StartupTimingMetric()),
        compilationMode = compilationMode,
        iterations = 10,
        startupMode = StartupMode.COLD,
        setupBlock = {
            pressHome()
        }
    ) {
        // Waits for the first rendered frame, which represents time to initial display.
        startActivityAndWait()
    }
}

Digging in to the benchmark sources, this should work on a first look, because ResultWriter is global object, and collects all the previous benchmark results in a list. However this is not working, because each @Test function seems to be running in a different process, therefore the previous result is lost from the ResultWriter.