Benchmark should warn if Test Orchestrator is used [286899049]

Fixed

Bug

Status Update

No update yet.

Description

cs...@supercharge.io

created issue #1

Jun 13, 2023 09:11AM

Component used: Macrobenchmark
Version used: 1.2.0-alpha06
Devices/Android versions reproduced on: Pixel 6 Android 13

When having a benchmark with multiple @Test functions, it seems that the final benchmarkData.json file only contains one benchmark result in the benchmarks JSON array.

This can be reproduced with the code taken from the Baseline profile documentation.

@RunWith(AndroidJUnit4ClassRunner::class)
class ColdStartupBenchmark {
    @get:Rule
    val benchmarkRule = MacrobenchmarkRule()

    @Test
    fun startupNoCompilation() = startup(CompilationMode.None())

    @Test
    fun startupBaselineProfile() = startup(CompilationMode.Partial())

    @Test
    fun startupFullCompilation() = startup(CompilationMode.Full())

    private fun startup(compilationMode: CompilationMode) = benchmarkRule.measureRepeated(
        packageName = "com.example.macrobenchmark.target",
        metrics = listOf(StartupTimingMetric()),
        compilationMode = compilationMode,
        iterations = 10,
        startupMode = StartupMode.COLD,
        setupBlock = {
            pressHome()
        }
    ) {
        // Waits for the first rendered frame, which represents time to initial display.
        startActivityAndWait()
    }
}

Digging in to the benchmark sources, this should work on a first look, because ResultWriter is global object, and collects all the previous benchmark results in a list. However this is not working, because each @Test function seems to be running in a different process, therefore the previous result is lost from the ResultWriter.

Comments

cc...@google.com <cc...@google.com> #2Jun 13, 2023 06:00PM

Any suggestions on where such an interface might live? I don't think it makes sense for the interface to be in the drawerlayout artifact if you're specifically going to use the interface to *not* use DrawerLayout.

cs...@supercharge.io <cs...@supercharge.io> #3Jun 14, 2023 11:21AM

androidx.customview would be a good candidate. This is used by all of the widget libraries.

ap...@google.com <ap...@google.com> #4Mar 18, 2024 10:16PM

Marked as fixed.

Project: platform/frameworks/support
Branch: androidx-master-dev

commit 3ed3fb003fa6c1244f923202859a616225b5b2fa
Author: Ian Lake <ilake@google.com>
Date: Fri Feb 14 11:17:46 2020

Create an interface for layouts that can be opened

Create a common interface that represents layouts
that have two states: open and closed. This allows
higher level libraries to rely on the interface,
rather than concrete implementations such as
DrawerLayout, making them more resilient to
changes in the current recommended implementation.

Fixes: 129979320
Test: ./gradlew checkApi
Change-Id: I0f2a1414977825aa053c6555261f2b7d4417bd19

M customview/customview/api/1.1.0-alpha02.txt
M customview/customview/api/current.txt
M customview/customview/api/public_plus_experimental_1.1.0-alpha02.txt
M customview/customview/api/public_plus_experimental_current.txt
M customview/customview/api/restricted_1.1.0-alpha02.txt
M customview/customview/api/restricted_current.txt
A customview/customview/src/main/java/androidx/customview/widget/Openable.java
M drawerlayout/drawerlayout/api/1.1.0-alpha03.txt
M drawerlayout/drawerlayout/api/current.txt
M drawerlayout/drawerlayout/api/public_plus_experimental_1.1.0-alpha03.txt
M drawerlayout/drawerlayout/api/public_plus_experimental_current.txt
M drawerlayout/drawerlayout/api/restricted_1.1.0-alpha03.txt
M drawerlayout/drawerlayout/api/restricted_current.txt
M drawerlayout/drawerlayout/build.gradle
M drawerlayout/drawerlayout/src/main/java/androidx/drawerlayout/widget/DrawerLayout.java
M jetifier/jetifier/migration.config
M slidingpanelayout/slidingpanelayout/api/1.1.0-alpha01.txt
M slidingpanelayout/slidingpanelayout/api/current.txt
M slidingpanelayout/slidingpanelayout/api/public_plus_experimental_1.1.0-alpha01.txt
M slidingpanelayout/slidingpanelayout/api/public_plus_experimental_current.txt
M slidingpanelayout/slidingpanelayout/api/restricted_1.1.0-alpha01.txt
M slidingpanelayout/slidingpanelayout/api/restricted_current.txt
M slidingpanelayout/slidingpanelayout/src/main/java/androidx/slidingpanelayout/widget/SlidingPaneLayout.java

https://android-review.googlesource.com/940787

na...@google.com <na...@google.com> #5Apr 17, 2024 05:17PM

The following release(s) address this bug.It is possible this bug has only been partially addressed:

androidx.benchmark:benchmark-common:1.3.0-alpha03

Issue 286899049

Description

Issue summary

Comments

cc...@google.com <cc...@google.com> #2Jun 13, 2023 06:00PM

cs...@supercharge.io <cs...@supercharge.io> #3Jun 14, 2023 11:21AM

ap...@google.com <ap...@google.com> #4Mar 18, 2024 10:16PM

na...@google.com <na...@google.com> #5Apr 17, 2024 05:17PM

Add comment

Issue metadata