IssueTracker

Project: platform/frameworks/support
Branch: androidx-main

commit c5056a9a2cf942986812968b9054cdd3d511ebce
Author: Chris Craik <ccraik@google.com>
Date:   Fri Sep 09 15:46:20 2022

    Implement dryRunMode.enable for macrobenchmarks
   
    Test: Manual (see below)
    Fixes:175149857
   
    Relnote:"""
    Add support for dryRunMode.enable instrumentation argument to
    macrobenchmark (already available in micro) for faster local
    development, and validating app automation (e.g. in presubmit). This
    overrides iterations to 1, skips compilation, suppresses all errors,
    and disables measurement .json file output.
   
    On Gradle command line: `./gradlew macrobenchmark:cC -P android.testInstrumentationRunnerArguments.androidx.benchmark.dryRunMode.enable=true`
   
    In build.gradle:
   
    ```
    android {
        defaultConfig {
            testInstrumentationRunnerArgument 'androidx.benchmark.dryRunMode.enable', 'true'
        }
    }
    ```
    """
   
    Test steps performed:
   
    1) Baseline run with >1 iteration and release build:
    > ./gradlew benchmark:integration-tests:macrobenchmark:cC -P android.testInstrumentationRunnerArguments.class=androidx.benchmark.integration.macrobenchmark.TrivialStartupBenchmark`
    > ls ../../out/androidx/benchmark/integration-tests/macrobenchmark/build/outputs/connected_android_test_additional_output/debugAndroidTest/connected/Pixel\ 4a\ \(5G\)\ -\ 13/
     additionaltestoutput.benchmark.message.txt
     androidx.benchmark.integration.macrobenchmark.test-benchmarkData.json
    'TrivialStartupBenchmark_startup[startup=COLD,compilation=None]_iter000_2022-09-09-23-08-11.perfetto-trace'
    'TrivialStartupBenchmark_startup[startup=COLD,compilation=None]_iter001_2022-09-09-23-08-14.perfetto-trace'
    'TrivialStartupBenchmark_startup[startup=COLD,compilation=None]_iter002_2022-09-09-23-08-17.perfetto-trace'
    'TrivialStartupBenchmark_startup[startup=COLD,compilation=None]_iter003_2022-09-09-23-08-21.perfetto-trace'
    'TrivialStartupBenchmark_startup[startup=COLD,compilation=None]_iter004_2022-09-09-23-08-24.perfetto-trace'
    'TrivialStartupBenchmark_startup[startup=COLD,compilation=None]_iter005_2022-09-09-23-08-27.perfetto-trace'
    'TrivialStartupBenchmark_startup[startup=COLD,compilation=None]_iter006_2022-09-09-23-08-30.perfetto-trace'
    'TrivialStartupBenchmark_startup[startup=COLD,compilation=None]_iter007_2022-09-09-23-08-33.perfetto-trace'
    'TrivialStartupBenchmark_startup[startup=COLD,compilation=None]_iter008_2022-09-09-23-08-37.perfetto-trace'
    'TrivialStartupBenchmark_startup[startup=COLD,compilation=None]_iter009_2022-09-09-23-08-40.perfetto-trace'
    ```
   
    2) Comment out `tasks.getByPath(":benchmark:integration-tests:macrobenchmark:connectedCheck").dependsOn...` in macrobench build.gradle (couldn't get it to depend on debug for some reason)
   
    3) `./gradlew benchmark:integration-tests:macrobenchmark-target:installDebug`
   
    4) Run benchmark without dryRunMode, fails with unsuppressed error
    ```
    > ./gradlew benchmark:integration-tests:macrobenchmark:cC -P android.testInstrumentationRunnerArguments.class=androidx.benchmark.integration.macrobenchmark.TrivialStartupBenchmark -P android.testInstrumentationRunnerArguments.androidx.benchmark.dryRunMode.enable=true
    ...
    androidx.benchmark.integration.macrobenchmark.TrivialStartupBenchmark > startup[startup=COLD,compilation=None][Pixel 4a (5G) - 13] FAILED
            java.lang.AssertionError: ERRORS (not suppressed): DEBUGGABLE
    ```
   
    5) Run with dryRunMode succeeds, doesn't output json file, and only one trace is observed:
   
    ```
    > ./gradlew benchmark:integration-tests:macrobenchmark:cC -P android.testInstrumentationRunnerArguments.class=androidx.benchmark.integration.macrobenchmark.TrivialStartupBenchmark -P android.testInstrumentationRunnerArguments.androidx.benchmark.dryRunMode.enable=true`
    > ls ../../out/androidx/benchmark/integration-tests/macrobenchmark/build/outputs/connected_android_test_additional_output/debugAndroidTest/connected/Pixel\ 4a\ \(5G\)\ -\ 13/
     additionaltestoutput.benchmark.message.txt  'TrivialStartupBenchmark_startup[startup=COLD,compilation=None]_iter000_2022-09-09-22-49-20.perfetto-trace'
    ```
   
    Change-Id: Ib51b446731deef7aec7ddd604f0f701e1b0d7feb

M       benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Macrobenchmark.kt
M       benchmark/benchmark-common/src/main/java/androidx/benchmark/Arguments.kt
M       benchmark/benchmark-common/src/main/java/androidx/benchmark/ConfigurationError.kt

https://android-review.googlesource.com/2213984