Benchmarks file many bugs for changes unrelated to the benchmark [354264743]

Assigned

Bug

Status Update

No update yet.

Description

mo...@google.com

created issue #1

Jul 19, 2024 09:47PM

I am currently the triage cop for toolkit and see many bugs created for false positives on benchmarks (both high and low). The symptom is that an unrelated change affects a benchmark. They usually bounce between two different values, but not in a noisy way. The metric will show high for a while, then low for a while, then high again. A bug is filed for both the high and low changes.

Comments

ap...@google.com <ap...@google.com> #2Jul 19, 2024 09:52PM

Project: platform/frameworks/support
Branch: androidx-main

commit 5f42f07ec2b8acf74af422829d8e95d775107af0
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Wed Oct 13 17:48:56 2021

Add a `measureRepeated` overload for `MacrobenchmarkRule`

* Uses a `Consumer<MacrobenchmarkScope>` which makes things easier for Java developers.

Test: Updated unit test.
Bug:

b/184546459

Relnote: Uses a `Consumer<MacrobenchmarkScope>` functional interface, that makes it easy for Java developers to author Macrobenchmarks.
Change-Id: If74abe75c4b0defd0016acd3de20689cf604badd

M benchmark/benchmark-macro-junit4/build.gradle
M benchmark/benchmark-macro-junit4/api/restricted_1.1.0-beta01.txt
M benchmark/benchmark-macro-junit4/src/main/java/androidx/benchmark/macro/junit4/MacrobenchmarkRule.kt
M benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/TrivialStartupJavaBenchmark.java
M benchmark/benchmark-macro-junit4/api/current.txt
M benchmark/benchmark-macro-junit4/api/restricted_current.txt
M benchmark/benchmark-macro-junit4/api/public_plus_experimental_1.1.0-beta01.txt
M benchmark/benchmark-macro-junit4/api/1.1.0-beta01.txt
M benchmark/benchmark-macro-junit4/api/public_plus_experimental_current.txt

https://android-review.googlesource.com/1856865

mo...@google.com <mo...@google.com> #3Jul 22, 2024 04:16PM

Project: platform/frameworks/support
Branch: androidx-main

commit 93b34a82a8608e02f25b02e85c43b0a59f4babda
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Wed Oct 13 16:21:53 2021

Better Java APIs for Macrobenchmarks.

* Add @JvmOverloads for default paramters.

Bug:

b/184546459

Test: Added a `TrivialStartupJavaBenchmark`
Change-Id: I644fe5b1ba90200ae264e8dabd68ae27689befa8
Relnote: Better Java APIs for Macrobenchmarks.

M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/MacrobenchmarkScope.kt
M benchmark/benchmark-macro/api/current.txt
M benchmark/benchmark-macro/api/restricted_1.1.0-beta01.txt
A benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/TrivialStartupJavaBenchmark.java
M benchmark/benchmark-macro/api/public_plus_experimental_1.1.0-beta01.txt
M benchmark/benchmark-macro/api/1.1.0-beta01.txt
M benchmark/benchmark-macro/api/public_plus_experimental_current.txt
M benchmark/benchmark-macro/api/restricted_current.txt

https://android-review.googlesource.com/1856860

ap...@google.com <ap...@google.com> #4Jul 31, 2024 08:57PM

Project: platform/frameworks/support
Branch: androidx-main

commit b3cff5328dfc448de4a05b4722806ea8b7ce33b9
Author: Chris Craik <ccraik@google.com>
Date: Tue Jul 30 15:48:16 2024

Reland Experiment: Enable R8 in foundation-library and compose-runtime benchmarks

Bug: 354264743
Test: DbMonsterBenchmark#dbMonster_count10_mutate10 (D8 = 1,677,855, R8 = 1,266,340)

Benchmarks appear to bounce around when something in the library
changes, regardless of whether it should affect the benchmark.
For example, adding a new collection could cause a benchmark
to increase or decrease, even though the collection hasn't
been used anywhere.

This CL set minifyEnabled to true for compose foundation-library
and compose runtime benchmarks, two libraries that have seen
recent performance variability. This should be considered an
experiment that may be removed in the future or expanded to
other libraries, depending on whether this helps performance
alerts.

(Reland with correct androidTest property)

Test: ran perf tests locally
Change-Id: I0740dd5fd83dd87e7bad95bbfaf2950593a1ec3a

M compose/foundation/foundation-layout/benchmark/build.gradle
M compose/runtime/runtime/compose-runtime-benchmark/build.gradle

https://android-review.googlesource.com/3199776

cc...@google.com <cc...@google.com> #5Aug 16, 2024 08:34PM

Note this had to be reverted again here: https://android-review.git.corp.google.com/c/platform/frameworks/support/+/3205832 due to failures to load listeners set by instrumentation arg.

ap...@google.com <ap...@google.com> #6Aug 28, 2024 08:28PM

Project: platform/frameworks/support
Branch: androidx-main

commit ef02fd556a27c5ff582c4363bf39ec23faf34541
Author: Chris Craik <ccraik@google.com>
Date: Mon Aug 26 16:06:43 2024

Add JUnit runlisteners to benchmark proguard

Bug: 354264743
Test: ./gradlew compose:foundation:foundation-benchmark:cC -P android.testInstrumentationRunnerArguments.class=androidx.compose.foundation.benchmark.NestedScrollerBenchmark -P android.testInstrumentationRunnerArguments.listener=androidx.benchmark.junit4.SideEffectRunListener # with R8 enabled in the module

Relnote: "Keep subclasses of org.junit.runner.notification.RunListener
in benchmark library proguard"

Previously we covered InstrumentationRunListeners (such as we use for
reporting json files) in the proguard file, but were missing the JUnit ones.

Change-Id: Ic8ed5d229c53d4f7779429e003331c251a5f1b39

M benchmark/benchmark-junit4/proguard-rules.pro

https://android-review.googlesource.com/3241410

Issue 354264743