Choose good default between until stable vs iterations for baseline profile generation [281078707]

Fixed

Bug

Status Update

No update yet.

Description

cc...@google.com

created issue #1

May 5, 2023 05:51PM

Before 1.2 goes stable, we should choose the correct default here. If we like the collectStableBaselineProfile approach, let's make it the default.

Comments

be...@google.com <be...@google.com> #2May 15, 2023 05:32PM

Project: platform/frameworks/support
Branch: androidx-main

commit 5f42f07ec2b8acf74af422829d8e95d775107af0
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Wed Oct 13 17:48:56 2021

Add a `measureRepeated` overload for `MacrobenchmarkRule`

* Uses a `Consumer<MacrobenchmarkScope>` which makes things easier for Java developers.

Test: Updated unit test.
Bug:

b/184546459

Relnote: Uses a `Consumer<MacrobenchmarkScope>` functional interface, that makes it easy for Java developers to author Macrobenchmarks.
Change-Id: If74abe75c4b0defd0016acd3de20689cf604badd

M benchmark/benchmark-macro-junit4/build.gradle
M benchmark/benchmark-macro-junit4/api/restricted_1.1.0-beta01.txt
M benchmark/benchmark-macro-junit4/src/main/java/androidx/benchmark/macro/junit4/MacrobenchmarkRule.kt
M benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/TrivialStartupJavaBenchmark.java
M benchmark/benchmark-macro-junit4/api/current.txt
M benchmark/benchmark-macro-junit4/api/restricted_current.txt
M benchmark/benchmark-macro-junit4/api/public_plus_experimental_1.1.0-beta01.txt
M benchmark/benchmark-macro-junit4/api/1.1.0-beta01.txt
M benchmark/benchmark-macro-junit4/api/public_plus_experimental_current.txt

https://android-review.googlesource.com/1856865

ra...@google.com <ra...@google.com> #3May 15, 2023 06:07PM

Project: platform/frameworks/support
Branch: androidx-main

commit 93b34a82a8608e02f25b02e85c43b0a59f4babda
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Wed Oct 13 16:21:53 2021

Better Java APIs for Macrobenchmarks.

* Add @JvmOverloads for default paramters.

Bug:

b/184546459

Test: Added a `TrivialStartupJavaBenchmark`
Change-Id: I644fe5b1ba90200ae264e8dabd68ae27689befa8
Relnote: Better Java APIs for Macrobenchmarks.

M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/MacrobenchmarkScope.kt
M benchmark/benchmark-macro/api/current.txt
M benchmark/benchmark-macro/api/restricted_1.1.0-beta01.txt
A benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/TrivialStartupJavaBenchmark.java
M benchmark/benchmark-macro/api/public_plus_experimental_1.1.0-beta01.txt
M benchmark/benchmark-macro/api/1.1.0-beta01.txt
M benchmark/benchmark-macro/api/public_plus_experimental_current.txt
M benchmark/benchmark-macro/api/restricted_current.txt

https://android-review.googlesource.com/1856860

be...@google.com <be...@google.com> #4May 22, 2023 02:52PM

Yes, collectStableBaselineProfile is the right approach.
The default value for observed stable iterations is 3. It's currently not clear from the ktdocs that this have to be three sequential iterations.

Do we know how many iterations generally are run until a profile is stable? This can help us inform the recommended number of maxIterations, which we currently default to 10 in our samples.

ra...@google.com <ra...@google.com> May 30, 2023 11:47PM

Accepted by ra...@google.com.

ap...@google.com <ap...@google.com> #5Jun 1, 2023 06:31PM

Marked as fixed.

Project: platform/frameworks/support
Branch: androidx-main

commit f9f2ec225dcc525555f4cd410528fc1ee80657db
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Tue May 30 16:41:51 2023

Always generate stable baseline profiles.

Test: Existing unit tests pass. Updated unit tests.
Fixes:

b/281078707

Relnote: "The `collectBaselineProfile` API always generates stable baseline profiles. The `collectStableBaselineProfile` API has been removed and `collectBaselineProfile` should be used instead."
Change-Id: I1726295116bbecbcbc72561aee96538c1ea9a514

M benchmark/benchmark-macro-junit4/api/current.txt
M benchmark/benchmark-macro-junit4/api/restricted_current.txt
M benchmark/benchmark-macro-junit4/src/main/java/androidx/benchmark/macro/junit4/BaselineProfileRule.kt
M benchmark/benchmark-macro/api/current.txt
M benchmark/benchmark-macro/api/restricted_current.txt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/BaselineProfiles.kt
D benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/ExperimentalStableBaselineProfilesApi.kt
M benchmark/integration-tests/macrobenchmark/src/main/java/androidx/benchmark/integration/macrobenchmark/BaselineProfileRuleTest.kt
M benchmark/integration-tests/macrobenchmark/src/main/java/androidx/benchmark/integration/macrobenchmark/TrivialListScrollBaselineProfile.kt
M benchmark/integration-tests/macrobenchmark/src/main/java/androidx/benchmark/integration/macrobenchmark/TrivialStartupBaselineProfile.kt

https://android-review.googlesource.com/2609630

pr...@google.com <pr...@google.com> #6Jun 8, 2023 01:30AM

The following release(s) address this bug.It is possible this bug has only been partially addressed:

androidx.benchmark:benchmark-macro:1.2.0-alpha15
androidx.benchmark:benchmark-macro-junit4:1.2.0-alpha15

Issue 281078707