Macrobenchmarks - Regression found between 11748432 and 11748459 [336588271]

Fixed

Bug

Status Update

No update yet.

Description

sk...@skia-public.iam.gserviceaccount.com

created issue on behalf of an...@google.com #1

Apr 23, 2024 07:12PM

Perf Regression (High) found, matching 214 tracked metrics from benchmarks.

To triage this regression, see the guide at go/androidx-bench-triage.

Tests affected:

 androidx.benchmark.integration.macrobenchmark.GridBenchmark_scroll_compilation_WarmupProfile_iterations_3__
 androidx.benchmark.integration.macrobenchmark.SeparateProcessBenchmark_startup
 androidx.benchmark.integration.macrobenchmark.SmallListStartupBenchmark_startup_startup_COLD_compilation_None_
 androidx.benchmark.integration.macrobenchmark.SmallListStartupBenchmark_startup_startup_COLD_compilation_WarmupProfile_iterations_3__
 androidx.benchmark.integration.macrobenchmark.StartupPressHomeBenchmark_startup_pressHomeInMeasure_false_startup_WARM_
 androidx.benchmark.integration.macrobenchmark.StartupPressHomeBenchmark_startup_pressHomeInMeasure_true_startup_COLD_
 androidx.benchmark.integration.macrobenchmark.TrivialListScrollBenchmark_start_compilation_Full_
 androidx.benchmark.integration.macrobenchmark.TrivialListScrollBenchmark_start_compilation_None_
 androidx.compose.integration.hero.macrobenchmark.jetsnack.JetsnackScrollBenchmark_scrollHome_compilation_WarmupProfile_iterations_3__
 androidx.compose.integration.hero.macrobenchmark.jetsnack.JetsnackStartupBenchmark_startup_startup_COLD_compilationMode_None_
 androidx.compose.integration.hero.macrobenchmark.jetsnack.JetsnackStartupBenchmark_startup_startup_HOT_compilationMode_BaselineProfile_
 androidx.compose.integration.macrobenchmark.AndroidViewListScrollBenchmark_scroll_compilation_BaselineProfile_
 androidx.compose.integration.macrobenchmark.AndroidViewListScrollBenchmark_scroll_compilation_None_
 androidx.compose.integration.macrobenchmark.AndroidViewListScrollBenchmark_scroll_compilation_WarmupProfile_iterations_3__
 androidx.constraintlayout.compose.integration.macrobenchmark.MotionLayoutBenchmark_messageDsl
 androidx.constraintlayout.compose.integration.macrobenchmark.MotionLayoutBenchmark_messageJson
 androidx.profileinstaller.integration.macrobenchmark.ProfileinstallerStartupBenchmark_startup_startup_COLD_compilation_WarmupProfile_iterations_3__
 androidx.wear.benchmark.integration.macrobenchmark.ScrollBenchmark_start_compilation_BaselineProfile_
 androidx.wear.benchmark.integration.macrobenchmark.ScrollBenchmark_start_compilation_Full_
 androidx.wear.benchmark.integration.macrobenchmark.ScrollBenchmark_start_compilation_None_
 androidx.wear.benchmark.integration.macrobenchmark.ScrollBenchmark_start_compilation_WarmupProfile_iterations_3__
 androidx.wear.benchmark.integration.macrobenchmark.StartupBenchmark_startup_startup_HOT_compilation_BaselineProfile_
 androidx.wear.benchmark.integration.macrobenchmark.StartupBenchmark_startup_startup_WARM_compilation_BaselineProfile_
 androidx.wear.benchmark.integration.macrobenchmark.SwipeBenchmark_start_compilation_None_
 androidx.wear.benchmark.integration.macrobenchmark.SwipeBenchmark_start_compilation_WarmupProfile_iterations_3__
 androidx.wear.compose.integration.macrobenchmark.PositionIndicatorBenchmark_start_compilation_Full_
 androidx.wear.compose.integration.macrobenchmark.PositionIndicatorBenchmark_start_compilation_None_
 androidx.wear.compose.integration.macrobenchmark.PositionIndicatorBenchmark_start_compilation_WarmupProfile_iterations_3__
 androidx.wear.compose.integration.macrobenchmark.test.ScrollBenchmark_start_compilation_BaselineProfile_
 androidx.wear.compose.integration.macrobenchmark.test.ScrollBenchmark_start_compilation_Full_
 androidx.wear.compose.integration.macrobenchmark.test.ScrollBenchmark_start_compilation_WarmupProfile_iterations_3__
 androidx.wear.compose.integration.macrobenchmark.test.StartupBenchmark_startup_startup_COLD_compilation_Full_
 androidx.wear.compose.integration.macrobenchmark.test.SwipeBenchmark_start_compilation_BaselineProfile_
 androidx.wear.compose.integration.macrobenchmark.test.SwipeBenchmark_start_compilation_Full_
 androidx.wear.compose.integration.macrobenchmark.test.SwipeBenchmark_start_compilation_None_
 androidx.wear.compose.integration.macrobenchmark.test.SwipeBenchmark_start_compilation_WarmupProfile_iterations_3__

Devices affected:

 sailfish

API Level:

 API_29_Q

Comments

ra...@google.com <ra...@google.com> #2Apr 24, 2024 06:50PM

Accepted by ra...@google.com.

This is expected for now. We are now turning on method traces where we did not do so previously. Given method tracing is turned-on in CI automatically, this will start affecting all Macrobenchmarks.

What we need to do, is to introduce a separate profiling phase for method tracing to avoid regressions in metrics.

cc...@google.com <cc...@google.com> #3Apr 25, 2024 09:50PM

This is expected for now. We are now turning on method traces where we did not do so previously. Given method tracing is turned-on in CI automatically, this will start affecting all Macrobenchmarks.

It should not be, even without your follow up CL - we turn on method tracing by default in micro, but macro attempts to avoid that - https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:benchmark/benchmark-common/src/main/java/androidx/benchmark/Arguments.kt;l=288;drc=e8c21d81df2242168e21b1310b62609887e70406

ap...@google.com <ap...@google.com> #4Apr 26, 2024 07:00PM

Marked as fixed.

Project: platform/frameworks/support
Branch: androidx-main

commit 95c0ded0f9d7b0853d72a732e57d86032ae5f568
Author: Chris Craik <ccraik@google.com>
Date: Fri Apr 26 10:49:10 2024

Fix method tracing to be off by default in macrobench

Fixes: 336588271
Test: SmallListStartupBenchmark (no tracing)

Change-Id: I9896030622be844e631c8aff69ba55c01f0661d7

M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Macrobenchmark.kt

https://android-review.googlesource.com/3062971

ap...@google.com <ap...@google.com> #5Apr 30, 2024 07:48PM

Project: platform/frameworks/support
Branch: androidx-main

commit 2d1785e9837b97c6d64878413efb96d873895cd0
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Wed Apr 24 16:57:02 2024

Move method tracing to a separate phase.

* Method tracing now runs as a separate phase, without interfering with metrics.

Fixes: 285912360
Fixes: 336588271
Test: Existing unit tests.
Change-Id: If9a504a7bce7228840339b34294ba3fdf98aceeb
Relnote: Method tracing runs as a seperate phase during a Macrobenchmark, and it no longer affects measurements.

M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Macrobenchmark.kt
A benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/MacrobenchmarkPhase.kt

https://android-review.googlesource.com/3059411

pr...@google.com <pr...@google.com> #6May 14, 2024 05:22PM

The following release(s) address this bug.It is possible this bug has only been partially addressed:

androidx.benchmark:benchmark-macro:1.3.0-alpha05

Issue 336588271