Assigned
Status Update
Comments
ap...@google.com <ap...@google.com> #2
Project: platform/frameworks/support
Branch: androidx-main
Author: Chris Craik <
Link:
Reimplement BenchmarkRule / BenchmarkState on top of coroutines
Expand for full commit details
Reimplement BenchmarkRule / BenchmarkState on top of coroutines
Test: benchmark-common, benchmark-junit4
Bug: 389149423
Fixes: 311242861
This refactor achieves several important goals:
- Move from the extremely complex stateful nature of BenchmarkState to layered suspending functions (the old state deferral system was very obtuse and brittle)
- Simplify the existing measureRepeatedOnMainThread yield behavior to use actual yield()
- Rewrite the thermal throttling sleep to yield() instead of sleep() (which works around ANRs when using measureRepeatedOnMainThread)
- Provide a clear config -> execution -> output control flow, including a clear top level measureRepeated function (which declares all its inputs, and almost wires up the outputs)
- Move as much logic as possible from the JUnit4 layer to benchmark-common
These will indirectly make it more possible to:
- Run trace post-processing within microbenchmark (e.g. interference detection and retry)
- More cleanly add configuration in MicrobenchmarkConfig (similar to what warmupCount / repeatCount BenchmarkState constructor params were previously trying to offer)
- Move benchmark-common to KMP if desired
- Write suspending variants of the benchmark APIs
The downsides are:
- Benchmarks with the `BenchmarkRule.getState().keepRunning()` pattern will no longer capture Perfetto traces
- In a follow up, a replacement API will be offered for Java that more closely matches kotlin
- BenchmarkState is implemented on top of a complex and custom coroutine logic mimicking a suspending sequence
- Ideally this implementation will be deprecated and not have to change much in the future
- Several experimental BenchmarkState APIs were removed, to make way for the new way of solving these problems (configuring benchmarks)
- The most useful of these moved to MicrobenchmarkConfig
The old BenchmarkRule/BenchmarkState are temporarily moved to internal
BenchmarkRuleLegacy/BenchmarkStateLegacy to enable easy internal
comparisons in the short term. Eventually these will be removed.
Relnote: "Refactored BenchmarkRule to be built on top of coroutines, and support better yield() behavior. This refactor removed several experimental BenchmarkState APIs, but will be followed by replacements as needed. Additionally, added runWithMeasurementDisabled to clarify behavior (all measurement is paused). In the future, runWithTimingDisabled will be deprecated."
Change-Id: I19837fc604b9e308957fc01ed009c0e921b8fe28
Files:
- M
benchmark/benchmark-common/api/current.txt
- M
benchmark/benchmark-common/api/restricted_current.txt
- M
benchmark/benchmark-common/build.gradle
- M
benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/BenchmarkStateLegacyConfigTest.kt
- M
benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/BenchmarkStateLegacyTest.kt
- M
benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/MicrobenchmarkPhaseConfigTest.kt
- M
benchmark/benchmark-common/src/main/java/androidx/benchmark/Arguments.kt
- M
benchmark/benchmark-common/src/main/java/androidx/benchmark/BenchmarkState.kt
- A
benchmark/benchmark-common/src/main/java/androidx/benchmark/BenchmarkStateLegacy.kt
- A
benchmark/benchmark-common/src/main/java/androidx/benchmark/Microbenchmark.kt
- M
benchmark/benchmark-common/src/main/java/androidx/benchmark/MicrobenchmarkConfig.kt
- A
benchmark/benchmark-common/src/main/java/androidx/benchmark/MicrobenchmarkOutput.kt
- M
benchmark/benchmark-common/src/main/java/androidx/benchmark/MicrobenchmarkPhase.kt
- M
benchmark/benchmark-common/src/main/java/androidx/benchmark/Profiler.kt
- A
benchmark/benchmark-common/src/main/java/androidx/benchmark/TestDefinition.kt
- M
benchmark/benchmark-common/src/main/java/androidx/benchmark/ThrottleDetector.kt
- M
benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoCaptureWrapper.kt
- M
benchmark/benchmark-junit4/api/current.txt
- M
benchmark/benchmark-junit4/api/restricted_current.txt
- M
benchmark/benchmark-junit4/src/androidTest/java/androidx/benchmark/junit4/BenchmarkRuleAnnotationTest.kt
- A
benchmark/benchmark-junit4/src/androidTest/java/androidx/benchmark/junit4/BenchmarkRuleLegacyTest.kt
- M
benchmark/benchmark-junit4/src/androidTest/java/androidx/benchmark/junit4/BenchmarkRuleTest.kt
- M
benchmark/benchmark-junit4/src/main/java/androidx/benchmark/junit4/BenchmarkRule.kt
- A
benchmark/benchmark-junit4/src/main/java/androidx/benchmark/junit4/BenchmarkRuleLegacy.kt
- M
benchmark/benchmark/src/androidTest/java/androidx/benchmark/benchmark/BenchmarkStateBenchmark.kt
- A
benchmark/benchmark/src/androidTest/java/androidx/benchmark/benchmark/BenchmarkStateLegacyBenchmark.kt
- A
benchmark/benchmark/src/androidTest/java/androidx/benchmark/benchmark/MeasureRepeatedSampleBenchmark.kt
- A
benchmark/benchmark/src/androidTest/java/androidx/benchmark/benchmark/TrivialKotlinBenchmarkLegacy.kt
- M
compose/runtime/runtime/compose-runtime-benchmark/src/androidTest/java/androidx/compose/runtime/benchmark/ComposeBenchmarkBase.kt
Hash: 329bd0d25247f1eb6c683b921b8cd06f60bf3ad2
Date: Fri Jul 12 15:09:22 2024
cc...@google.com <cc...@google.com> #3
Shorter term follow up tasks:
- @Deprecate
runWithTimingDisabled
(not done in this CL to limit file changes) - Add more clarifying docs for new BenchmarkState implementation (in code comments as well as
)in the arch doc - Remove BenchmarkRuleLegacy / BenchmarkStateLegacy (after confirmed no significant regressions)
Longer term:
- Add replacements for removed BenchmarkState experimental APIs in MicrobenchmarkConfig (and probably need a MicrobenchmarkOutput)
ap...@google.com <ap...@google.com> #4
Project: platform/frameworks/support
Branch: androidx-main
Author: Chris Craik <
Link:
Deprecate runWithTimingDisabled in favor of runWithMeasurementDisabled
Expand for full commit details
Deprecate runWithTimingDisabled in favor of runWithMeasurementDisabled
Bug: 389149423
Fixes: 149979716
Test: Existing / presubmit
Opens up MicrobenchmarkScope since that's necessary to expose
runWithMeasurementDisabled properly. Inline functions of a restricted
superclass cannot be exposed by a subclass.
Relnote: "Deprecate runWithTimingDisabled in favor of runWithMeasurementDisabled, which more clearly describes the behavior - all metrics are paused. Additionally, expose the MicrobenchmarkScope superclass since redeclaring the runWithMeasurementDisabled function to open access isn't possible, since it's inline."
Change-Id: I9e23b0dfcff18f162ca0d2517734f3038870b59c
Files:
- M
benchmark/benchmark-common/api/current.txt
- M
benchmark/benchmark-common/api/restricted_current.txt
- M
benchmark/benchmark-common/src/main/java/androidx/benchmark/Microbenchmark.kt
- M
benchmark/benchmark-common/src/main/java/androidx/benchmark/MicrobenchmarkPhase.kt
- M
benchmark/benchmark-junit4/api/current.txt
- M
benchmark/benchmark-junit4/api/restricted_current.txt
- M
benchmark/benchmark-junit4/src/main/java/androidx/benchmark/junit4/BenchmarkRule.kt
- M
benchmark/benchmark-junit4/src/main/java/androidx/benchmark/junit4/BenchmarkRuleLegacy.kt
- M
benchmark/benchmark/src/androidTest/java/androidx/benchmark/benchmark/CpuEventCounterBenchmark.kt
- M
benchmark/benchmark/src/androidTest/java/androidx/benchmark/benchmark/PerfettoOverheadBenchmark.kt
- M
benchmark/benchmark/src/androidTest/java/androidx/benchmark/benchmark/PerfettoSdkOverheadBenchmark.kt
- M
benchmark/benchmark/src/androidTest/java/androidx/benchmark/benchmark/SampleCustomBenchmark.kt
- M
benchmark/benchmark/src/androidTest/java/androidx/benchmark/benchmark/SleepyBenchmark.kt
- M
benchmark/samples/src/main/java/androidx/benchmark/samples/BenchmarkRuleSample.kt
- M
compose/benchmark-utils/src/main/java/androidx/compose/testutils/benchmark/BenchmarkFirstExtensions.kt
- M
compose/benchmark-utils/src/main/java/androidx/compose/testutils/benchmark/BenchmarksExtensions.kt
- M
compose/foundation/foundation/benchmark/src/androidTest/java/androidx/compose/foundation/benchmark/ScrollerBenchmark.kt
- M
compose/foundation/foundation/benchmark/src/androidTest/java/androidx/compose/foundation/benchmark/focusable/FocusableInLazyListBenchmark.kt
- M
compose/foundation/foundation/benchmark/src/androidTest/java/androidx/compose/foundation/benchmark/lazy/LazyBenchmarkCommon.kt
- M
compose/foundation/foundation/benchmark/src/androidTest/java/androidx/compose/foundation/benchmark/lazy/LazySwitchingStatesBenchmark.kt
- M
compose/foundation/foundation/benchmark/src/androidTest/java/androidx/compose/foundation/benchmark/lazy/ReuseBenchmark.kt
- M
compose/foundation/foundation/benchmark/src/androidTest/java/androidx/compose/foundation/benchmark/text/empirical/baselines/StaticLayoutBaseline.kt
- M
compose/integration-tests/macrobenchmark/src/main/java/androidx/compose/integration/macrobenchmark/TrivialStartupBenchmark.kt
- M
compose/material/material-ripple/benchmark/src/androidTest/java/androidx/compose/material/ripple/benchmark/RippleBenchmark.kt
- M
compose/material/material/benchmark/src/androidTest/java/androidx/compose/material/benchmark/view/AndroidTextViewBenchmark.kt
- M
compose/material3/benchmark/src/androidTest/java/androidx/compose/material3/benchmark/MaterialBenchmarkExtensions.kt
- M
compose/runtime/runtime/compose-runtime-benchmark/src/androidTest/java/androidx/compose/runtime/benchmark/ComposeBenchmarkBase.kt
- M
compose/runtime/runtime/compose-runtime-benchmark/src/androidTest/java/androidx/compose/runtime/benchmark/KotlinCoroutinesBenchmark.kt
- M
compose/runtime/runtime/compose-runtime-benchmark/src/androidTest/java/androidx/compose/runtime/benchmark/SnapshotStateObserverBenchmark.kt
- M
compose/runtime/runtime/compose-runtime-benchmark/src/androidTest/java/androidx/compose/runtime/benchmark/collection/MutableVectorBenchmark.kt
- M
compose/ui/ui-text/benchmark/src/androidTest/java/androidx/compose/ui/text/benchmark/FrameworkTextLayoutBenchmark.kt
- M
compose/ui/ui-text/benchmark/src/androidTest/java/androidx/compose/ui/text/benchmark/NonLinearFontScalingBenchmark.kt
- M
compose/ui/ui-text/benchmark/src/androidTest/java/androidx/compose/ui/text/benchmark/ParagraphBenchmark.kt
- M
compose/ui/ui-text/benchmark/src/androidTest/java/androidx/compose/ui/text/benchmark/ParagraphWithLineHeightBenchmark.kt
- M
compose/ui/ui-text/benchmark/src/androidTest/java/androidx/compose/ui/text/benchmark/font/PlatformFontLookup.kt
- M
compose/ui/ui-text/benchmark/src/androidTest/java/androidx/compose/ui/text/benchmark/input/EditProcessorBenchmark.kt
- M
compose/ui/ui-text/benchmark/src/androidTest/java/androidx/compose/ui/text/benchmark/input/GapBufferBenchmark.kt
- M
compose/ui/ui/benchmark/src/androidTest/java/androidx/compose/ui/benchmark/BenchmarkModifierExtensions.kt
- M
compose/ui/ui/benchmark/src/androidTest/java/androidx/compose/ui/benchmark/LifecycleAwareWindowRecomposerBenchmark.kt
- M
compose/ui/ui/benchmark/src/androidTest/java/androidx/compose/ui/benchmark/NestedScrollingBenchmark.kt
- M
compose/ui/ui/benchmark/src/androidTest/java/androidx/compose/ui/benchmark/accessibility/TextAccessibilityBenchmark.kt
- M
compose/ui/ui/benchmark/src/androidTest/java/androidx/compose/ui/benchmark/accessibility/TextFieldAccessibilityBenchmark.kt
- M
compose/ui/ui/benchmark/src/androidTest/java/androidx/compose/ui/benchmark/focus/ParameterizedFocusBenchmark.kt
- M
compose/ui/ui/benchmark/src/androidTest/java/androidx/compose/ui/benchmark/spatial/RectListBenchmark.kt
- M
datastore/datastore-benchmark/src/androidTest/java/androidx/datastore/core/MultiProcessDataStoreTest.kt
- M
datastore/datastore-benchmark/src/androidTest/java/androidx/datastore/core/SingleProcessDatastoreTest.kt
- M
emoji2/emoji2-benchmark/src/androidTest/java/androidx/emoji2/benchmark/text/CachedEmojiCompatInitBenchmark.kt
- M
emoji2/emoji2-benchmark/src/androidTest/java/androidx/emoji2/benchmark/text/EmojiProcessBenchmark.kt
- M
profileinstaller/profileinstaller-benchmark/src/androidTest/java/androidx/profileinstaller/benchmark/ProfileInstallerTranscodeBenchmark.kt
- M
room/benchmark/src/androidTest/java/androidx/room/benchmark/AmbiguousColumnResolverBenchmark.kt
- M
wear/compose/compose-foundation/benchmark/src/androidTest/java/androidx/wear/compose/foundation/benchmark/ScalingLazyColumnBenchmark.kt
- M
wear/compose/compose-material/benchmark/src/androidTest/java/androidx/wear/compose/material/benchmark/PositionIndicatorBenchmark.kt
- M
wear/compose/compose-material/benchmark/src/androidTest/java/androidx/wear/compose/material/benchmark/SLCBasedBenchmarkHelper.kt
- M
work/work-benchmark/src/androidTest/java/androidx/work/benchmark/InitializeBenchmark.kt
Hash: f95ae3f3d30cb7486c27a0857856dbfd15920c3a
Date: Wed Jan 22 16:04:15 2025
gr...@google.com <gr...@google.com> #5
Good news! Wear Compose has been updated to ab/12983551 in https://critique.corp.google.com/cl/725560567 and this bug has been affected.
Description
Currently, there are several problems with the architecture of BenchmarkState/BenchmarkRule:
measureRepeatedOnMainThread()
is awkward, and happens within the measurement loopThis bug tracks an effort to rewrite much of the top level control flow of BenchmarkRule / BenchmarkState to be based on coroutines, and address all the above problems.