Macrobench startup traces cutting off too early, failing to capture metric [193827052]

Duplicate

of 194105203

Bug

Status Update

No update yet.

Description

cc...@google.com

created issue #1

Jul 15, 2021 10:14PM

Seeing traces cutting off too early, before e.g. startActivityAndWait have fully completed, which results in an exception like the following:

java.lang.IllegalStateException: Error, different metrics observed in different iterations.

Iteration 1 missing keys [startupMs]Please report a bug, and include a logcat capture, and all traces captured by this test run:
/storage/emulated/0/Android/media/androidx.benchmark.integration.macrobenchmark.test/TrivialStartupBenchmark_startup[startup=COLD,compilation=SpeedProfile[iterations=3]]_iter000_2021-07-15-22-21-37.trace
/storage/emulated/0/Android/media/androidx.benchmark.integration.macrobenchmark.test/TrivialStartupBenchmark_startup[startup=COLD,compilation=SpeedProfile[iterations=3]]_iter001_2021-07-15-22-21-41.trace
/storage/emulated/0/Android/media/androidx.benchmark.integration.macrobenchmark.test/TrivialStartupBenchmark_startup[startup=COLD,compilation=SpeedProfile[iterations=3]]_iter002_2021-07-15-22-21-44.trace
DeviceInfo(Brand=google, Model=Pixel 4a (5G), SDK=30, BuildFp=google/bramble/bramble:11/RQ3A.210605.003/7295351:userdebug/dev-keys)
...

See attached trace.

Seeing this in CI frequently with androidx.benchmark.integration.macrobenchmark.SmallListStartupBenchmark, e.g. https://android-build.googleplex.com/builds/tests/view?invocationId=I51900009972024845&redirect=http%3A%2F%2Fab%2FI51900009972024845&testClass=SmallListStartupBenchmark&testMethod=startup%5Bstartup%3DHOT%2Ccompilation%3DNone%5D&testPackage=androidx.benchmark.integration.macrobenchmark

This is likely due to switch either to am start (http://aosp/1751627), or reduction of sleep before killing the trace (http://aosp/1751631).

nometric.trace

3 MB

Download

Comments

ap...@google.com <ap...@google.com> #2Jul 16, 2021 05:36PM

Marked as fixed.

Project: platform/frameworks/support
Branch: androidx-master-dev

commit c60f33e229e31ab328ef6b59dab63b264954831c
Author: Alexandre Elias <aelias@google.com>
Date: Fri Jul 10 16:23:09 2020

Semantics no-op cleanups

Partly in response to lmr's broad code review, I did a pass of
superficial API/implementation cleanup. The main changes are:

- I changed each Boolean SemanticsProperty where false is equivalent to
not being present to take "Unit" instead. This is conceptually
clearer: it avoids questions like "can I cancel out a semantics from a
merged child by setting it to false?" Because "property = Unit" looks
weird, I also changed the style of these to "property()".

- I moved the Semantics id generator closer to where it's used, in
SemanticsModifierCore. I made it internal and an AtomicInt.
(Note that integer ids are heavily used in the Android
AccessibilityNodeInfo APIs so I can't simply remove them entirely.)

- I deleted dead code. Some examples include SemanticsHintOverrides,
a public API not connected to anything, and SemanticsPropertyKey
merge() open method which is never called. (In both cases I have
a different plan in mind for accessibility.)

Fixes: 145951226
Fixes: 145955412
Test: existing tests
Relnote: "Single-value semantics properties now use a calling style.
For example, 'semantics { hidden = true }' is now written as:
'semantics { hidden() }'."

Change-Id: Ic1afd12ea22c926babc9662f1804d80b33aa0cfc

M ui/integration-tests/benchmark/src/androidTest/java/androidx/ui/benchmark/test/LayoutNodeModifierBenchmark.kt
M ui/ui-core/api/0.1.0-dev15.txt
M ui/ui-core/api/current.txt
M ui/ui-core/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-core/api/public_plus_experimental_current.txt
M ui/ui-core/api/restricted_0.1.0-dev15.txt
M ui/ui-core/api/restricted_current.txt
M ui/ui-core/src/androidAndroidTest/kotlin/androidx/ui/graphics/vector/VectorTest.kt
M ui/ui-core/src/androidAndroidTest/kotlin/androidx/ui/semantics/SemanticsTests.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidActuals.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidComposeView.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidComposeViewAccessibilityDelegateCompat.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidPopup.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/Expect.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsConfiguration.kt
D ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsHintOverrides.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsModifier.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsNode.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsOwner.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsWrapper.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/semantics/SemanticsProperties.kt
M ui/ui-foundation/api/0.1.0-dev15.txt
M ui/ui-foundation/api/current.txt
M ui/ui-foundation/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-foundation/api/public_plus_experimental_current.txt
M ui/ui-foundation/api/restricted_0.1.0-dev15.txt
M ui/ui-foundation/api/restricted_current.txt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Clickable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Dialog.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Scroller.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/selection/Selectable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/selection/Toggleable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/semantics/FoundationSemanticsProperties.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ButtonTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/CardTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/CheckboxScreenshotTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/RadioButtonScreenshotTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ScaffoldTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/SnackbarTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/SurfaceTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ripple/RippleIndicationTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/textfield/TextFieldScreenshotTest.kt
M ui/ui-material/src/main/java/androidx/ui/material/AppBar.kt
M ui/ui-material/src/main/java/androidx/ui/material/TextFieldImpl.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/AssertsTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/CallSemanticsActionTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/ErrorMessagesTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/FindersTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/PrintToStringTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/ScrollToTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/TextActionsTest.kt
M ui/ui-test/src/main/java/androidx/ui/test/Actions.kt
M ui/ui-test/src/main/java/androidx/ui/test/Filters.kt
M ui/ui-text/api/0.1.0-dev15.txt
M ui/ui-text/api/current.txt
M ui/ui-text/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-text/api/public_plus_experimental_current.txt
M ui/ui-text/api/restricted_0.1.0-dev15.txt
M ui/ui-text/api/restricted_current.txt
M ui/ui-text/src/commonMain/kotlin/androidx/ui/text/CoreTextField.kt
M ui/ui-text/src/commonMain/kotlin/androidx/ui/text/TextSemanticsProperties.kt

https://android-review.googlesource.com/1360099

cc...@google.com <cc...@google.com> Jul 20, 2021 05:58PM

Status: Assigned (reopened)

ap...@google.com <ap...@google.com> #3Jul 28, 2021 01:21AM

Project: platform/frameworks/support
Branch: androidx-main

commit 91d9b082ebc3265cdd568e69a1920881602f325a
Author: Chris Craik <ccraik@google.com>
Date: Tue Jul 27 17:15:46 2021

Workaround missing metrics

Bug: 193827052
Test: MetricResultExtensionsTest

Change-Id: I9bf08b13aeb5f362a7aee7f2a5adc818240c2d3c

M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MetricResultExtensionsTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/MetricResultExtensions.kt

https://android-review.googlesource.com/1780807

ap...@google.com <ap...@google.com> #4Oct 13, 2021 12:08AM

Project: platform/frameworks/support
Branch: androidx-main

commit 58159e0dc33a81efa1179e35d50a2ded001dd561
Author: Chris Craik <ccraik@google.com>
Date: Tue Oct 12 15:33:01 2021

Poll /sys/kernel/tracing/tracing_on before returning from trace capture start

Test: PerfettoCaptureTest
Bug: 193827052
Bug: 194105203
Bug: 200302931

Relnote: """Fix for traces not starting early enough, and missing
metric data. This is expected to fix "Unable to read any metrics
during benchmark" exceptions that were caused by library itself."

Ran PerfettoCaptureTest hundreds of times in a row on a old device
(shamu API 24), the sleep() at the beginning no longer is required, as
it's handled by the polling.

From Perfetto team (

b/194105203

)- tracing_on is always a better way to
ensure that new trace events won't be missing from the trace.

Additionally, changed PerfettoCaptureTest to clarify between begin/end
of trace content dropping problems.

Change-Id: I6dfcbda59c6f5887433f3b1227613f8789ca6307

M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoHelper.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoCaptureTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessor.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Shell.kt

https://android-review.googlesource.com/1855765

cc...@google.com <cc...@google.com> Nov 3, 2021 06:15PM

Status: Duplicate of

194105203

Issue 193827052