After deeply use macrobenchmark , I find lots of pitfall in this test framework [306235276]

Assigned

Bug

Status Update

No update yet.

Description

dv...@gmail.com

created issue #1

Oct 19, 2023 08:48AM

because of lag or some reasons, unexpected trace data will be recorded by perfetto. If we don't choose slice in measureBlock only, there maybe dirty data in metric.

refer to first screenshot, that's drag down systemui bar, and swip back to launcher trace.

some metrics don't work FrameTimingMetric doesn't work because of the way of getting frameid from slice name is wrong.
can't control what kind of datasouces which perfetto server to record. Some datasources are added by ODM, we can't add datasource to record.
lack of web application to show result of benchmark test.
there is no community to talk each other , to slove those problems.

Thu Oct 19 2023 16:42:29 GMT+0800 (中国标准时间).png

295 KB

View

Download

Comments

cc...@google.com <cc...@google.com> #2Oct 19, 2023 07:48PM

Project: platform/frameworks/support
Branch: androidx-master-dev

commit c60f33e229e31ab328ef6b59dab63b264954831c
Author: Alexandre Elias <aelias@google.com>
Date: Fri Jul 10 16:23:09 2020

Semantics no-op cleanups

Partly in response to lmr's broad code review, I did a pass of
superficial API/implementation cleanup. The main changes are:

- I changed each Boolean SemanticsProperty where false is equivalent to
not being present to take "Unit" instead. This is conceptually
clearer: it avoids questions like "can I cancel out a semantics from a
merged child by setting it to false?" Because "property = Unit" looks
weird, I also changed the style of these to "property()".

- I moved the Semantics id generator closer to where it's used, in
SemanticsModifierCore. I made it internal and an AtomicInt.
(Note that integer ids are heavily used in the Android
AccessibilityNodeInfo APIs so I can't simply remove them entirely.)

- I deleted dead code. Some examples include SemanticsHintOverrides,
a public API not connected to anything, and SemanticsPropertyKey
merge() open method which is never called. (In both cases I have
a different plan in mind for accessibility.)

Fixes: 145951226
Fixes: 145955412
Test: existing tests
Relnote: "Single-value semantics properties now use a calling style.
For example, 'semantics { hidden = true }' is now written as:
'semantics { hidden() }'."

Change-Id: Ic1afd12ea22c926babc9662f1804d80b33aa0cfc

M ui/integration-tests/benchmark/src/androidTest/java/androidx/ui/benchmark/test/LayoutNodeModifierBenchmark.kt
M ui/ui-core/api/0.1.0-dev15.txt
M ui/ui-core/api/current.txt
M ui/ui-core/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-core/api/public_plus_experimental_current.txt
M ui/ui-core/api/restricted_0.1.0-dev15.txt
M ui/ui-core/api/restricted_current.txt
M ui/ui-core/src/androidAndroidTest/kotlin/androidx/ui/graphics/vector/VectorTest.kt
M ui/ui-core/src/androidAndroidTest/kotlin/androidx/ui/semantics/SemanticsTests.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidActuals.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidComposeView.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidComposeViewAccessibilityDelegateCompat.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidPopup.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/Expect.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsConfiguration.kt
D ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsHintOverrides.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsModifier.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsNode.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsOwner.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsWrapper.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/semantics/SemanticsProperties.kt
M ui/ui-foundation/api/0.1.0-dev15.txt
M ui/ui-foundation/api/current.txt
M ui/ui-foundation/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-foundation/api/public_plus_experimental_current.txt
M ui/ui-foundation/api/restricted_0.1.0-dev15.txt
M ui/ui-foundation/api/restricted_current.txt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Clickable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Dialog.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Scroller.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/selection/Selectable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/selection/Toggleable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/semantics/FoundationSemanticsProperties.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ButtonTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/CardTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/CheckboxScreenshotTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/RadioButtonScreenshotTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ScaffoldTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/SnackbarTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/SurfaceTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ripple/RippleIndicationTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/textfield/TextFieldScreenshotTest.kt
M ui/ui-material/src/main/java/androidx/ui/material/AppBar.kt
M ui/ui-material/src/main/java/androidx/ui/material/TextFieldImpl.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/AssertsTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/CallSemanticsActionTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/ErrorMessagesTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/FindersTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/PrintToStringTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/ScrollToTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/TextActionsTest.kt
M ui/ui-test/src/main/java/androidx/ui/test/Actions.kt
M ui/ui-test/src/main/java/androidx/ui/test/Filters.kt
M ui/ui-text/api/0.1.0-dev15.txt
M ui/ui-text/api/current.txt
M ui/ui-text/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-text/api/public_plus_experimental_current.txt
M ui/ui-text/api/restricted_0.1.0-dev15.txt
M ui/ui-text/api/restricted_current.txt
M ui/ui-text/src/commonMain/kotlin/androidx/ui/text/CoreTextField.kt
M ui/ui-text/src/commonMain/kotlin/androidx/ui/text/TextSemanticsProperties.kt

https://android-review.googlesource.com/1360099

dv...@gmail.com <dv...@gmail.com> #3Nov 14, 2023 12:56PM

about lag in perfetto Sorry , I forget to tell the background I use Macrobenchmark for.

I am in device maker company, I'm using this macrobenchmark framework to test system performance.

When the system get slow, perfetto will not start as meant to be, there'll a lag between the time when marobenchmark starts perfetto and the time when perfetto trace really starts recording.

trace file please see the screenshot and trace file I upload
data sources I'd like to add in perfetto config

because some jank frame doesn't matter, I only interested user percepted jank frame.

device maker add some trace to record user percepted jank fame in the trace

I need open it , so that I can read in the trace by sql querying.

thanks , I will look it.

Tue Nov 14 2023 20:51:22 GMT+0800 (中国标准时间).png

50 KB

View

Download

systrace that framemetric can't work.perfetto_trace.gz

65 MB

Download

dv...@gmail.com <dv...@gmail.com> #4Nov 20, 2023 03:05AM

is there any more detail explaination about https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:development/plot-benchmarks/

what does P button mean?

what does x axis mean for sampled metric and single value metric?

how does it normalize metrics? it seems y values are not between 0 and 1 after enable normalized metrics.

Message last modified on Nov 20, 2023 03:06AM

Mon Nov 20 2023 11:02:56 GMT+0800 (中国标准时间).jpeg

21 KB

View

Download

dv...@gmail.com <dv...@gmail.com> #5Nov 20, 2023 03:54AM

what does this picture for ?

Mon Nov 20 2023 11:54:09 GMT+0800 (中国标准时间).png

69 KB

View

Download

ap...@google.com <ap...@google.com> #6Dec 7, 2023 09:53PM

Project: platform/frameworks/support
Branch: androidx-main

commit ec13e616312e4026429dfa6a86b820725f30e2be
Author: Chris Craik <ccraik@google.com>
Date: Wed Dec 06 09:59:35 2023

Fix FrameId parsing in FrameTimingMetric

Fixes: 303823815
Bug: 306235276
Test: SliceTest
Test: FrameTimingQueryTest
Relnote: "Fixed crash in FrameTimingMetric failing to parse frame id,
especially on certain OEM devices."

Change-Id: Ia24bc3c90b317252f579cc2425547f034ef96ef8

A benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/perfetto/SliceTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/FrameTimingQuery.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/perfetto/Slice.kt

https://android-review.googlesource.com/2862728

dv...@gmail.com <dv...@gmail.com> #7Dec 12, 2023 09:16AM

thank you

Issue 306235276

Description

Issue summary

Comments

cc...@google.com <cc...@google.com> #2Oct 19, 2023 07:48PM

dv...@gmail.com <dv...@gmail.com> #3Nov 14, 2023 12:56PM

dv...@gmail.com <dv...@gmail.com> #4Nov 20, 2023 03:05AM

dv...@gmail.com <dv...@gmail.com> #5Nov 20, 2023 03:54AM

ap...@google.com <ap...@google.com> #6Dec 7, 2023 09:53PM

dv...@gmail.com <dv...@gmail.com> #7Dec 12, 2023 09:16AM

Add comment

Issue metadata