Fixed
Status Update
No update yet.
Comments
de...@tokopedia.com
Update, this is intermittent error when running macrobenchmark for my app. And I don't know if it is a coincidence or not: if the generated perfetto-trace file exceed 15mb file size, it will not contains `android_startup` field for my application package, therefore it will causing the above error message. The number of result in .json file (which exist in startupMs -> runs) also equals with the number of perfetto-trace file that not exceed 15mb file size.
cc...@google.com
(Sorry for the late reply, was OOO for a while)
Between alpha02 and alpha07 we significantly improved iteration speed by tightening up a lot of timing throughout the library, but I expect this is an instance of a race condition in starting trace capture, which we're investigating on the trace capture side (internal bug
Few questions:
- Are you using
MacrobenchmarkScope.pressHome()before eachstartActivityAndWait()? This shouldn't be strictly necessary, but the built in sleep() seems to be how we workaround the problem in a lot of internal macrobenchmarks. - Can you try adding a Thread.sleep(500) to see if it works around the problem?
- Can you attach a trace from a failed iteration? The path is printed in logcat with the tag "PerfettoCapture". This would clarify if it's the beginning or end of the trace that's getting cut off.
ap...@google.com
Project: platform/frameworks/support
Branch: androidx-main
commit 58159e0dc33a81efa1179e35d50a2ded001dd561
Author: Chris Craik <ccraik@google.com>
Date: Tue Oct 12 15:33:01 2021
Poll /sys/kernel/tracing/tracing_on before returning from trace capture start
Test: PerfettoCaptureTest
Bug: 193827052
Bug: 194105203
Bug: 200302931
Relnote: """Fix for traces not starting early enough, and missing
metric data. This is expected to fix "Unable to read any metrics
during benchmark" exceptions that were caused by library itself."
Ran PerfettoCaptureTest hundreds of times in a row on a old device
(shamu API 24), the sleep() at the beginning no longer is required, as
it's handled by the polling.
From Perfetto team (
ensure that new trace events won't be missing from the trace.
Additionally, changed PerfettoCaptureTest to clarify between begin/end
of trace content dropping problems.
Change-Id: I6dfcbda59c6f5887433f3b1227613f8789ca6307
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoHelper.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoCaptureTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessor.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Shell.kt
https://android-review.googlesource.com/1855765
Branch: androidx-main
commit 58159e0dc33a81efa1179e35d50a2ded001dd561
Author: Chris Craik <ccraik@google.com>
Date: Tue Oct 12 15:33:01 2021
Poll /sys/kernel/tracing/tracing_on before returning from trace capture start
Test: PerfettoCaptureTest
Bug: 193827052
Bug: 194105203
Bug: 200302931
Relnote: """Fix for traces not starting early enough, and missing
metric data. This is expected to fix "Unable to read any metrics
during benchmark" exceptions that were caused by library itself."
Ran PerfettoCaptureTest hundreds of times in a row on a old device
(shamu API 24), the sleep() at the beginning no longer is required, as
it's handled by the polling.
From Perfetto team (
ensure that new trace events won't be missing from the trace.
Additionally, changed PerfettoCaptureTest to clarify between begin/end
of trace content dropping problems.
Change-Id: I6dfcbda59c6f5887433f3b1227613f8789ca6307
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoHelper.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoCaptureTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessor.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Shell.kt
cc...@google.com
I expect the above change to fix the problem (or at minimum drastically reduce frequency). Marking as fixed, though this won't go out until the release after alpha09 (scheduled for tomorrow), likely to be beta01.
Marking as fixed, but if you still see in beta01 please comment and I'll keep investigating. Workarounds suggested in
c....@gmail.com
Not strictly related to this bug itself, but when following the instructions on androidx.benchmark:benchmark-macro-junit4:1.1.0-rc03, this issue still surfaces as the auto-generated sample benchmark uses pressHome() before startActivityAndWait(). Removing pressHome() solves the issue
ni...@gmail.com
implementation 'androidx.benchmark:benchmark-macro-junit4:1.1.0-rc03'
java.lang.IllegalArgumentException: Unable to read any metrics during benchmark (metric list: [androidx.benchmark.macro.StartupTimingMetric@b851af5]).
Check that you're performing the operations to be measured. For example, if
using StartupTimingMetric, are you starting an activity for the specified package
in the measure block?
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:237)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:301)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:106)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated$default(MacrobenchmarkRule.kt:96)
at com.nitin.benchmark.ExampleStartupBenchmark.startup(ExampleStartupBenchmark.kt:33)
java.lang.IllegalArgumentException: Unable to read any metrics during benchmark (metric list: [androidx.benchmark.macro.StartupTimingMetric@b851af5]).
Check that you're performing the operations to be measured. For example, if
using StartupTimingMetric, are you starting an activity for the specified package
in the measure block?
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:237)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:301)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:106)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated$default(MacrobenchmarkRule.kt:96)
at com.nitin.benchmark.ExampleStartupBenchmark.startup(ExampleStartupBenchmark.kt:33)
ni...@gmail.com
it is working onwards android 10
va...@gmail.com
Getting this error on androidx.benchmark:benchmark-macro-junit4:1.1.0,
things already tried:
1. removing pressHome()
2. applied Thread.sleep(5000) before startActivityAndWait()
Error details:
> Task :macro-benchmark:connectedReleaseAndroidTest
Starting 2 tests on SM-F415F - 11
Timed out waiting for process (com.example.benchmark) to appear on samsung-sm_f41
..
..
java.lang.IllegalArgumentException: Unable to read any metrics during benchmark (metric list: [androidx.benchmark.macro.StartupTimingMetric@eb8ded8]).
Check that you're performing the operations to be measured. For example, if
using StartupTimingMetric, are you starting an activity for the specified package
in the measure block?
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:237)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:301)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:106)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated$default(MacrobenchmarkRule.kt:96)
at com.example.benchmark.StartupBenchmark.startup(StartupBenchmark.kt:56)
at com.example.benchmark.StartupBenchmark.startupCompilationPartial(StartupBenchmark.kt:51)
things already tried:
1. removing pressHome()
2. applied Thread.sleep(5000) before startActivityAndWait()
Error details:
> Task :macro-benchmark:connectedReleaseAndroidTest
Starting 2 tests on SM-F415F - 11
Timed out waiting for process (com.example.benchmark) to appear on samsung-sm_f41
..
..
java.lang.IllegalArgumentException: Unable to read any metrics during benchmark (metric list: [androidx.benchmark.macro.StartupTimingMetric@eb8ded8]).
Check that you're performing the operations to be measured. For example, if
using StartupTimingMetric, are you starting an activity for the specified package
in the measure block?
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:237)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:301)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:106)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated$default(MacrobenchmarkRule.kt:96)
at com.example.benchmark.StartupBenchmark.startup(StartupBenchmark.kt:56)
at com.example.benchmark.StartupBenchmark.startupCompilationPartial(StartupBenchmark.kt:51)
ma...@gmail.com
Yesterday it was working without any issues.
Today I got the above-mentioned error.
What I did to make it work:
1. Removed the app from the real device
2. Clean the project
3. Make sure that the `build` folder inside the benchmark model is totally removed. If not, remove it manually.
Then start the benchmark.
Today I got the above-mentioned error.
What I did to make it work:
1. Removed the app from the real device
2. Clean the project
3. Make sure that the `build` folder inside the benchmark model is totally removed. If not, remove it manually.
Then start the benchmark.
ma...@propertyfinder.ae
It happed again, I restarted the android device and it worked.
ro...@gmail.com
Comment has been deleted.
Description
There was 1 failure:
1) macroBenchmarkHomeLaunchTime[mode=COLD](com.testapp.macrobenchmark.test.home.HomeStartupBenchmark)
java.lang.IllegalArgumentException: Unable to read any metrics during benchmark (metric list: [androidx.benchmark.macro.StartupTimingMetric@1fce2f9]).
Check that you're performing the operations to be measured. For example, if
using StartupTimingMetric, are you starting an activity for the specified package
in the measure block?
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:198)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:259)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:68)
at com.testapp.macrobenchmark.util.MacroTestExtKt.measureStartup(MacroTestExt.kt:19)
at com.testapp.macrobenchmark.test.home.HomeStartupBenchmark.macroBenchmarkHomeLaunchTime(HomeStartupBenchmark.kt:40)
at java.lang.reflect.Method.invoke(Native Method)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule$applyInternal$1.evaluate(MacrobenchmarkRule.kt:93)
at androidx.test.rule.GrantPermissionRule$RequestPermissionStatement.evaluate(GrantPermissionRule.java:134)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:395)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2205)
FAILURES!!!
After some debugging I checked that `android_startup` field is always empty from generated perfetto-trace file, I think this is the root cause of empty traces during test run. Is there any workaround to prevent this error?