Fixed
Status Update
No update yet.
Comments
de...@tokopedia.com
With this change
The things we need to tackle after this CL lands are:
- Follow up on this CL (
https://android-review.googlesource.com/c/platform/external/perfetto/+/1680145 ). The binaries checked in, include this change but currently its failing presubmits because of a compiler warning. Something primiano@ is currently looking into. - Expand trace_processor_shells ABI support to include arm, and x86_64. Remove the explicit checks for API Q.
- Remove the use of @RequiresApi(29) in MacrobenchmarkRule / Lower it to API 21, andexplicit checks for API Q.
- verifyTraceEnabled(...) does seem to cause issues on lower versions of the platform. Something we need to investigate further. This does not happen consistently though.
- Fix our perfetto trace configs. There are options that are unsupported on older versions of the platform.
- M, and N dont support several ART compilation modes. No op on them.
cc...@google.com
Project: platform/frameworks/support
Branch: androidx-main
commit 22fe26bc907af50d557254d406eb38469a7bb81b
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Fri Apr 16 11:15:58 2021
Add unbundled perfetto support.
With this change, we are much closer in bringing Macrobenchmarks to API 21.
The things we need to tackle after this CL lands are:
* Follow up on this CL (https://android-review.googlesource.com/c/platform/external/perfetto/+/1680145 ).
The binaries checked in, include this change but currently its failing presubmits because of a compiler warning.
Something primiano@ is currently looking into.
* Expand trace_processor_shells ABI support to include arm, and x86_64.
Remove the explicit checks for API Q.
* Remove the use of @RequiresApi(...) in MacrobenchmarkRule, and explicit checks for API Q.
* verifyTraceEnabled(...) does seem to cause issues on lower versions of the platform. Something we need to investigate further.
This does not happen consistently though.
* Fix our perfetto trace configs. There are options that are unsupported on older versions of the platform.
* M, and N dont support several ART compilation modes. No op on them.
Test: PerfettoCaptureTest is now parameterized
Bug:
Change-Id: I888fa15e02acc4f99fbf548722a11a9953988357
M benchmark/macro/build.gradle
M benchmark/macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoCaptureTest.kt
M benchmark/macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoCapture.kt
M benchmark/macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoCaptureWrapper.kt
M benchmark/macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoHelper.kt
https://android-review.googlesource.com/1675367
Branch: androidx-main
commit 22fe26bc907af50d557254d406eb38469a7bb81b
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Fri Apr 16 11:15:58 2021
Add unbundled perfetto support.
With this change, we are much closer in bringing Macrobenchmarks to API 21.
The things we need to tackle after this CL lands are:
* Follow up on this CL (
The binaries checked in, include this change but currently its failing presubmits because of a compiler warning.
Something primiano@ is currently looking into.
* Expand trace_processor_shells ABI support to include arm, and x86_64.
Remove the explicit checks for API Q.
* Remove the use of @RequiresApi(...) in MacrobenchmarkRule, and explicit checks for API Q.
* verifyTraceEnabled(...) does seem to cause issues on lower versions of the platform. Something we need to investigate further.
This does not happen consistently though.
* Fix our perfetto trace configs. There are options that are unsupported on older versions of the platform.
* M, and N dont support several ART compilation modes. No op on them.
Test: PerfettoCaptureTest is now parameterized
Bug:
Change-Id: I888fa15e02acc4f99fbf548722a11a9953988357
M benchmark/macro/build.gradle
M benchmark/macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoCaptureTest.kt
M benchmark/macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoCapture.kt
M benchmark/macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoCaptureWrapper.kt
M benchmark/macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoHelper.kt
ap...@google.com
Project: platform/prebuilts/androidx/traceprocessor
Branch: androidx-main
commit 9f143a6b876ef5b2be841ed2ce1be138e325bf51
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Wed Mar 31 14:49:19 2021
Add stripped perfetto and trace_processor_shell binaries.
* These binaries are built from source (https://github.com/google/perfetto )
* Build instructions arehttps://perfetto.dev/docs/contributing/build-instructions
* The config looks something like:
```
target_os = "android"
target_cpu = "arm|arm64|arm_x86_x86"
is_debug = false
monolithic_binaries = true
```
* After building, we stripped the binaries using ndk `22.1.7171670` using `/path/to/sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/<variant>-linux-android/bin/strip`
Test: ./gradlew :benchmark:integration-tests:macrobenchmark:connectedDebugAndroidTest
Bug:
Change-Id: I052b6caf214a3d0dfd101302040f6ae43bc42b22
M README.md
A perfetto/perfetto_aarch64
A perfetto/perfetto_arm
A perfetto/perfetto_x86_64
M trace_processor_shell/trace_processor_shell_aarch64
A trace_processor_shell/trace_processor_shell_arm
A trace_processor_shell/trace_processor_shell_x86_64
A traced/traced_aarch64
A traced/traced_arm
A traced/traced_x86_64
A traced_probes/traced_probes_aarch64
A traced_probes/traced_probes_arm
A traced_probes/traced_probes_x86_64
https://android-review.googlesource.com/1660781
Branch: androidx-main
commit 9f143a6b876ef5b2be841ed2ce1be138e325bf51
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Wed Mar 31 14:49:19 2021
Add stripped perfetto and trace_processor_shell binaries.
* These binaries are built from source (
* Build instructions are
* The config looks something like:
```
target_os = "android"
target_cpu = "arm|arm64|arm_x86_x86"
is_debug = false
monolithic_binaries = true
```
* After building, we stripped the binaries using ndk `22.1.7171670` using `/path/to/sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/<variant>-linux-android/bin/strip`
Test: ./gradlew :benchmark:integration-tests:macrobenchmark:connectedDebugAndroidTest
Bug:
Change-Id: I052b6caf214a3d0dfd101302040f6ae43bc42b22
M README.md
A perfetto/perfetto_aarch64
A perfetto/perfetto_arm
A perfetto/perfetto_x86_64
M trace_processor_shell/trace_processor_shell_aarch64
A trace_processor_shell/trace_processor_shell_arm
A trace_processor_shell/trace_processor_shell_x86_64
A traced/traced_aarch64
A traced/traced_arm
A traced/traced_x86_64
A traced_probes/traced_probes_aarch64
A traced_probes/traced_probes_arm
A traced_probes/traced_probes_x86_64
cc...@google.com
Project: platform/frameworks/support
Branch: androidx-main
commit ce3df5c24d36ac56873d13bb27a084e54e420dd7
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Tue Apr 20 12:16:02 2021
Expand ABI support for `perfetto_trace_processor`.
Test: ./gradlew :benchmark:benchmark-macro:connectedCheck
Bug: Bug:
Change-Id: Ic7c5579b497890b2eab8f202b41d2a2c2560c1fb
M benchmark/macro/src/androidTest/java/androidx/benchmark/macro/StartupTimingMetricTest.kt
M benchmark/macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessorTest.kt
M benchmark/macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessor.kt
https://android-review.googlesource.com/1681590
Branch: androidx-main
commit ce3df5c24d36ac56873d13bb27a084e54e420dd7
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Tue Apr 20 12:16:02 2021
Expand ABI support for `perfetto_trace_processor`.
Test: ./gradlew :benchmark:benchmark-macro:connectedCheck
Bug: Bug:
Change-Id: Ic7c5579b497890b2eab8f202b41d2a2c2560c1fb
M benchmark/macro/src/androidTest/java/androidx/benchmark/macro/StartupTimingMetricTest.kt
M benchmark/macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessorTest.kt
M benchmark/macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoTraceProcessor.kt
c....@gmail.com
Punting from initial alpha - unfortunately, the trace signals that perfetto uses for startup metrics MetricsLogger:launchObserverNotifyIntentStarted and launching: $packageName were added in Q, so we'll need to figure out an alternative timing mechanism on older devices. Quite likely that looks like looking for the end of drawFrame on RenderThread, but coming up with a clear definition with sample traces will take some time.
ni...@gmail.com
Do we have a pre-Q trace handy?
ni...@gmail.com
Project: platform/frameworks/support
Branch: androidx-main
commit 7fd221c0c8f7948351293a7fbf75eef0bf698a42
Author: Chris Craik <ccraik@google.com>
Date: Fri Apr 30 17:19:22 2021
Fix output issue on API 21, suppress still-broken tests
Bug: 183129298
Test: ./gradlew benchmark:b-c:cC benchmark:b-m:cC
Shell can't access externalCacheDir (but somehow can access cacheDir)
on API 21.
Also on API 21, UiDevice.executeShellCommand doesn't capture the
output of `echo foo`. executeShellScript still partially works
(e.g. `ls` works), but we'll need more investigation to better
understand shell command limitations on that API level.
Change-Id: I2bafe8218ed50ac54cd990190d2f9a2e2364028d
M benchmark/common/src/main/java/androidx/benchmark/Outputs.kt
A benchmark/macro/src/androidTest/java/androidx/benchmark/macro/OutputsTest.kt
M benchmark/macro/src/androidTest/java/androidx/benchmark/macro/perfetto/ShellUtilsTest.kt
M benchmark/macro/src/main/java/androidx/benchmark/macro/perfetto/ShellUtils.kt
https://android-review.googlesource.com/1692806
Branch: androidx-main
commit 7fd221c0c8f7948351293a7fbf75eef0bf698a42
Author: Chris Craik <ccraik@google.com>
Date: Fri Apr 30 17:19:22 2021
Fix output issue on API 21, suppress still-broken tests
Bug: 183129298
Test: ./gradlew benchmark:b-c:cC benchmark:b-m:cC
Shell can't access externalCacheDir (but somehow can access cacheDir)
on API 21.
Also on API 21, UiDevice.executeShellCommand doesn't capture the
output of `echo foo`. executeShellScript still partially works
(e.g. `ls` works), but we'll need more investigation to better
understand shell command limitations on that API level.
Change-Id: I2bafe8218ed50ac54cd990190d2f9a2e2364028d
M benchmark/common/src/main/java/androidx/benchmark/Outputs.kt
A benchmark/macro/src/androidTest/java/androidx/benchmark/macro/OutputsTest.kt
M benchmark/macro/src/androidTest/java/androidx/benchmark/macro/perfetto/ShellUtilsTest.kt
M benchmark/macro/src/main/java/androidx/benchmark/macro/perfetto/ShellUtils.kt
va...@gmail.com
(Coming back to this after I/O) Carmen - filed
ma...@gmail.com
Project: platform/frameworks/support
Branch: androidx-main
commit 3dff7103380e8f0561029da051bd444d8bbcd26b
Author: Chris Craik <ccraik@google.com>
Date: Wed Jul 28 16:58:05 2021
Only use atrace tags supported on the local platform version
Bug: 183129298
Test: PerfettoConfigTest
Test: AtraceTagTest
Codifies list of in-use atrace tags with which platform versions
they're supported on. Validated on API 21 -> 29 emulators.
Currently, the way we invoke perfetto doesn't tell us if starting the
process failed, so this eliminates one source of this failure.
Planned follow up work
- investigate shell wrapper issues on API 21/22.
- improve perfetto launch failures to throw clearly
- add additional config validation
- (low prior) stash shell session rooted status
Change-Id: I99f5db15d1d0ee84d340b6879ef363f41c4327b5
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkScopeTest.kt
A benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/AtraceTagTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoConfigTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/ShellUtilsTest.kt
A benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/AtraceTag.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoCapture.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoConfig.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/ShellUtils.kt
https://android-review.googlesource.com/1780588
Branch: androidx-main
commit 3dff7103380e8f0561029da051bd444d8bbcd26b
Author: Chris Craik <ccraik@google.com>
Date: Wed Jul 28 16:58:05 2021
Only use atrace tags supported on the local platform version
Bug: 183129298
Test: PerfettoConfigTest
Test: AtraceTagTest
Codifies list of in-use atrace tags with which platform versions
they're supported on. Validated on API 21 -> 29 emulators.
Currently, the way we invoke perfetto doesn't tell us if starting the
process failed, so this eliminates one source of this failure.
Planned follow up work
- investigate shell wrapper issues on API 21/22.
- improve perfetto launch failures to throw clearly
- add additional config validation
- (low prior) stash shell session rooted status
Change-Id: I99f5db15d1d0ee84d340b6879ef363f41c4327b5
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkScopeTest.kt
A benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/AtraceTagTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoConfigTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/ShellUtilsTest.kt
A benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/AtraceTag.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoCapture.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoConfig.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/ShellUtils.kt
ma...@propertyfinder.ae
Project: platform/frameworks/support
Branch: androidx-main
commit 48ae41ec2aada7a167afbd9e03ce67e9114aed23
Author: Chris Craik <ccraik@google.com>
Date: Thu Jul 29 09:43:38 2021
Fix ShellUtils on API 21/22 - chmod +x not supported
Bug: 183129298
Test: ShellUtilsTest # verified on API 21 through 29
Change-Id: I8f235b8044fb54f9f23129a62e8a732536900add
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/ShellUtilsTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/ShellUtils.kt
https://android-review.googlesource.com/1782028
Branch: androidx-main
commit 48ae41ec2aada7a167afbd9e03ce67e9114aed23
Author: Chris Craik <ccraik@google.com>
Date: Thu Jul 29 09:43:38 2021
Fix ShellUtils on API 21/22 - chmod +x not supported
Bug: 183129298
Test: ShellUtilsTest # verified on API 21 through 29
Change-Id: I8f235b8044fb54f9f23129a62e8a732536900add
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/ShellUtilsTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/ShellUtils.kt
ro...@gmail.com
Comment has been deleted.
Description
There was 1 failure:
1) macroBenchmarkHomeLaunchTime[mode=COLD](com.testapp.macrobenchmark.test.home.HomeStartupBenchmark)
java.lang.IllegalArgumentException: Unable to read any metrics during benchmark (metric list: [androidx.benchmark.macro.StartupTimingMetric@1fce2f9]).
Check that you're performing the operations to be measured. For example, if
using StartupTimingMetric, are you starting an activity for the specified package
in the measure block?
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:198)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:259)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:68)
at com.testapp.macrobenchmark.util.MacroTestExtKt.measureStartup(MacroTestExt.kt:19)
at com.testapp.macrobenchmark.test.home.HomeStartupBenchmark.macroBenchmarkHomeLaunchTime(HomeStartupBenchmark.kt:40)
at java.lang.reflect.Method.invoke(Native Method)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule$applyInternal$1.evaluate(MacrobenchmarkRule.kt:93)
at androidx.test.rule.GrantPermissionRule$RequestPermissionStatement.evaluate(GrantPermissionRule.java:134)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:395)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2205)
FAILURES!!!
After some debugging I checked that `android_startup` field is always empty from generated perfetto-trace file, I think this is the root cause of empty traces during test run. Is there any workaround to prevent this error?