Perfetto Traces are missing for several Benchmarks [296453339]

Fixed

Bug

Status Update

No update yet.

Description

ra...@google.com

created issue #1

Aug 17, 2023 08:38PM

Missing Perfetto traces in CI

Examples:

ATI

https://android-build.corp.google.com/test_investigate/?invocationId=I24900010188074885&testResultId=TR26028865991392712&tab=artifacts

Look at device_logcat_test_8BSAY00GLA_12904347790383197078.txt and you will see a lot of entries that look like.

08-16 10:30:03 W/NativeDevice: Failed to pull None]_iter009=/storage/emulated/0/Android/media/androidx.benchmark.integration.macrobenchmark/TrivialListScrollBenchmark_start[compilation-None]_iter009_2023-08-16-10-29-59.perfetto-trace from 8BSAY00GLA to /tmp/TrivialListScrollBenchmark_start[compilation-None]_iter009_2023-08-16-10-29-59_6402794257239941600.perfetto-trace. Message Remote object doesn't exist!
08-16 10:30:03 W/NativeDevice: SyncException (Remote object doesn't exist!) when attempting pull None]_iter009=/storage/emulated/0/Android/media/androidx.benchmark.integration.macrobenchmark/TrivialListScrollBenchmark_start[compilation-None]_iter009_2023-08-16-10-29-59.perfetto-trace to /tmp/TrivialListScrollBenchmark_start[compilation-None]_iter009_2023-08-16-10-29-59_6402794257239941600.perfetto-trace on device 8BSAY00GLA
08-16 10:30:03 W/NativeDevice: com.android.ddmlib.SyncException: Remote object doesn't exist!
	at com.android.ddmlib.SyncService.pullFile(SyncService.java:345)
	at com.android.tradefed.device.NativeDevice$6.run(NativeDevice.java:1496)
	at com.android.tradefed.device.NativeDevice.performDeviceAction(NativeDevice.java:2593)
	at com.android.tradefed.device.NativeDevice.pullFileInternal(NativeDevice.java:1517)
	at com.android.tradefed.device.NativeDevice.pullFile(NativeDevice.java:1411)
	at com.android.tradefed.device.NativeDevice.pullFile(NativeDevice.java:1433)
	at com.android.tradefed.device.metric.FilePullerDeviceMetricCollector.retrieveFile(FilePullerDeviceMetricCollector.java:241)
	at com.android.tradefed.device.metric.FilePullerDeviceMetricCollector.pullMetricFile(FilePullerDeviceMetricCollector.java:203)
	at com.android.tradefed.device.metric.FilePullerDeviceMetricCollector.processMetricRequest(FilePullerDeviceMetricCollector.java:149)
	at com.android.tradefed.device.metric.FilePullerDeviceMetricCollector.onTestEnd(FilePullerDeviceMetricCollector.java:85)
	at com.android.tradefed.device.metric.BaseDeviceMetricCollector.onTestEnd(BaseDeviceMetricCollector.java:254)
	at com.android.tradefed.device.metric.BaseDeviceMetricCollector.testEnded(BaseDeviceMetricCollector.java:517)
	at com.android.tradefed.device.metric.BaseDeviceMetricCollector.testEnded(BaseDeviceMetricCollector.java:532)
	at com.android.tradefed.result.ResultForwarder.testEnded(ResultForwarder.java:321)
	at com.android.tradefed.result.LogcatCrashResultForwarder.testEnded(LogcatCrashResultForwarder.java:130)
	at com.android.tradefed.testtype.InstrumentationListener.testEnded(InstrumentationListener.java:134)
	at com.android.tradefed.result.ResultForwarder.testEnded(ResultForwarder.java:313)
	at com.android.tradefed.result.ddmlib.TestRunToTestInvocationForwarder.testEnded(TestRunToTestInvocationForwarder.java:160)
	at com.android.ddmlib.testrunner.InstrumentationResultParser.reportResult(InstrumentationResultParser.java:533)
	at com.android.ddmlib.testrunner.InstrumentationResultParser.parseStatusCode(InstrumentationResultParser.java:440)
	at com.android.ddmlib.testrunner.InstrumentationResultParser.parse(InstrumentationResultParser.java:278)
	at com.android.ddmlib.testrunner.InstrumentationResultParser.processNewLines(InstrumentationResultParser.java:249)
	at com.android.ddmlib.MultiLineReceiver.addOutput(MultiLineReceiver.java:104)
	at com.android.ddmlib.AdbHelper.executeRemoteCommand(AdbHelper.java:746)
	at com.android.ddmlib.AdbHelper.executeRemoteCommand(AdbHelper.java:511)
	at com.android.ddmlib.internal.DeviceImpl.executeShellCommand(DeviceImpl.java:741)
	at com.android.ddmlib.testrunner.RemoteAndroidTestRunner.run(RemoteAndroidTestRunner.java:359)
	at com.android.tradefed.device.NativeDevice$5.run(NativeDevice.java:1205)
	at com.android.tradefed.device.NativeDevice.performDeviceAction(NativeDevice.java:2593)
	at com.android.tradefed.device.NativeDevice.runInstrumentationTests(NativeDevice.java:1209)
	at com.android.tradefed.device.NativeDevice.runInstrumentationTests(NativeDevice.java:1224)
	at com.android.tradefed.testtype.InstrumentationTest.runInstrumentationTests(InstrumentationTest.java:947)
	at com.android.tradefed.testtype.InstrumentationTest.runWithRerun(InstrumentationTest.java:965)
	at com.android.tradefed.testtype.InstrumentationTest.doTestRun(InstrumentationTest.java:927)
	at com.android.tradefed.testtype.InstrumentationTest.run(InstrumentationTest.java:765)
	at com.android.tradefed.testtype.AndroidJUnitTest.run(AndroidJUnitTest.java:436)
	at com.android.tradefed.testtype.suite.GranularRetriableTestWrapper.intraModuleRun(GranularRetriableTestWrapper.java:355)
	at com.android.tradefed.testtype.suite.GranularRetriableTestWrapper.run(GranularRetriableTestWrapper.java:289)
	at com.android.tradefed.testtype.suite.ModuleDefinition.run(ModuleDefinition.java:595)
	at com.android.tradefed.testtype.suite.ITestSuite.runSingleModule(ITestSuite.java:951)
	at com.android.tradefed.testtype.suite.ITestSuite.run(ITestSuite.java:828)
	at com.android.tradefed.invoker.InvocationExecution.runTest(InvocationExecution.java:1359)
	at com.android.tradefed.invoker.InvocationExecution.runTests(InvocationExecution.java:1138)
	at com.android.tradefed.invoker.TestInvocation.prepareAndRun(TestInvocation.java:626)
	at com.android.tradefed.invoker.TestInvocation.performInvocation(TestInvocation.java:278)
	at com.android.tradefed.invoker.TestInvocation.invoke(TestInvocation.java:1357)
	at com.android.tradefed.command.CommandScheduler$InvocationThread.run(CommandScheduler.java:686)

Comments

cc...@google.com <cc...@google.com> #2Aug 22, 2023 10:08PM

This trace name looks perfectly sanitized, not sure what's the issue here: TrivialListScrollBenchmark_start[compilation-None]_iter009_2023-08-16-10-29-59.perfetto-trace

ra...@google.com <ra...@google.com> #3Aug 22, 2023 10:17PM

https://android-build.corp.google.com/test_investigate/?invocationId=I80000010190009616&testResultId=TR80828881026879391 has files with ,s in the name.

Look at the device_logcat_test_8BSAY00GKS_6727296058051998950.

ra...@google.com <ra...@google.com> Aug 29, 2023 04:12PM

Marked as fixed, reassigned to ra...@google.com.

ap...@google.com <ap...@google.com> #4Aug 29, 2023 04:13PM

Project: platform/frameworks/support
Branch: androidx-main

commit 5dd0781ddeb7d345cab1cd97bd041e6d178642c6
Author: Rahul Ravikumar <rahulrav@google.com>
Date: Mon Aug 28 14:10:01 2023

Sanitize trace names.

Test: ./gradlew :benchmark:benchmark-benchmark:cAT && ./gradlew :benchmark:benchmark-macro:cAT
Bug: 296453339

```
/storage/emulated/0/Android/media/androidx.compose.integration.macrobenchmark/TrivialStartupBenchmark_startup_startup_WARM_compilation_Partial_baselineProfile_UseIfAvailable_iterations_0__iter000_2023_08_28_21_11_51.perfetto_trace
```

Change-Id: I36016289b17d657d87a6d5c0bfc25aca83b86f50

M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/OutputsTest.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/PerfettoTraceTest.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/ProfilerTest.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Outputs.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Profiler.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkScopeTest.kt

https://android-review.googlesource.com/2729281

ap...@google.com <ap...@google.com> #5Sep 5, 2023 10:58PM

Project: platform/frameworks/support
Branch: androidx-main

commit a4c4bb416d1a231eb874ed8cd48999ccc77ecbf5
Author: Chris Craik <ccraik@google.com>
Date: Tue Sep 05 14:41:43 2023

Use sanitized filename as inst result key

Bug:296453339
Test: ParameterizedBenchmark # fails with new require() block, without fix

Fixes errors in parsing keys containing '=', which leads to significant filename collision in CI, and missing traces.

Change-Id: Ia72c6cea174ad5602fbabafd9d310f6dba3314a8

M benchmark/benchmark-common/src/main/java/androidx/benchmark/InstrumentationResults.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Outputs.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Profiler.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/ResultWriter.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/perfetto/PerfettoCaptureWrapper.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/BaselineProfiles.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/MacrobenchmarkScope.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/MethodTracing.kt

https://android-review.googlesource.com/2741117

na...@google.com <na...@google.com> #6Sep 20, 2023 07:16PM

The following release(s) address this bug.It is possible this bug has only been partially addressed:

androidx.benchmark:benchmark-common:1.2.0-rc01
androidx.benchmark:benchmark-macro:1.2.0-rc01

ap...@google.com <ap...@google.com> #7Oct 4, 2023 03:14AM

Project: platform/frameworks/support
Branch: androidx-main

commit 0023956cafcfd113afa3f89ed31d155ccda4914a
Author: Chris Craik <ccraik@google.com>
Date: Tue Oct 03 13:43:51 2023

Avoid over-sanitizing filenames breaking BP plugin file filtering

Fixes:303034735
Bug:296453339
Test: OutputTest
Relnote: "Fix Benchmark file output to no longer break BaselineProfile
Plugin file copying. Files were generated and copied off device, but
had been renamed such that the gradle plugin wouldn't see them."

The previous change to fix

b/296453339

sanitized filenames to avoid
certain characters (e.g. the equals sign which caused issues in CI /
instrumentation result reporting), but unintentionally broke a load
bearing assumption between BPGP and the Benchmark library.

This wasn't caught in testing because continuous test are on either
the host side or device side, and don't cover the exchange of files in
between.

To workaround this coverage gap for now, added a test to OutputTest to
cover the BPGP filtering - duplicating the filtering that BPGP does to
look for baseline and startup profiles.

Change-Id: I8dbccc5a11b4c391169cff6dfbd2a0001fd41001

M benchmark/baseline-profile-gradle-plugin/src/main/kotlin/androidx/baselineprofile/gradle/producer/tasks/CollectBaselineProfileTask.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/OutputsTest.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/PerfettoTraceTest.kt
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/ProfilerTest.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/Outputs.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkScopeTest.kt

https://android-review.googlesource.com/2768511

pr...@google.com <pr...@google.com> #8Oct 6, 2023 05:19PM