Save (or copy) traces recorded by Macrobenchmark library [286199266]

Verified

Feature Request

Status Update

No update yet.

Description

ml...@google.com

created issue #1

Jun 7, 2023 11:08AM

When running macrobenchmarks from Studio, traces and results can be accessed from build/outputs/connected_android_test_additional_output/**. The problem is that with every test run, the results are cleared, therefore you can't access traces from the previous run.

However, results in test history are kept ( https://screenshot.googleplex.com/6Mn83y7c6ZnABTB ) and the links to traces from the history work and open the traces in studio.

Is it possible to either:

a) access the linked trace files from somewhere? (I tried searching them in the studio folders, but couldn't find them) https://screenshot.googleplex.com/coTUdDf5HbA9wQE

b) download / copy any trace from studio profiler? Similarly as you can save the trace file when you run a profiler directly from studio https://screenshot.googleplex.com/5vCZo8aQ5JPjVZ7

Android Studio Hedgehog | 2023.1.1 Canary 6
Build #AI-231.8109.175.2311.10205233, built on May 25, 2023
Runtime version: 17.0.6+0-17.0.6b829.9-10027231 aarch64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
macOS 13.4
GC: G1 Young Generation, G1 Old Generation
Memory: 6144M
Cores: 10
Metal Rendering is ON
Registry:
    external.system.auto.import.disabled=true
    debugger.new.tool.window.layout=true
    ide.text.editor.with.preview.show.floating.toolbar=false
    ide.experimental.ui=true

Non-Bundled Plugins:
    idea.plugin.protoeditor (231.8109.91)
    com.developerphil.adbidea (1.6.10)
    com.android.studio.ml (0.0.9)
    com.rahulrav.baseline-profiles (0.1.9)

Comments

cc...@google.com <cc...@google.com> #2Jun 7, 2023 08:50PM

What version of kotlin are you using?

ml...@google.com <ml...@google.com> #3Jun 7, 2023 08:53PM

1.7.10

ys...@google.com <ys...@google.com> #4Jun 8, 2023 07:44AM

related b/266687218

sh...@google.com <sh...@google.com> #5Jun 9, 2023 11:19PM

Fixed by https://cs.android.com/android-studio/platform/tools/base/+/2553f154c9148529f7bdad459553ca33fffd38d2 for the next canary.

ys...@google.com <ys...@google.com> #6Jun 10, 2023 09:28AM

Thank you for your patience while our engineering team worked to resolve this issue. A fix for this issue is now available in:

Android Studio Iguana | 2023.2.1 Canary 6
Android Gradle Plugin 8.3.0-alpha06

We encourage you to try the latest update.

If you notice further issues or have questions, please file a new bug report.

Thank you for taking the time to submit feedback — we really appreciate it!

be...@google.com <be...@google.com> #7Jun 12, 2023 02:01PM

I agree that visualizing this would be nice to have. We haven't actively maintained or looked into this for the performance samples in a while.

ml...@google.com <ml...@google.com> #8Jun 12, 2023 02:10PM

Profiler provides a way to export any trace file, i.e., option b) in comment #1. This isn't ideal IMHO ...

The files aren't overwritten by the next macrobenchmark run. You can open traces in Studio Profiler from the test history results even without opening it first from the profiler. The benchmark results probably are copied to /tmp even before opening with the profiler.

But anyway, I know it's not ideal to open a profiler just to save the file. From my perspective it was just to have some way that might be implemented faster and come up with the more robust solutions later.

Alternatively to saving from the profiler, saving/copying the file from the context menu (right-click on a trace) https://screenshot.googleplex.com/6LVPb56Vkk4u2Et

It would be good to be able to submit them to a server, say posting to a locally hosted ui.perfetto.dev and generating permalinks. Since that's been my flow...

Yeah, exactly. Currently you need to copy the results yourself before running another benchmark otherwise you loose them.

sh...@google.com <sh...@google.com> #9Jun 12, 2023 09:07PM

The files aren't overwritten by the next macrobenchmark run. You can open traces in Studio Profiler from the test history results even without opening it first from the profiler. The benchmark results probably are copied to /tmp even before opening with the profiler.

Sorry I wasn't clear. I was talking about the overwritten happens at build/outputs/connected_android_test_additional_output/**. It's likely the Test Results feature in Studio has copied the files to somewhere, so the profiler can open them later. The FileUtil.createTempFile() line in comment #5 is running only when the users attempt to open it, which copies the file to /tmp.

cc...@google.com <cc...@google.com> #10Jun 12, 2023 11:01PM

Please note that Profiler makes a copy of the trace file only if the developers has asked to open it

The FileUtil.createTempFile() line in comment #5 is running only when the users attempt to open it, which copies the file to /tmp.

Is that separate from the file pull destination? I see file names like /tmp/CpuCounterBenchmark_fullIteration_2023-03-02-22-25-26.perfetto-trace, which is missing the "cpu_trace_" prefix and ".trace" suffix. I think the "cpu_trace_" version of the file gets created on opening in the profiler in /tmp, but I see the original, not renamed files show up simply when the test method completes, without clicking the benchmark trace links.

sh...@google.com <sh...@google.com> #11Jun 13, 2023 12:02AM

I think you are right Chris. Looks like both Test Results and Profilers are copy files to /tmp on Linux. Profiler's copy has a name like cpu_trace_*.trace.

ml...@google.com <ml...@google.com> #12Aug 23, 2023 01:14PM

Just a revive ping here.

Is there anything we can do here (relatively cheaply?). We've heard from go-no-go that Baseline Profiles (and benchmarks) are disconnected from the Android Studio flow and I think this might be one of the smaller things that adds up to the overal picture.

Message last modified on Aug 23, 2023 01:15PM

sh...@google.com <sh...@google.com> Sep 14, 2023 06:57PM

Reassigned to yd...@google.com.

yd...@google.com <yd...@google.com> Sep 14, 2023 07:04PM

Status: Assigned (reopened)

cc...@google.com <cc...@google.com> #13Sep 14, 2023 10:55PM

Comment has been deleted.

Message last modified on Sep 14, 2023 10:55PM

yd...@google.com <yd...@google.com> #14Sep 19, 2023 09:02PM

Marked as fixed.

The ability to export the traces taken from Macrobenchmark (from the sessions panel) will be available now in Android Studio Iguana Canary 5.

yd...@google.com <yd...@google.com> #15Sep 19, 2023 09:09PM

The UX/solution pursued in the fix is that of route (b) as outlined in the bug description. The save icon (as done for artifacts in the regular Profiler) will now appear under the Session corresponding to the selected trace taken by the Macrobenchmark. A user can click on this icon (that appears on hover) and save the file to disk.

@ml...@google.com please verify that this is the desired behvaior.

Message last modified on Sep 19, 2023 09:11PM

an...@google.com <an...@google.com> #16Sep 21, 2023 09:04PM

Thank you for your patience while our engineering team worked to resolve this issue. A fix for this issue is now available in:

Android Studio Iguana | 2023.2.1 Canary 5
Android Gradle Plugin 8.3.0-alpha05

We encourage you to try the latest update.

If you notice further issues or have questions, please file a new bug report.

Thank you for taking the time to submit feedback — we really appreciate it!

ml...@google.com <ml...@google.com> #17Sep 25, 2023 08:43AM

Thanks for fixing this. I can see the icon.

I have a question -- can we preserve the original filename? VerticalScrollBenchmark_benchmarkLazyGridCompFull_iter000_2023_09_25_08_11_09.perfetto-trace is the opened trace file while when downloading it's called cpu-perfetto-20230925T104037.trace

I assume the different suffix doesn't matter .perfetto-trace vs .trace

(Created a bug for how to prevent overwriting the folder here b/301902205)

Message last modified on Sep 25, 2023 09:03AM

ml...@google.com <ml...@google.com> #18Sep 25, 2023 09:02AM

Comment has been deleted.

Message last modified on Sep 25, 2023 09:03AM

ml...@google.com <ml...@google.com> #19Sep 27, 2023 06:30AM

Verified by ml...@google.com.

Created a follow up b/302236511 to fix the naming

an...@google.com <an...@google.com> #20Oct 5, 2023 09:26PM

Further fixes for this issue are now available in:

Android Studio Iguana | 2023.2.1 Canary 7
Android Gradle Plugin 8.3.0-alpha07

We encourage you to try the latest update.

If you notice further issues or have questions, please file a new bug report.

[Deleted User] <[Deleted User]> #21May 23, 2024 11:21AM

The problem we have is that you can copy off the trace files for a narrow window at the end of the tests. After this the SDCard folder is deleted, presumably as the MacroBenchmark is uninstalling the app at the end.
So we see:
2024-05-23 12:09:48.867 19537-19565 Benchmark com.navenio.ils.benchmark D Moving /storage/emulated/0/Android/media/com.navenio.ils.benchmark/com_navenio_ils_benchmark-benchmarkData.json to /sdcard/Android/media/com.navenio.ils.benchmark/additional_test_output/com_navenio_ils_benchmark-benchmarkData.json
However the folder (/sdcard/Android/media/com.navenio.ils.benchmark/additional_test_output) is deleted at the end of the MacroBenchmark and I have yet to see how to stop it deleting the App.

Managing to copy off the runs by watching in Device Explorer has a folder with all the Perfetto traces.

In case it's Android version specific, this is running the benchmark on a real Pixel 6 Pro running Android 15 Beta 2.
I forgot to say AGP is :
classpath 'com.android.tools.build:gradle:8.4.1'
AS:
Android Studio Jellyfish | 2023.3.1
Build #AI-233.14808.21.2331.11709847, built on April 12, 2024
Runtime version: 17.0.10+0-17.0.10b1087.21-11572160 aarch64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
macOS 14.5
GC: G1 Young Generation, G1 Old Generation
Memory: 4096M
Cores: 10
Metal Rendering is ON
Registry:
ide.instant.shutdown=false
Non-Bundled Plugins:
wu.seal.tool.jsontokotlin (3.7.4)
idea.plugin.protoeditor (233.14808.14)
com.jetbrains.kmm (0.8.2(233)-8)

Message last modified on May 23, 2024 11:23AM

Screenshot 2024-05-23 at 12.18.17.png

108 KB

View

Download

Issue 286199266

Description

Issue summary

Comments

cc...@google.com <cc...@google.com> #2Jun 7, 2023 08:50PM

ml...@google.com <ml...@google.com> #3Jun 7, 2023 08:53PM

ys...@google.com <ys...@google.com> #4Jun 8, 2023 07:44AM

sh...@google.com <sh...@google.com> #5Jun 9, 2023 11:19PM

ys...@google.com <ys...@google.com> #6Jun 10, 2023 09:28AM

be...@google.com <be...@google.com> #7Jun 12, 2023 02:01PM

ml...@google.com <ml...@google.com> #8Jun 12, 2023 02:10PM

sh...@google.com <sh...@google.com> #9Jun 12, 2023 09:07PM

cc...@google.com <cc...@google.com> #10Jun 12, 2023 11:01PM

sh...@google.com <sh...@google.com> #11Jun 13, 2023 12:02AM

ml...@google.com <ml...@google.com> #12Aug 23, 2023 01:14PM

sh...@google.com <sh...@google.com> Sep 14, 2023 06:57PM

yd...@google.com <yd...@google.com> Sep 14, 2023 07:04PM

cc...@google.com <cc...@google.com> #13Sep 14, 2023 10:55PM

yd...@google.com <yd...@google.com> #14Sep 19, 2023 09:02PM

yd...@google.com <yd...@google.com> #15Sep 19, 2023 09:09PM

an...@google.com <an...@google.com> #16Sep 21, 2023 09:04PM

ml...@google.com <ml...@google.com> #17Sep 25, 2023 08:43AM

ml...@google.com <ml...@google.com> #18Sep 25, 2023 09:02AM

ml...@google.com <ml...@google.com> #19Sep 27, 2023 06:30AM

an...@google.com <an...@google.com> #20Oct 5, 2023 09:26PM

[Deleted User] <[Deleted User]> #21May 23, 2024 11:21AM

Add comment

Issue metadata