TraceSectionMetric optional custom name [301892382]

Fixed

Feature Request

Status Update

No update yet.

Description

ml...@google.com

created issue #1

Sep 25, 2023 07:58AM

TraceSectionMetric reports name as the sectionName + Ms or Count (in case of Mode.Sum).

Sometimes this might be super long and breaks standard horizontal size:

JIT compiling %com.google.composeviewbenchmarks.target%Ms      min      30.7,   median      97.7,   max     204.1

What do you think of adding optional metricName parameter to reduce this and give developers the option to define the name.

Pros:

shorter names easily glanceable
if we're improving metrics as in b/300887146 you can define the names yourself

Potential issues:

conflicting names
renaming something and forgetting what it was in reality

Comments

cc...@google.com <cc...@google.com> #2Sep 26, 2023 04:34PM

This was partially addressed by

https://android-review.googlesource.com/c/platform/frameworks/support/+/2076902, available now in 1.1 RC02, see

b/230665435

. As compilation on user builds requires a full target reinstall (which is a behavior change), we offer an opt out, which can be used to accomplish this feature request: 1) configure every macrobench to use `CompilationMode.Full()` 1) manually issuing the compile command `cmd package compile -f -m speed <packagename>` for your target 1) pass the instrumentation arg `androidx.benchmark.compilation.enable` = `false` to skip compilation/reinstall for each macrobenchmark. This should still give you the numbers you've been seeing, while avoiding the cost of a large AOT each test. Leaving this bug open, since in general we should be able to do this more automatically for everything without warmup driven profiles. (Somewhat related bug - there have been excess compilations issued specifically for `Compilation.None`, `StartupMode.COLD` benchmarks, which has been fixed, but not shipped publicly yet:

b/231976084

)

ml...@google.com <ml...@google.com> #3Sep 27, 2023 08:22AM

Note: ideally there's be a way to turn off all install related behavior without having to pass in an arg flag. If I want to handle installation & optimization without benchmark getting in the way, there should be an API for that. Arguably that's what Compilation.None should do, or there should be a Compilation.Interpreted and a Compilation.NoOp.

Message last modified on Sep 27, 2023 08:31AM

ml...@google.com <ml...@google.com> #4Dec 4, 2023 09:22AM

Also, generally clearing profiles doesn't make much sense for Full compilation mode. I understand this makes sense for profile based benchmarks, but not really for full where the output shouldn't vary.

Message last modified on Dec 4, 2023 09:22AM

ap...@google.com <ap...@google.com> #5Dec 5, 2023 05:50PM

> ideally there's be a way to turn off all install related behavior without having to pass in an arg flag. We can't change APIs in an RC, but we'll consider it for 1.2 > Arguably that's what Compilation.None should do, or there should be a Compilation.Interpreted and a Compilation.NoOp. `None` means "none of my app should be compiled", so at least a reset (if compilation has happened before) should occur. `Compilation.NoOp` or equivalent sounds reasonable. > Also, generally clearing profiles doesn't make much sense for Full compilation mode. I assume so too, I would want to validate that with ART and test it, but either way, we opted for full reinstallation for every test in 1.1 since it's predictable and consistent.

cc...@google.com <cc...@google.com> Jan 5, 2024 12:00AM

Marked as fixed.

pr...@google.com <pr...@google.com> #6Feb 21, 2024 07:32PM

Project: platform/frameworks/support
Branch: androidx-main

commit 78a3847fafbb5a810b60b27b1361400bf8dcfa58
Author: Marcello Albano <maralb@google.com>
Date: Mon Jul 11 14:07:07 2022

Added a new compilation mode `CompilationMode.Ignore` to skip profile reset and compilation.

Bug: 230453509
Test: ./gradlew :benchmark:integration-tests:macrobenchmark:cC -Pandroid.testInstrumentationRunnerArguments.class=androidx.benchmark.integration.macrobenchmark.CompilationModeTest

This change adds a new compilation mode `CompilationMode.Ignore` that will skip profile reset and compilation. This can be used to customize the benchmark loop.

Change-Id: Ibbcf8e8c627d227e0f290bddf464b67d9b6653bc
Relnote: Added a new compilation mode `CompilationMode.Ignore` to skip profile reset and compilation.

A benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/ExperimentalMacrobenchmarkApi.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/CompilationMode.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkTest.kt
A benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/CompilationModeTest.kt
M benchmark/benchmark-macro/api/public_plus_experimental_current.txt

https://android-review.googlesource.com/2150519

Issue 301892382