[BPGP] When running ./gradlew assemble, it runs GMD setup task [283447020]

Fixed

Bug

Status Update

No update yet.

Description

ml...@google.com

created issue #1

May 19, 2023 02:16PM

Component Used: BPGP 1.2.0-alpha14 Our performance samples CI pipeline was running `./gradlew build`, which triggers `pixel6Api31Setup` GMD task for the `:baselineProfile` plugin that applies BPGP. I tried using `build -x test` or even `assemble -x test` but this task is always triggerd and fails (probably due to using `ubuntu-latest`). Can we run this task only when `generateBaselineProfile` task is running?

Comments

ma...@google.com <ma...@google.com> #2May 19, 2023 08:24PM

This was partially addressed by

https://android-review.googlesource.com/c/platform/frameworks/support/+/2076902, available now in 1.1 RC02, see

b/230665435

. As compilation on user builds requires a full target reinstall (which is a behavior change), we offer an opt out, which can be used to accomplish this feature request: 1) configure every macrobench to use `CompilationMode.Full()` 1) manually issuing the compile command `cmd package compile -f -m speed <packagename>` for your target 1) pass the instrumentation arg `androidx.benchmark.compilation.enable` = `false` to skip compilation/reinstall for each macrobenchmark. This should still give you the numbers you've been seeing, while avoiding the cost of a large AOT each test. Leaving this bug open, since in general we should be able to do this more automatically for everything without warmup driven profiles. (Somewhat related bug - there have been excess compilations issued specifically for `Compilation.None`, `StartupMode.COLD` benchmarks, which has been fixed, but not shipped publicly yet:

b/231976084

)

ml...@google.com <ml...@google.com> #3May 19, 2023 08:33PM

Note: ideally there's be a way to turn off all install related behavior without having to pass in an arg flag. If I want to handle installation & optimization without benchmark getting in the way, there should be an API for that. Arguably that's what Compilation.None should do, or there should be a Compilation.Interpreted and a Compilation.NoOp.

ma...@google.com <ma...@google.com> May 23, 2023 06:53PM

Accepted by ma...@google.com.

ma...@google.com <ma...@google.com> #4May 24, 2023 04:46PM

Also, generally clearing profiles doesn't make much sense for Full compilation mode. I understand this makes sense for profile based benchmarks, but not really for full where the output shouldn't vary.

ma...@google.com <ma...@google.com> #5May 24, 2023 05:35PM

> ideally there's be a way to turn off all install related behavior without having to pass in an arg flag. We can't change APIs in an RC, but we'll consider it for 1.2 > Arguably that's what Compilation.None should do, or there should be a Compilation.Interpreted and a Compilation.NoOp. `None` means "none of my app should be compiled", so at least a reset (if compilation has happened before) should occur. `Compilation.NoOp` or equivalent sounds reasonable. > Also, generally clearing profiles doesn't make much sense for Full compilation mode. I assume so too, I would want to validate that with ART and test it, but either way, we opted for full reinstallation for every test in 1.1 since it's predictable and consistent.

ml...@google.com <ml...@google.com> #6May 24, 2023 09:09PM

Project: platform/frameworks/support
Branch: androidx-main

commit 78a3847fafbb5a810b60b27b1361400bf8dcfa58
Author: Marcello Albano <maralb@google.com>
Date: Mon Jul 11 14:07:07 2022

Added a new compilation mode `CompilationMode.Ignore` to skip profile reset and compilation.

Bug: 230453509
Test: ./gradlew :benchmark:integration-tests:macrobenchmark:cC -Pandroid.testInstrumentationRunnerArguments.class=androidx.benchmark.integration.macrobenchmark.CompilationModeTest

This change adds a new compilation mode `CompilationMode.Ignore` that will skip profile reset and compilation. This can be used to customize the benchmark loop.

Change-Id: Ibbcf8e8c627d227e0f290bddf464b67d9b6653bc
Relnote: Added a new compilation mode `CompilationMode.Ignore` to skip profile reset and compilation.

A benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/ExperimentalMacrobenchmarkApi.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/CompilationMode.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkTest.kt
A benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/CompilationModeTest.kt
M benchmark/benchmark-macro/api/public_plus_experimental_current.txt

https://android-review.googlesource.com/2150519

ma...@google.com <ma...@google.com> #7May 24, 2023 09:21PM

I agree but there isn't much we can do. This is also not related to AGP but to how gradle works: assemble is a task that also "prepares" the configurations (that in gradle are used to carry artifacts). When triggering the configuration we use to expose the baseline profile artifacts the GMD device setup also starts, since it's part of the task chain to produce the artifact.

The only option we've is to decide when to disable those specific tasks, so that the configuration won't have any artifact and don't generate/update the baseline profile.

The property I've implemented is one way. The other way I've explored is to check what task the user invoked but this can potentially break other flows. For example, consider if the user launched:

./gradlew :baselineprofile:assemble :app:generateBaselineProfile :app:assemble

The result is that the baseline profile wouldn't be generated (or updated), the app would be assembled and there would be no failure. Other examples could be cases in which the user defined its own task and adds a dependency on the generation task.

Disabling some tasks based on what the user launched might ultimately be worse in other settings. I think a good trade-off is documentation and the above mentioned property but I'm open to ideas.

Message last modified on May 24, 2023 10:19PM

ap...@google.com <ap...@google.com> #8May 27, 2023 12:02AM

Project: platform/frameworks/support
Branch: androidx-main

commit 79938912e132e8fb24ece4e24d067637e8ea5fe0
Author: Marcello Albano <maralb@google.com>
Date: Wed May 24 11:42:52 2023

Added option to skip baseline profile generation

This PR introduces a new property
`androidx.baselineprofile.skipgeneration` used by the baseline profile
producer plugin. When the property is on it will disable test tasks and
provide empty artifact for baseline profiles. This is helpful for
Github CI projects that don't support KVM.

Enabling this property has the same effect of:
./gradlew assemble \
-x :baselineProfile:pixel6Api31NonMinifiedReleaseAndroidTest \
-x :baselineProfile:collectNonMinifiedReleaseBaselineProfile

Test: ./gradlew :benchmark:benchmark-baseline-profile-gradle-plugin:test
Bug:

b/283447020

Change-Id: I37fda3f4b5761d8229c83be7d7a2cda197e8dbc8
Relnote: Added property to skip baseline profile generation. Usage: `./gradlew assemble -Pandroidx.baselineprofile.skipgeneration`.

M benchmark/baseline-profile-gradle-plugin/src/main/kotlin/androidx/baselineprofile/gradle/consumer/task/MergeBaselineProfileTask.kt
M benchmark/baseline-profile-gradle-plugin/src/main/kotlin/androidx/baselineprofile/gradle/producer/BaselineProfileProducerPlugin.kt
A benchmark/baseline-profile-gradle-plugin/src/main/kotlin/androidx/baselineprofile/gradle/producer/BaselineProfileProducerProperties.kt
M benchmark/baseline-profile-gradle-plugin/src/main/kotlin/androidx/baselineprofile/gradle/producer/tasks/CollectBaselineProfileTask.kt
M benchmark/baseline-profile-gradle-plugin/src/main/kotlin/androidx/baselineprofile/gradle/producer/tasks/InstrumentationTestTaskWrapper.kt
M benchmark/baseline-profile-gradle-plugin/src/test/kotlin/androidx/baselineprofile/gradle/consumer/BaselineProfileConsumerPluginTest.kt
M benchmark/baseline-profile-gradle-plugin/src/test/kotlin/androidx/baselineprofile/gradle/producer/BaselineProfileProducerPluginTest.kt

https://android-review.googlesource.com/2602790

ma...@google.com <ma...@google.com> May 30, 2023 05:45PM

Marked as fixed.

pr...@google.com <pr...@google.com> #9Jun 8, 2023 01:31AM

The following release(s) address this bug.It is possible this bug has only been partially addressed: * `androidx.benchmark:benchmark-baseline-profile-gradle-plugin:1.2.0-alpha15`