Move to Instrumentation.addResults() [154248456]

Assigned

Bug

Status Update

No update yet.

Description

cc...@google.com

created issue #1

Apr 16, 2020 07:33PM

Currently we use Instrumentation.sendStatus() to report our internal metrics and studio output.

Using addResults should fix the following CLI warning that's started showing up from AGP in recent versions, once per test:

[InstrumentationResultParser]: invalid instrumentation status bundle unknown result
[InstrumentationResultParser]: invalid instrumentation status bundle unknown result
[InstrumentationResultParser]: invalid instrumentation status bundle unknown result
[InstrumentationResultParser]: invalid instrumentation status bundle unknown result
[InstrumentationResultParser]: invalid instrumentation status bundle unknown result

Comments

ap...@google.com <ap...@google.com> #2Apr 20, 2020 07:45PM

Marked as fixed.

This was partially addressed by

https://android-review.googlesource.com/c/platform/frameworks/support/+/2076902, available now in 1.1 RC02, see

b/230665435

. As compilation on user builds requires a full target reinstall (which is a behavior change), we offer an opt out, which can be used to accomplish this feature request: 1) configure every macrobench to use `CompilationMode.Full()` 1) manually issuing the compile command `cmd package compile -f -m speed <packagename>` for your target 1) pass the instrumentation arg `androidx.benchmark.compilation.enable` = `false` to skip compilation/reinstall for each macrobenchmark. This should still give you the numbers you've been seeing, while avoiding the cost of a large AOT each test. Leaving this bug open, since in general we should be able to do this more automatically for everything without warmup driven profiles. (Somewhat related bug - there have been excess compilations issued specifically for `Compilation.None`, `StartupMode.COLD` benchmarks, which has been fixed, but not shipped publicly yet:

b/231976084

)

cc...@google.com <cc...@google.com> #3Apr 27, 2020 07:29PM

Status: Assigned (reopened)

Note: ideally there's be a way to turn off all install related behavior without having to pass in an arg flag. If I want to handle installation & optimization without benchmark getting in the way, there should be an API for that. Arguably that's what Compilation.None should do, or there should be a Compilation.Interpreted and a Compilation.NoOp.

cc...@google.com <cc...@google.com> #4Apr 27, 2020 07:30PM

Also, generally clearing profiles doesn't make much sense for Full compilation mode. I understand this makes sense for profile based benchmarks, but not really for full where the output shouldn't vary.

ap...@google.com <ap...@google.com> #5May 26, 2020 10:45PM

> ideally there's be a way to turn off all install related behavior without having to pass in an arg flag. We can't change APIs in an RC, but we'll consider it for 1.2 > Arguably that's what Compilation.None should do, or there should be a Compilation.Interpreted and a Compilation.NoOp. `None` means "none of my app should be compiled", so at least a reset (if compilation has happened before) should occur. `Compilation.NoOp` or equivalent sounds reasonable. > Also, generally clearing profiles doesn't make much sense for Full compilation mode. I assume so too, I would want to validate that with ART and test it, but either way, we opted for full reinstallation for every test in 1.1 since it's predictable and consistent.

cc...@google.com <cc...@google.com> #6May 26, 2020 10:45PM

Project: platform/frameworks/support
Branch: androidx-main

commit 78a3847fafbb5a810b60b27b1361400bf8dcfa58
Author: Marcello Albano <maralb@google.com>
Date: Mon Jul 11 14:07:07 2022

Added a new compilation mode `CompilationMode.Ignore` to skip profile reset and compilation.

Bug: 230453509
Test: ./gradlew :benchmark:integration-tests:macrobenchmark:cC -Pandroid.testInstrumentationRunnerArguments.class=androidx.benchmark.integration.macrobenchmark.CompilationModeTest

This change adds a new compilation mode `CompilationMode.Ignore` that will skip profile reset and compilation. This can be used to customize the benchmark loop.

Change-Id: Ibbcf8e8c627d227e0f290bddf464b67d9b6653bc
Relnote: Added a new compilation mode `CompilationMode.Ignore` to skip profile reset and compilation.

A benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/ExperimentalMacrobenchmarkApi.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/CompilationMode.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkTest.kt
A benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/CompilationModeTest.kt
M benchmark/benchmark-macro/api/public_plus_experimental_current.txt

https://android-review.googlesource.com/2150519

ap...@google.com <ap...@google.com> #7Jun 3, 2020 03:31PM

Project: platform/frameworks/support
Branch: androidx-master-dev

commit 6b8eee4a0ff1cd0bd197aa36329da7e19e605e27
Author: Chris Craik <ccraik@google.com>
Date: Tue Jun 02 12:54:25 2020

Fix additional invalid instrumentation status error

Bug: 154248456
Test: ./gradlew benchmark:b-b:cC (no longer prints error)
Test: TrivialKotlinBenchmark (from Studio, console output still works)

Change-Id: I80fd74f30af6afb84435f818ea6d598696c3f342

M benchmark/common/src/main/java/androidx/benchmark/BenchmarkState.kt

https://android-review.googlesource.com/1321947

Issue 154248456