Unable to read any metrics during benchmark in macrobenchmark version 1.1.0-alpha07 [200302931]

Fixed

Bug

Status Update

No update yet.

Description

de...@tokopedia.com

created issue #1

Sep 18, 2021 07:15AM

I was trying to upgrade my macrobenchmark version from 1.1.0-alpha02 to 1.1.0-alpha07. And now I'm always encountered this error:

There was 1 failure:
1) macroBenchmarkHomeLaunchTime[mode=COLD](com.testapp.macrobenchmark.test.home.HomeStartupBenchmark)
java.lang.IllegalArgumentException: Unable to read any metrics during benchmark (metric list: [androidx.benchmark.macro.StartupTimingMetric@1fce2f9]).
Check that you're performing the operations to be measured. For example, if
using StartupTimingMetric, are you starting an activity for the specified package
in the measure block?
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:198)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:259)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:68)
at com.testapp.macrobenchmark.util.MacroTestExtKt.measureStartup(MacroTestExt.kt:19)
at com.testapp.macrobenchmark.test.home.HomeStartupBenchmark.macroBenchmarkHomeLaunchTime(HomeStartupBenchmark.kt:40)
at java.lang.reflect.Method.invoke(Native Method)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule$applyInternal$1.evaluate(MacrobenchmarkRule.kt:93)
at androidx.test.rule.GrantPermissionRule$RequestPermissionStatement.evaluate(GrantPermissionRule.java:134)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:395)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2205)

FAILURES!!!

After some debugging I checked that `android_startup` field is always empty from generated perfetto-trace file, I think this is the root cause of empty traces during test run. Is there any workaround to prevent this error?

Comments

de...@tokopedia.com <de...@tokopedia.com> #2Sep 18, 2021 12:23PM

Thanks for your detailed post.

However, benchmark build type is under configured: at least isProfileable is not set to true for existing build type, probably there's more.

We should set isProfileable = true by default when overriding an existing benchmark build type.

This issue is not about some specific configuration flag, but the general approach of dealing with external configuration. As a developer adopting baseline profiles, it seems extremely risky to me using a custom configuration due to how it's applied under the hood and the fact it may break default configuration.

I agree that is not great but this is a little tricky to do. For custom baseline profile build types we override all the properties. For benchmark I left it open to configure but it's mostly about these 2 properties:

isMinifyEnabled
isShrinkResources

I don't have a way to see if the user is setting them before overriding, so for this reason, I'd prefer not to. I agree with you that some other properties could be set by default to make this easier, i.e.:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

The reason why I mentioned the release signing config in the beginning is because I want to use debug signing config.

In the specific of your issue, i.e. using a debug certificate can you override the benchmark and baseline profile setting? You should be able to do something like:

android {
    buildTypes {
        release { ... }
        debug { ... }
        benchmarkRelease {
            ...
            signingConfig signingConfigs.debug
        }
        nonMinifiedRelease {
            ...
            signingConfig signingConfigs.debug
        }
    }
}

cc...@google.com <cc...@google.com> #3Oct 6, 2021 11:36PM

Project: platform/frameworks/support
Branch: androidx-main
Author: Marcello Albano <maralb@google.com>
Link: https://android-review.googlesource.com/3303459

Added override for debuggable and profileable for benchmark builds in bpgp

Expand for full commit details

Added override for debuggable and profileable for benchmark builds in bpgp

Test: ./gradlew :benchmark:benchmark-baseline-profile-gradle-plugin:test
Bug: 369213505
Relnote: "isProfileable is always overridden in benchmark builds,
  and isDebuggable is also now always overridden in both benchmark and
  nonMinified (baseline profile capture) builds."
Change-Id: I487fa71083921682173f04fcbb477be5baf165f8

Files:

M benchmark/baseline-profile-gradle-plugin/src/main/kotlin/androidx/baselineprofile/gradle/apptarget/BaselineProfileAppTargetPlugin.kt
M benchmark/baseline-profile-gradle-plugin/src/test/kotlin/androidx/baselineprofile/gradle/apptarget/BaselineProfileAppTargetPluginTest.kt

Hash: 1906bbe52ba7ccb9ca0e1c1d6de33e7c91b5c6f0
Date: Fri Oct 11 10:07:06 2024

ap...@google.com <ap...@google.com> #4Oct 13, 2021 12:08AM

I've landed a change that will set the following properties also when the benchmark build type already exists:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

As well as the following for agp 8.0:

isDebuggable = false

I'm going ahead and closing this - if you've further questions please answer here and will reopen. Thanks.

cc...@google.com <cc...@google.com> #5Oct 13, 2021 12:59AM

Marked as fixed.

The following release(s) address this bug.It is possible this bug has only been partially addressed:

androidx.benchmark:benchmark-baseline-profile-gradle-plugin:1.4.0-alpha04

c....@gmail.com <c....@gmail.com> #6Jun 3, 2022 06:33AM

Since release signing config is used by default instead of debug is not mentioned in release notes - maybe this is a bug?

Cause the last time I found it mentioned in the release notes was in version 1.1

signingConfig.debug is used as the default signing config (b/153583269)

So, if the switch to the release one indeed happened - maybe it's an issue?

ni...@gmail.com <ni...@gmail.com> #7Jun 7, 2022 11:04AM

The new build types should inherit the certificates from the release build type, unless none is specified in which case debug is used: code link

This should be updated in the documentation, thanks for mentioning it. Will take care of updating it.

Message last modified on Jun 7, 2022 11:05AM

ni...@gmail.com <ni...@gmail.com> #8Jun 8, 2022 08:01AM

it is working onwards android 10

va...@gmail.com <va...@gmail.com> #9Oct 13, 2022 06:06PM

Getting this error on androidx.benchmark:benchmark-macro-junit4:1.1.0,
things already tried:
1. removing pressHome()
2. applied Thread.sleep(5000) before startActivityAndWait()

Error details:

> Task :macro-benchmark:connectedReleaseAndroidTest
Starting 2 tests on SM-F415F - 11
Timed out waiting for process (com.example.benchmark) to appear on samsung-sm_f41
..
..
java.lang.IllegalArgumentException: Unable to read any metrics during benchmark (metric list: [androidx.benchmark.macro.StartupTimingMetric@eb8ded8]).
Check that you're performing the operations to be measured. For example, if
using StartupTimingMetric, are you starting an activity for the specified package
in the measure block?
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:237)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:301)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:106)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated$default(MacrobenchmarkRule.kt:96)
at com.example.benchmark.StartupBenchmark.startup(StartupBenchmark.kt:56)
at com.example.benchmark.StartupBenchmark.startupCompilationPartial(StartupBenchmark.kt:51)

ma...@gmail.com <ma...@gmail.com> #10Mar 1, 2023 07:36AM

Yesterday it was working without any issues.
Today I got the above-mentioned error.

What I did to make it work:
1. Removed the app from the real device
2. Clean the project
3. Make sure that the `build` folder inside the benchmark model is totally removed. If not, remove it manually.

Then start the benchmark.

ma...@propertyfinder.ae <ma...@propertyfinder.ae> #11Mar 3, 2023 05:41AM

It happed again, I restarted the android device and it worked.

ro...@gmail.com <ro...@gmail.com> #12Jan 31, 2024 07:25PM

Comment has been deleted.

Message last modified on Jan 31, 2024 07:25PM