Thanks for your detailed post.

However, benchmark build type is under configured: at least isProfileable is not set to true for existing build type, probably there's more.

We should set isProfileable = true by default when overriding an existing benchmark build type.

This issue is not about some specific configuration flag, but the general approach of dealing with external configuration. As a developer adopting baseline profiles, it seems extremely risky to me using a custom configuration due to how it's applied under the hood and the fact it may break default configuration.

I agree that is not great but this is a little tricky to do. For custom baseline profile build types we override all the properties. For benchmark I left it open to configure but it's mostly about these 2 properties:

isMinifyEnabled
isShrinkResources

I don't have a way to see if the user is setting them before overriding, so for this reason, I'd prefer not to. I agree with you that some other properties could be set by default to make this easier, i.e.:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

The reason why I mentioned the release signing config in the beginning is because I want to use debug signing config.

In the specific of your issue, i.e. using a debug certificate can you override the benchmark and baseline profile setting? You should be able to do something like:

android {
    buildTypes {
        release { ... }
        debug { ... }
        benchmarkRelease {
            ...
            signingConfig signingConfigs.debug
        }
        nonMinifiedRelease {
            ...
            signingConfig signingConfigs.debug
        }
    }
}

Project: platform/frameworks/support
Branch: androidx-main
Author: Marcello Albano <maralb@google.com>
Link:      https://android-review.googlesource.com/3303459

Added override for debuggable and profileable for benchmark builds in bpgp

Added override for debuggable and profileable for benchmark builds in bpgp

Test: ./gradlew :benchmark:benchmark-baseline-profile-gradle-plugin:test
Bug: 369213505
Relnote: "isProfileable is always overridden in benchmark builds,
  and isDebuggable is also now always overridden in both benchmark and
  nonMinified (baseline profile capture) builds."
Change-Id: I487fa71083921682173f04fcbb477be5baf165f8

I've landed a change that will set the following properties also when the benchmark build type already exists:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

As well as the following for agp 8.0:

isDebuggable = false

I'm going ahead and closing this - if you've further questions please answer here and will reopen. Thanks.

The following release(s) address this bug.It is possible this bug has only been partially addressed:

• androidx.benchmark:benchmark-baseline-profile-gradle-plugin:1.4.0-alpha04

Since release signing config is used by default instead of debug is not mentioned in release notes - maybe this is a bug?

Cause the last time I found it mentioned in the release notes was in version 1.1

signingConfig.debug is used as the default signing config (b/153583269)

So, if the switch to the release one indeed happened - maybe it's an issue?

The new build types should inherit the certificates from the release build type, unless none is specified in which case debug is used: code link

This should be updated in the documentation, thanks for mentioning it. Will take care of updating it.

Macrobenchmark doesn't inject it.

It should, you just have to add it to the macrobench module itself. The goal is to have full tracing work in macrobench and studio without you needing to bundle the binary in your app.

+1 to what Chris said in comment#8

To clarify:

• Macrobenchmark (library) does not depend on tracing-perfetto-binary
• If an app-specific Macrobenchmark (test module) declares a dependency on tracing-perfetto-binary, Macrobenchmark (library) will inject it to the app at the time of tracing

As an example, Compose Macrobenchmark target (an app, so neither library of test module) does not depend on tracing-perfetto-binary:

• https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/integration-tests/macrobenchmark-target/build.gradle;l=20;drc=02eda44f1d669bcda58a997aebcb90fb113c8f2a

..but its Macrobenchmark (test module) does (indirectly through internal-testutils-macrobenchmark):

• https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/integration-tests/macrobenchmark/build.gradle;l=40;drc=d80729fe56e7bfdcab04e9812acba0724d6e15ac
• https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:testutils/testutils-macrobenchmark/build.gradle;l=28;drc=7717bcc0169cca4a1c54d45b620f0392223c9e90

So, you should not have to do what Compose Wear did here (though it still works as Macrobenchmark knows how to handle it):

• https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:wear/compose/integration-tests/macrobenchmark-target/build.gradle;l=54;drc=1fd05eef6c740c8b189f3081cda6b002659a6424

Hey, might there be a bug introduced or some misconfiguration?

Folks in performance samples mention this doesn't work the way you described (having binaries in macrobench + runtime in target).

I tried to verify it and when I have the dependencies in :macrobench, I don't see the full traces in perfetto, but when I add it to target, it crashes with the exception mentioned in the samples.

Version incompatibility

Regarding:

         Process: com.google.samples.apps.nowinandroid.demo.benchmark, PID: 20835
         java.lang.NoClassDefFoundError: Failed resolution of: Landroidx/tracing/perfetto/Tracing;
             at androidx.compose.runtime.tracing.TracingInitializer$create$1.isTraceInProgress(Unknown Source:0)
             at coil.util.-Contexts.isTraceInProgress(SourceFile:1)
             at androidx.compose.ui.platform.WrappedComposition$setContent$1$1.invoke(SourceFile:3)
             at androidx.compose.ui.platform.WrappedComposition$setContent$1$1.invoke(SourceFile:1)
             at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(SourceFile:7)

We've had to do a bunch of API renames in order to move from Alpha -> Beta (we're trying to go to Stable ASAP)

Temporarily (until tomorrow, Aug 23, 2023), this is a safe configuration to use (versions-wise):

androidx.compose.runtime.runtime-tracing-1.0.0-alpha03 # in app (target)
androidx.tracing.tracing-perfetto-1.0.0-alpha11 # in benchmark
androidx.tracing.tracing-perfetto-binary-1.0.0-alpha11 # in benchmark

Apologies for a state of things. We're moving to stable with both Benchmark and Tracing, which will make things easier. And hopefully no more breaking renames.

From tomorrow's Jetpack release:

androidx.compose.runtime.runtime-tracing-1.0.0-alpha04 # in app (target)
androidx.tracing.tracing-perfetto-1.0.0-beta02 # in benchmark
androidx.tracing.tracing-perfetto-binary-1.0.0-beta02 # in benchmark

Easier (maybe) alternative

1. Declare a dependency in the target app on androidx.compose.runtime.runtime-tracing
2. Check what version of tracing-perfetto it uses through a command below (separate section below)
3. Use that version of tracing-perfetto and tracing-perfetto-binary in the benchmark

These might come in handy

Verifying dependencies actually used

./gradlew app:dependencies | grep tracing

Forcing a particular version

implementation("androidx.tracing:tracing-perfetto:1.0.0-alpha11") {
        version {
            strictly("1.0.0-alpha11")
        }
}

I have also raised the same issue under Tracing component as the documentation pointed to that: https://issuetracker.google.com/u/1/issues/296768173

Re comment#12, can you confirm comment#11 unblocks you?

Phew! Yes, that works now. It looks like maybe some transitive dependency was updating the tracing-perfetto to 1.0.0-beta01 and hence compose traces were not available. Forcing it to the particular version mentioned fixed the issue. Hoping tomorrow's release do not break anything further!

Suggestion: I know alpha/beta versions are bound to breaking changes, but can we have a section in the documentation which can point out the working approach for scenarios like this and make it easier to adopt? This can also help to identify the last working set and check from there on-wards for breaking changes.

Makes sense. Apologies for the issue, we try not to leave libraries in a state that requires workarounds in the form of comment#11

Jakub, is there any more we can do here for this release (potentially clarifying in relnotes or error message), or should we close?

I think a compatibility matrix (compose-runtime-tracing, tracing-perfetto, benchmark) could be too complicated

The simplest I can see is:

• moving to RC/Stable soon (WIP)
• throw an exception (crash) the benchmark when we cannot find the binaries and tell the user what dependency to add and where (CL in progress: http://aosp/2743602)
• on DAC, mention that if a user gets the exception: unable to find class androidx.tracing.perfetto.Trace..., they should upgrade compose-runtime-tracing to the latest version

@Chris, @Tomáš, WDYT?

I don't think DAC is a great place to surface the unable to find class error - can we hook the failed classload somehow, can we give a better error message in the target app in some way from within the tracing perfetto library?

I kind of agree as well DAC is the right place for this. We have some error messages / misconfiguration for Macrobenchmark on DAC around build variants and multi-module setup. But these errors are a good fit, because the libraries can't do it in any away and it's a form of guidance to properly configure your project.

Regarding configuration matrix for dependencies. We're having more and more dependencies that are compatible only with certain versions, maybe it's time to rethink having a compatibility matrix?

You need to have proper versions of:

• AGP
• Macrobenchmark
• Profile Installer
• (potentially) Baseline Profile Gradle Plugin
• (and now) Perfetto

If misconfiguration crashes with some reasonable message (we could ask/answer that on StackOverflow), that SG to me. It's worse when it just doesn't work.

Crashing with some reasonable message (we could ask/answer that on StackOverflow) is what we now moved to - changes will go out in the next Benchmark+Tracing release.

Both possible crash scenarios described below.

NoClassDefFoundError

java.lang.NoClassDefFoundError: Failed resolution of: Landroidx/tracing/perfetto/Tracing; means the app is using an old version of compose-runtime-tracing. Updating to the latest version fixes the issue.

If we try to detect a failed classload, we'd have to release a newer version with this logic, which defeats the point of the change, as the fix is to simply use a newer version. And since we're now in Beta, the name of the class won't change.

RuntimeException

RuntimeException: Issue while enabling Perfetto SDK tracing in ${perfettoSdkConfig.targetPackage}: Perfetto SDK binary dependencies missing. Required version: [...] To fix, declare the following dependency [...] will be thrown from the next Benchmark release whenever we fail to resolve tracing-perfetto-binary in the benchmark. The error message is very descriptive (see in full).

Simplifying the workflow

From the user's perspective, the easiest would be to just use the latest version of compose-runtime-tracing. I'll try to get it to 1.0.0 stable ASAP to encourage users to upgrade (currently they're using an Alpha). If they need to supply tracing-perfetto-binary dependency, Benchmark will inform them how to do that (through the RuntimeException described above). Does that sound reasonable?

I'm going to close the bug as IMO http://aosp/2743602 fixed the issue, but please re-open if you think more work is required here.

The following release(s) address this bug.It is possible this bug has only been partially addressed:

• androidx.benchmark:benchmark-common:1.2.0-rc01
• androidx.benchmark:benchmark-macro:1.2.0-rc01

Something's wrong, I think.

When I use

androidx.compose.runtime.runtime-tracing-1.0.0-alpha04 # in app (target)
androidx.tracing.tracing-perfetto-1.0.0-rc01 # in benchmark
androidx.tracing.tracing-perfetto-binary-1.0.0-rc01 # in benchmark

androidx.benchmark:benchmark-macro-juni4:1.2.0-rc01 #in benchmark

It fails with

java.lang.RuntimeException: Issue while enabling Perfetto SDK tracing in com.google.composeviewbenchmarks.target: Perfetto SDK binary verification failed. Required version: 1.0.0-beta02. Error: androidx.tracing.perfetto.security.IncorrectChecksumException: Invalid checksum for file: /data/user/0/com.google.composeviewbenchmarks.target/code_cache/libtracing_perfetto.so. Ensure you are using correct version of the library and clear local caches.. If working with an unreleased snapshot, ensure all modules are built against the same snapshot (e.g. clear caches and rebuild).

Should we check only older versions? Or does this need to be an exact version?

Adding

version {
   strictly("1.0.0-beta02")
}

resolves the problem

Re comment#24, error message from compose-runtime-tracing (the target app) is saying it requires tracing-perfetto-binary beta02, so that's what should be set in benchmark. Not sure why strict keyword is required. This is a separate issue (from what the bug was originally opened for), and since we now track SLO, let's open a new one for new issues instead of reusing this one as a grab-all. For issue from comment#24 I've created b/302672687.

(marking as fixed as the original issue was fixed, for comment#24, please follow b/302672687)