IssueTracker

Thanks for your detailed post.

However, benchmark build type is under configured: at least isProfileable is not set to true for existing build type, probably there's more.

We should set isProfileable = true by default when overriding an existing benchmark build type.

This issue is not about some specific configuration flag, but the general approach of dealing with external configuration. As a developer adopting baseline profiles, it seems extremely risky to me using a custom configuration due to how it's applied under the hood and the fact it may break default configuration.

I agree that is not great but this is a little tricky to do. For custom baseline profile build types we override all the properties. For benchmark I left it open to configure but it's mostly about these 2 properties:

isMinifyEnabled
isShrinkResources

I don't have a way to see if the user is setting them before overriding, so for this reason, I'd prefer not to. I agree with you that some other properties could be set by default to make this easier, i.e.:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

The reason why I mentioned the release signing config in the beginning is because I want to use debug signing config.

In the specific of your issue, i.e. using a debug certificate can you override the benchmark and baseline profile setting? You should be able to do something like:

android {
    buildTypes {
        release { ... }
        debug { ... }
        benchmarkRelease {
            ...
            signingConfig signingConfigs.debug
        }
        nonMinifiedRelease {
            ...
            signingConfig signingConfigs.debug
        }
    }
}

Project: platform/frameworks/support
Branch: androidx-main
Author: Marcello Albano <maralb@google.com>
Link:      https://android-review.googlesource.com/3303459

Added override for debuggable and profileable for benchmark builds in bpgp

Added override for debuggable and profileable for benchmark builds in bpgp

Test: ./gradlew :benchmark:benchmark-baseline-profile-gradle-plugin:test
Bug: 369213505
Relnote: "isProfileable is always overridden in benchmark builds,
  and isDebuggable is also now always overridden in both benchmark and
  nonMinified (baseline profile capture) builds."
Change-Id: I487fa71083921682173f04fcbb477be5baf165f8

I've landed a change that will set the following properties also when the benchmark build type already exists:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

As well as the following for agp 8.0:

isDebuggable = false

I'm going ahead and closing this - if you've further questions please answer here and will reopen. Thanks.

The following release(s) address this bug.It is possible this bug has only been partially addressed:

• androidx.benchmark:benchmark-baseline-profile-gradle-plugin:1.4.0-alpha04

Since release signing config is used by default instead of debug is not mentioned in release notes - maybe this is a bug?

Cause the last time I found it mentioned in the release notes was in version 1.1

signingConfig.debug is used as the default signing config (b/153583269)

So, if the switch to the release one indeed happened - maybe it's an issue?

The new build types should inherit the certificates from the release build type, unless none is specified in which case debug is used: code link

This should be updated in the documentation, thanks for mentioning it. Will take care of updating it.

Chris suggested I share some feedback from my recent try to have method tracing for the macro bench run. I was running a compose TrivialListScrollBenchmark I added

android { 
   defaultConfig {
     testInstrumentationRunnerArguments["androidx.benchmark.profiling.mode"] = 'MethodTracing'
   }
}

Into gradle and made a run. Each iteration created two files:

1. perfetto-trace which only contains the manually added traces (as if MethodTracing wasn't enabled)
2. method.trace files. a file for the 0th iteration contains some trace which I can open in studio or https://profiler.firefox.com, but I don't understand what part of the execution was tested. it shows a trace for a one Choreographer.doFrame where we composed a new item, but the whole run includes many frames for multiple flings and I am not sure what frame it chose. another interesting part is that method.trace files for each other iteration are just zero sized files Attaching those files

I landed this change to make this more consistent.

https://android-review.git.corp.google.com/c/platform/frameworks/support/+/2724235

--

In general, MethodTracing when enabled captures the time between a cold start to the point where the iteration is complete. We only turn on method tracing for iterations that involve cold starts. You will therefore not see a 1-1 mapping between the number of iterations and the number of method traces given you might be measuring HOT starts for e.g.

I fixed the issue where you were seeing 0 byte method trace files as well. Let me know if you have other questions.

I'm going to do one more pass on the behavior here to make it match micro more closely for the final 1.2 version:

• One method tracing pass at the end, after measurements, with measurements disabled (may want to still throw if no metrics captured, though)
• Document behavior surprises for micro vs macro
  • Starts in first startActivityAndWait for now
  • Add warning to output about why missing in living process case

With the above fix, macrobench method tracing is in a good enough/experimental state to ship with 1.2, so punting further work to 1.3

Remaining issues / gaps between method tracing in micro vs macro:

• Method trace should be done in separate measurement phase, more like microbench
• Method tracing can only start for an iteration if process is launched cold, so only one occurs frequently, e.g. when measuring warm start (CL in #comment12 adds warning about this)
• Method trace not embedded into perfetto trace

Bugjuggler: wait 1 week

Last thing here should be to not capture metrics (or at least label metrics separately) for method tracing iterations.

The following release(s) address this bug.It is possible this bug has only been partially addressed:

• androidx.benchmark:benchmark-macro:1.3.0-alpha05