In recent 1.3.0 update of androidx.benchmark library there's a breaking change. Now release signing config is used by default instead of debug. It's not mentioned in a corresponding release notes whatsoever.

As per official documentation, there's a way to customize benchmark and baseline profiles build types.

Having reviewed the source code of baseline profiles Gradle plugin, I can see that creating a custom configuration (build type) results in inconsistent final variant properties. I'll explain directly in code.

There's a class BaselineProfileAppTargetAgpPlugin responsible for creating corresponding build types (nonMinifiedRelease, benchmarkRelease). It's actually done in createBuildTypesWithAgp81AndAbove() method. There's a helper method createExtendedBuildTypes() that checks whether a build type exists or not, and executes different configuration blocks based on build type existence (newConfigureBlock, overrideConfigureBlock).

Now, baseline profiles build type configuration is identical both for new and existing build type. However, benchmark build type is under configured: at least isProfileable is not set to true for existing build type, probably there's more.

This issue is not about some specific configuration flag, but the general approach of dealing with external configuration. As a developer adopting baseline profiles, it seems extremely risky to me using a custom configuration due to how it's applied under the hood and the fact it may break default configuration.

The reason why I mentioned the release signing config in the beginning is because I want to use debug signing config. I'm unable to find any way to configure already created by your plugin build type. So the ideal solution I see is to allow configuring existing (already created by plugin) build types (nonMinifiedRelease, benchmarkRelease) instead of forcing developers actually create these. This way there won't be any discrepancies between the default and customized build types.

Summing up, there are two action points:

Determine if it's possible to eliminate discrepancies between configuring new and existing build types
Figure out how to change signing config back to debug without affecting default build type configuration (ref. to 1)