IssueTracker

I am not sure I understand the use case. how can the benchmark be code to real world scenario when it's not possible to do right now ? which scenario is it ?

In any case, since this would be for benchmarking, this would clearly not be available through the public DSL. We should find a semi-private way of doing this (maybe the private variant API object could offer that functionality for instance or a property).

We want benchmarks to measure code after Progaurd / R8, but it's not possible to turn that on for androidTests in library modules at the moment (to my knowledge?)

Benchmarks are also a public facing thing, but we have a plugin to help configure gradle builds for our users, so if support for this ends up in a private API, we could try to keep those usages localized to our code perhaps.

this is not part of our OKR at this point so we are not talking soon. at first glance, we would need to simulate usage patterns to minify against and such, this seems substantial amount of work. there are not a lot of library module that have android tests, most only rely on unit-tests.

how important is this ? we are out of PM right now but I suspect the next step will be to negotiate with J. Eason and xav@ to scale a priority level.

This is a high priority request for Compose, to enable their benchmarks to measure release accurate performance. (Micro) Benchmarks are library modules, as they don't need the complexity of multi-apk tests - they're self measuring APKs that depend on libraries. (d.android.com/benchmark)

there are not a lot of library module that have android tests, most only rely on unit-tests.

To clarify, this is for com.android.library modules, not jars - I'd expect most of those to use android tests (all of the libraries in jetpack for example do).

we would need to simulate usage patterns to minify against and such, this seems substantial amount of work

Simulate usage patterns? I don't understand - the dev can themselves provide a keep rule for test infra / classes if necessary. Long term, keep rules should be provided by test libraries.

We've been experimenting with ways to work around this for Compose. Performance results from R8 seem significantly different, and would enable us to measure much more accurately. I've tried to come up with a workaround using a com.android.app module, and while it almost works (and we can get measurements), it's extremely hacky and doesn't let us run tests anymore via Studio: b/184378053#comment3

Bumping this request, as Compose has recently had more interest in the ability to benchmark with and without R8 enabled.

We're fine if the default implementation doesn't work with minification (tree shaking) - we're happy to supply those rules ourselves, or simply evaluating with minification off to take advantage of other optimizations.

Juan, this might be something to put on our OKR in the near future, I think you chat with Amanda to set the priority.

Hey everyone, I am catching up on feature requests and saw this one. I'll schedule time for us to talk about this in a few days.

Ivan, can you provide a rough estimate on how long this would take ?

I think this is a duplicate of http://b/263197720. That bug talks about inter-project publishing, but it also applies to library androidTest component getting a minified main library component in its runtime classpath.

My understanding of this feature request is the following:

• As a library author, I'd like to write microbenchmarks in my library subproject that measure performance of my library after it's been processed by R8 [1]

Chris/Dustin, is this correct? If [1] is correct, this is a duplicate of a duplicate of http://b/263197720.

As a small correction, it's:

• As a user of microbenchmarks, I'd like to measure performance of code processed by R8

For a bit more context, we generally recommend library and app devs create a completely new, empty library module to add microbenchmarks to, e.g. in androidx. It's generally done in a separate module as it wants different compilation behavior (profileable, non-debuggable, ideally R8...) than the regular instrumentation tests of the library in mylib/src/androidTest/.

For this reason, we don't particularly care about the minified main library component from the same module.

This librarytest-only focus does have the downside that app devs must pull code into a library to be microbenchmarks, and maybe that could be something improved if there was a good minification story across the test and app module boundary, but that's a separate (but related) concern for the future.

For right now, we care about libraries with empty main directories running R8 on the code in androidTest and its dependencies.

Thanks for clarifying this. In terms of benchmarking R8-processed library there seem to be two options:

1. minify both library and its usages together (in this case they are coming from microbenchmark tests), and collect performance data
2. run R8 on the library only, and use non-minified (i.e. public library APIs) from microbenchmark tests to collect performance data; this is captured in http://b/263197720.

IMO both 1) and 2) are valid use-cases. For 1) you'd like to see how your library behaves given concrete API usages, while 2) tests library "full-surface".

Could 1) also be modelled with an application that depends on the library, uses some APIs, and then microbenchmark tests from its androidTest are doing the profiling?

For 1) you'd like to see how your library behaves given concrete API usages,

Yes, this is it exactly. We want 1, assuming that it means all of the library dependencies of the benchmark and it's androidTest code would be run through R8 together.

As there's no content in my-benchmark/src/, I don't think 2 helps our issue.

Could 1) also be modelled with an application that depends on the library, uses some APIs, and then microbenchmark tests from its androidTest are doing the profiling?

We experimented with this years ago when we started the project, it could work conceptually, but the issue is that the androidTest dir build wasn't factored into the tree shaking step, so every test had to be accompanied by manual keep rules/annotations to the code it invodes. I'd be fine having to specify keep rules for JUnit and @Test methods, but if we wanted libraries to be minified, they had to be in the app, and the benchmark had to cross that test/apk boundary constantly.

Another option we explored was declaring each test twice (as a @Keep function in src/ and wrapper in androidTest/ so Studio could see/run the test method). which technically worked, but was similarly cumbersome.

I think I have all requirements right now.

http://b/37054038 tracks a FR to have an option to run R8 on tests, and for:

• library androidTest: there is a single APK AGP produces, and this would allow you to run R8 on both androidTest code and its runtime classpath (which includes all of its dependencies&main). Every class from your tests would be an entry point. Also, when main component of the library is processed with R8, we'd apply its mapping file.
• application androidTest: we produce 2 APKs, so R8 runs on androidTest code and its runtime classpath (includes only its dependencies, and all duplicate libraries from main are removed). Every class from your tests would be an entry point. Also, when main component of the application is processed with R8, we'd apply its mapping file.

Is this a correct summary? If so, we can de-duplicate?

P.S. Related to issues with taking test usages into account when running R8 on application code, this is http://b/126429384.

Yes, thanks, duping against b/37054038 sounds good to me.

IIUC, for library androidTest, we are going to run R8 for its program code(e.g. my-benchmark/src/ suppose it is not empty) together with androidTest code and other dependencies, right?

If so, I think b/37054038 is slightly different from the specs mentioned in comment#16

b/37054038 wants to:

enable it on the test apk for libraries. I do not want progaurd to run on the library itself, however. which seems to suggest the program code should not be minified.

I do not want progaurd to run on the library itself, however.

My reading of this statement on that bug is that they don't want the library's .aar or .jar to be minified (as part of the actual library build), but they DO want the library minified as part of the library test build, which is why I agreed with the duping suggestion.

From a later comment, comment13:

The original comment is pretty clear that it's for libraries which do not want to run R8 on their main code but want to on their test APK to validate things like the embedded rules.

Testing embedded rules implies that the .aar/.jar from the library won't be minifed, but they want to test an environment where the library is minified when embedded in an apk, via the tests.