IssueTracker

Context

For the past few months, we have been seeing more and more frequent minify*WithR8 tasks taking up to 1h30 (our CI timeout is set to 2h for the entire pipeline so we definitely have even longer durations that are terminated early to avoid waiting indefinitely).
When everything goes smooth, this task usually takes ~10 min, but when something goes wrong (and we currently don't really know why) it takes way more time.

Setup

Our current CI setup is as follow:

• Debian bullseye
• OpenJDK 17
• 16 vCPUs
• 128GB RAM
• Ephemeral machines
  • They are not re-used between builds
  • No configuration cache is used
  • No daemon are persisted
  • All Gradle build dependencies are downloaded upfront and available in cache

Here is our Gradle build setup, I can give more details if you think they are important:

• AGP 8.7.2 (R8 8.7.18)
• Kotlin 2.0.21
• minSdk 24
• compileSdk/targetSdk 35
• gradle.properties

  org.gradle.daemon=false
  org.gradle.vfs.watch=false
  org.gradle.workers.max=8
  org.gradle.configuration-cache=false
  org.gradle.jvmargs=-Xmx90g -Xms2g
  kotlin.daemon.jvmargs=-Xmx30g -Xms1g
  

And last but not least, our app is a fairly large app, with many third party dependencies.
The universal APK size is ~56MB (~14MB of native libraries, ~20MB of dex files).

Results

Now, here is the graph (graph.png) of an attempt leading to 1h50m build with ~1h30m of minify*WithR8.

...and the corresponding Gradle Scan insights (I do not trust these values very much, because they only track the value at the end of the build, but we don't have the new Develocity RAM/CPU usage graph yet):

• Peak heap memory usage G1 Old Gen 20.8 GiB/90 GiB (23.2%)

• Gradle Process Info
  Gradle-Process-1741-capacity	33.13 GB
  Gradle-Process-1741-gcTime	0.42 minutes
  Gradle-Process-1741-gcType	-XX:+UseG1GC
  Gradle-Process-1741-max	90.0 GB
  Gradle-Process-1741-uptime	110.74 minutes
  Gradle-Process-1741-usage	19.26 GB

• Kotlin Process Info
  Kotlin-Process-2161-capacity	1.67 GB
  Kotlin-Process-2161-gcTime	0.35 minutes
  Kotlin-Process-2161-gcType	-XX:+UseG1GC
  Kotlin-Process-2161-max	30.0 GB
  Kotlin-Process-2161-uptime	106.85 minutes
  Kotlin-Process-2161-usage	0.5 GB

• We did not print R8 timings yet (com.android.tools.r8.printtimes), but the early reports did not show anything particular. I'll keep an eye on it and report back when it happens again.

Our hypotheses

• There is a deadlock/leak/bug somewhere in AGP/R8
• Proguard rules:
  We have a significant amount of proguard rules (5k+ lines) coming from our dependencies.
  And one in particular (adyen-3ds2-2.2.21) is declaring 3.3k lines of rules (we are going to ask them why, and probably remove them in our build pipeline)… This will likely increase the total R8 time, but I'm not sure this is the root cause of the main issue.

What we’ve already tested

• In the past, we used to build multiple variants at the same time, with more than 1 R8 task in parallel, and it worked fine. We had to turn that off because of the increased duration and timeouts.
• Allocate even more memory… but this doesn't seem to be the issue. We are already giving 90G to Gradle…

What we will (or could) test in the future

• Manually update R8 versions.
  While this solution seems good, it is very… very.. time consuming for us, and let's be honest, it's a real pain that we don't want to go through!
• Tool execution profiles
  To run R8 (and Lint) outside the Gradle process and (maybe?) prevent unexpected interactions.
• Run a full build without R8, kill all daemons, then run the full R8 build. This will still increase our overall build time (we have a fairly long configuration phase that we will pay twice) at the benefit of maybe a better stability.

In the meantime, we’ve also generated a compiler-input dump that I can share with you (not sure how, let me know), and here is the overview:

  219.1 MiB [######################]  program.jar
    5.7 MiB [                      ]  library.jar
    4.2 MiB [                      ]  art-profile-1.txt
  364.0 KiB [                      ]  proguard.config
  112.0 KiB [                      ]  desugared-library.json
    8.0 KiB [                      ]  classpath.jar
    4.0 KiB [                      ]  build.properties
    4.0 KiB [                      ]  r8-version

Thanks for your support.