IssueTracker

Thank you for the report.

When the compile time fluctuate this much, it could be caused by the JVM getting close to max memory and the GC portion of the compile time goes up.

As you have a dump, then you can easily reproduce the same build with different settings using tools/compiledump.py. tools/compiledump.py use the gsutil to download the compiler from our build repository. If you don't have that tool, you can get the compiler with curl -O https://storage.googleapis.com/r8-releases/raw/8.3.35/r8lib.jar (for version 8.3.35), and then add --r8-jar r8lib.jar to the commands below. The commands all pass -da which disables assertions. tools/compiledump.py enabled assertions by default, but that makes compilation even slower, and AGP does not do that.

1. Are you seeing the same loooooong compile time locally when running tools/compiledump.py -da -d <dump>?
2. If so you can try to pass the -verbose:gc to the JVM when running, should be doable like this: tools/compiledump.py -da -J=-verbose:gc -d <dump>. Seems like the output is not streamed so you will have to wait for the command to finish.
3. If that shows massive GC, you can try different heap sizes, to see if changing that affects compilation time, e.g. like this: tools/compiledump.py -da --xmx 8G -J=-verbose:gc -d <dump>

If possible you can share the dump privately with sgjesse@google.com and ricow@google.com, then we can also try to see if we can reproduce the issue.

The CI calculates the values for xmx etc... depending on the current machine. The one use here should have enough resources to build the app without issues.

This is what it calculated for this particular machine, which has 32 GB of RAM (properties written inside of ~/.gradle/gradle.proprties).

org.gradle.jvmargs = -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -XX:MaxMetaspaceSize=1g -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -Xmx16854m -Xms16854m
kotlin.daemon.jvmargs = -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -XX:MaxMetaspaceSize=1g -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -Xmx7223m -Xms7223m

I let gradle and kotlin use 75% of the total memory, 70% of that is for gradle, the remaining 30% for kotlin.

I once ran the build with -Dcom.android.tools.r8.printmemory, but the CI stopped after it reached its limit of 3.5 hours. This is what I saw:

R8 is running with total memory:17683185664
R8 is running with free memory:14891000672
R8 is running with max memory:17683185664

I have multiple zips inside the directory of dumpinputtodirectory and compiledump.py takes roughly 10 minutes to process the largest one. Even when I pass --Xmx 8G.

I'll try to play a bit with the memory settings and see if that's the problem, as you suspect. I could also try to run R8 in a separate process as mentioned here.

I'll try to can figure this out and I'll see if I can share the dump in case I can't find anything.

Thank you for the prompt response.