IssueTracker

I was able to reproduce it without disabling the cache for the transform. This issue is always reproducible if the transform is executed: https://ge.solutions-team.gradle.com/c/xsxlfl5c3odri/agxvui3h3w2f2/task-inputs?expanded=WyIzaHhqN3N4eWgyejdjLWRleGRpcnMiXQ

Thanks, I can reproduce this issue with the following steps (adapted from your steps):

1. git clone https://github.com/cdsap/CpuExperiment
2. git checkout reproducing_issue_dex
3. Run

./gradlew --stop && ANDROID_HOME=/path/to/Android/Sdk ./gradlew clean :app:mergeExtDexDemoDebug --no-scan --no-build-cache

4. Move (copy and delete) the following files in ~/.gradle/caches/8.9/transforms to Desktop/Build1:

instrumented_ui-graphics-release-runtime/classes.dex
instrumented_ui-graphics-release-runtime.jar

5. Run step 3 again
6. Move those files again to Desktop/Build2
7. Compare Desktop/Build1 and Desktop/Build2 (e.g., with the diff tool), we'll see that the jar files (containing .class files) are the same, but the dex files are different.

This seems to be a bug in D8.

=======

Btw, by repeating the same experiment for the :app:hiltJavaCompileDemoDebug task, I found and filed a different bug for Dagger/Hilt: https://github.com/google/dagger/issues/4398

Hi Søren, could you look into why the output dex files are non-deterministic even though the input jar files are the same?

When I run dexdump -h (-h -- display file header details -- is needed), I see the differences in the dex files are in the following form:

76952,76953c76952,76953
< annotations_off     : 838108 (0x0cc9dc)
< class_data_off      : 811976 (0x0c63c8)
---
> annotations_off     : 838104 (0x0cc9d8)
> class_data_off      : 811975 (0x0c63c7)

I'm attaching the relevant files for your investigation.

$ dexsegments.py --no-build /usr/local/google/home/ricow/Downloads/Build1_classes.dex > /tmp/dexsegments1
$ dexsegments.py --no-build /usr/local/google/home/ricow/Downloads/Build2_classes.dex > /tmp/dexsegments2
$ diff /tmp/dexsegments{1,2}
1c1
< Running: /usr/local/google/home/ricow/src/r8/third_party/openjdk/jdk-11/linux/bin/java -ea -jar /usr/local/google/home/ricow/src/r8/build/libs/r8.jar dexsegments /usr/local/google/home/ricow/Downloads/Build1_classes.dex
---
> Running: /usr/local/google/home/ricow/src/r8/third_party/openjdk/jdk-11/linux/bin/java -ea -jar /usr/local/google/home/ricow/src/r8/build/libs/r8.jar dexsegments /usr/local/google/home/ricow/Downloads/Build2_classes.dex
3c3
<  - EncodedArrays: 259 / 13
---
>  - EncodedArrays: 256 / 13
13c13
<  - StringData: 264817 / 5873
---
>  - StringData: 264816 / 5873

Inputs are actually the exact same jar:

$ diff ~/Downloads/Build{1,2}_instrumented_ui-graphics-release-runtime.jar

Our disassembly is the same:

$ disasm.py --no-build --all ~/Downloads/Build1_classes.dex > /tmp/dis1
$ disasm.py --no-build --all ~/Downloads/Build2_classes.dex > /tmp/dis2
$ diff /tmp/dis{1,2}
1c1
< Running: /usr/local/google/home/ricow/src/r8/third_party/openjdk/jdk-11/linux/bin/java -jar /usr/local/google/home/ricow/src/r8/build/libs/r8.jar disasm --all /usr/local/google/home/ricow/Downloads/Build1_classes.dex
---
> Running: /usr/local/google/home/ricow/src/r8/third_party/openjdk/jdk-11/linux/bin/java -jar /usr/local/google/home/ricow/src/r8/build/libs/r8.jar disasm --all /usr/local/google/home/ricow/Downloads/Build2_classes.dex

manually just compiling the zips does not give a difference:

$ for i in {1..20}; do d8.py --no-build --version 8.5.35 -- --debug --output /tmp/foo$i.zip ~/Downloads/Build1_instrumented_ui-graphics-release-runtime.jar --min-api 21; done
$ for i in {1..20}; do diff /tmp/foo1.zip /tmp/foo$i.zip; done

Using the repro project, and dumping the inputs to R8 I get:

$ rm /usr/local/google/home/ricow/.gradle/caches/8.9/transforms/014eb84519673e0b0674f560b3e06212/transformed/instrumented_ui-graphics-release-runtime/classes.dex && ./gradlew --stop && ANDROID_HOME=~/Android/Sdk ./gradlew -Dcom.android.tools.r8.dumpinputtodirectory=/tmp/dumps/ clean :app:mergeExtDexDemoDebug --no-scan --no-build-cache
$ rm /usr/local/google/home/ricow/.gradle/caches/8.9/transforms/014eb84519673e0b0674f560b3e06212/transformed/instrumented_ui-graphics-release-runtime/classes.dex && ./gradlew --stop && ANDROID_HOME=~/Android/Sdk ./gradlew -Dcom.android.tools.r8.dumpinputtodirectory=/tmp/dumps/ clean :app:mergeExtDexDemoDebug --no-scan --no-build-cache
$ ls /tmp/dumps
$ ls -l /tmp/dumps
total 77720
-rw-r--r-- 1 ricow primarygroup 13250142 Aug 21 09:47 dump866702451939926.zip
-rw-r--r-- 1 ricow primarygroup 26538677 Aug 21 09:48 dump866709161754606.zip
-rw-r--r-- 1 ricow primarygroup 13250142 Aug 21 09:48 dump866738596947431.zip
-rw-r--r-- 1 ricow primarygroup 26538704 Aug 21 09:48 dump866745176793724.zip

$ diff /tmp/dumps/{dump866702451939926.zip,dump866738596947431.zip}
$ diff /tmp/dumps/{dump866709161754606.zip,dump866745176793724.zip}
Binary files /tmp/dumps/dump866709161754606.zip and /tmp/dumps/dump866745176793724.zip differ

i.e., the task is doing two d8 compilations and the input to the last one is different

$ unzip -d /tmp/adump /tmp/dumps/dump866709161754606.zip > /dev/null
$ unzip -d /tmp/bdump /tmp/dumps/dump866745176793724.zip > /dev/null

$ diff -rq /tmp/{a,b}dump
Files /tmp/adump/program.jar and /tmp/bdump/program.jar differ
$ unzip -d /tmp/adump/progout /tmp/adump/program.jar > /dev/null
$ unzip -d /tmp/bdump/progout /tmp/bdump/program.jar > /dev/null

$ diff -rq /tmp/{a,b}dump/progout 
Files /tmp/adump/progout/classes47.dex and /tmp/bdump/progout/classes47.dex differ

Based on the class names in classes47 it is obvious that this is the output of the first compilation.

Just compiling the dump for the first compilation in a loop:

$ for i in {1..5}; do compiledump.py -o /tmp/compiledump/out_$i.jar -d /tmp/dumps/dump866702451939926.zip --temp /tmp/compiledump; done
$ for i in {1..5}; do diff /tmp/compiledump/out_1.jar /tmp/compiledump/out_$i.jar; done

No difference.

Compare markers

$ extractmarker.py --no-build /tmp/compiledump/out_1.jar
Running: /usr/local/google/home/ricow/src/r8/third_party/openjdk/jdk-11/linux/bin/java -ea -jar /usr/local/google/home/ricow/src/r8/build/libs/r8.jar extractmarker /tmp/compiledump/out_1.jar
/tmp/compiledump/out_1.jar: ~~D8{"backend":"dex","compilation-mode":"debug","desugared-library-identifiers":["com.tools.android:desugar_jdk_libs_configuration:2.0.4"],"has-checksums":false,"min-api":21,"sha-1":"engineering","version":"8.5.35"}

$ extractmarker.py --no-build /usr/local/google/home/ricow/.gradle/caches/8.9/transforms/014eb84519673e0b0674f560b3e06212/transformed/instrumented_ui-graphics-release-runtime/classes.dex
Running: /usr/local/google/home/ricow/src/r8/third_party/openjdk/jdk-11/linux/bin/java -ea -jar /usr/local/google/home/ricow/src/r8/build/libs/r8.jar extractmarker /usr/local/google/home/ricow/.gradle/caches/8.9/transforms/014eb84519673e0b0674f560b3e06212/transformed/instrumented_ui-graphics-release-runtime/classes.dex
/usr/local/google/home/ricow/.gradle/caches/8.9/transforms/014eb84519673e0b0674f560b3e06212/transformed/instrumented_ui-graphics-release-runtime/classes.dex: ~~D8{"backend":"dex","compilation-mode":"debug","desugared-library-identifiers":["com.tools.android:desugar_jdk_libs_configuration:2.0.4"],"has-checksums":true,"min-api":21,"version":"8.5.35"}
                                                                                                                                                                                                                                                                                                                         ^^^^

Hardcode 8.5.35 to always use checksums, rerun experiment:

$ for i in {1..5}; do compiledump.py --r8-jar ~/src/r8/build/libs/r8.jar -o /tmp/compiledump/out_$i.jar -d /tmp/dumps/dump866702451939926.zip --temp /tmp/compiledump; done
$ for i in {1..5}; do diff /tmp/compiledump/out_1.jar /tmp/compiledump/out_$i.jar; done
Binary files /tmp/compiledump/out_1.jar and /tmp/compiledump/out_2.jar differ
Binary files /tmp/compiledump/out_1.jar and /tmp/compiledump/out_3.jar differ
Binary files /tmp/compiledump/out_1.jar and /tmp/compiledump/out_4.jar differ
Binary files /tmp/compiledump/out_1.jar and /tmp/compiledump/out_5.jar differ

Compiledump compilation does not use store and apply the checksum bit! and we produce different output

Dumping out all checksums, sorting and comparing:

$ diff /tmp/sorta /tmp/sortb
72c72
< Landroidx/compose/ui/graphics/colorspace/ColorModel$$ExternalSyntheticBackport0; -> 13062835561
---
> Landroidx/compose/ui/graphics/colorspace/ColorModel$$ExternalSyntheticBackport0; -> 14849102197
78,79c78,79
< Landroidx/compose/ui/graphics/colorspace/ColorSpaces$$ExternalSyntheticLambda0; -> 25549622955
< Landroidx/compose/ui/graphics/colorspace/ColorSpaces$$ExternalSyntheticLambda1; -> 3134614992
---
> Landroidx/compose/ui/graphics/colorspace/ColorSpaces$$ExternalSyntheticLambda0; -> 24725033931
> Landroidx/compose/ui/graphics/colorspace/ColorSpaces$$ExternalSyntheticLambda1; -> 23954955228
94,106c94,106
< Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda0; -> 47122138535
< Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda10; -> 19292026373
< Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda11; -> 20125650349
< Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda12; -> 45462420566
< Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda1; -> 34675813201
< Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda2; -> 21499457898
< Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda3; -> 42082955340
< Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda4; -> 62924783402
< Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda5; -> 28649796034
< Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda6; -> 48803259148
< Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda7; -> 58946385808
< Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda8; -> 49147697842
< Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda9; -> 32763723714
---
> Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda0; -> 33676326501
> Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda10; -> 31544494519
> Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda11; -> 66762457719
> Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda12; -> 28381587114
> Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda1; -> 46362517052
> Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda2; -> 32680400023
> Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda3; -> 33718907870
> Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda4; -> 37572941042
> Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda5; -> 51025827436
> Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda6; -> 11884588970
> Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda7; -> 48762032254
> Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda8; -> 31031173456
> Landroidx/compose/ui/graphics/colorspace/Rgb$$ExternalSyntheticLambda9; -> 34442289895
111c111
< Landroidx/compose/ui/graphics/colorspace/TransferParameters$$ExternalSyntheticBackport0; -> 2496629645
---
> Landroidx/compose/ui/graphics/colorspace/TransferParameters$$ExternalSyntheticBackport0; -> 9540772156
113,116c113,116
< Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda0; -> 40598807322
< Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda1; -> 33998558250
< Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda2; -> 45599148700
< Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda3; -> 42377598099
---
> Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda0; -> 48185619588
> Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda1; -> 25155936879
> Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda2; -> 27977579564
> Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda3; -> 48875607879
188,189c188,189
< Landroidx/compose/ui/graphics/layer/LayerManager$$ExternalSyntheticLambda0; -> 22856299547
< Landroidx/compose/ui/graphics/layer/LayerManager$$ExternalSyntheticLambda1; -> 39411524736
---
> Landroidx/compose/ui/graphics/layer/LayerManager$$ExternalSyntheticLambda0; -> 28289561799
> Landroidx/compose/ui/graphics/layer/LayerManager$$ExternalSyntheticLambda1; -> 36685213490
293c293
< Landroidx/compose/ui/graphics/vector/FastFloatParserKt$$ExternalSyntheticBackport0; -> 1340032763
---
> Landroidx/compose/ui/graphics/vector/FastFloatParserKt$$ExternalSyntheticBackport0; -> 13287982770
296c296
< Landroidx/compose/ui/graphics/vector/PathNode$ArcTo$$ExternalSyntheticBackport0; -> 11167643626
---
> Landroidx/compose/ui/graphics/vector/PathNode$ArcTo$$ExternalSyntheticBackport0; -> 10293143671

So for synthetics, we are not calculating stable checksums!

if we can it would be good to cherry-pick this in KFD/8.6.

Yes, I will merge this back to 8.6 and even 8.5 (so that people can use a custom R8 version in 8.5 if this is something that is causing issues for them)

Thank you for your patience while our engineering team worked to resolve this issue. A fix for this issue is now available in:

• Android Studio Koala Feature Drop | 2024.1.2 Patch 1
• Android Gradle Plugin 8.6.1

We encourage you to try the latest update.

If you notice further issues or have questions, please file a new bug report.

Thank you for taking the time to submit feedback — we really appreciate it!