IssueTracker

Hey Clement, do you have any ideas on producing different dex files from build to build?

The example shown above matches a non deterministic issue we had in L8 for classes ending with $-EL. This issue should be fixed on tip of tree. If you can reproduce on ToT let us know, I'll work on another fix. If the changes include something else than ordering inside the $-EL methods, let us know too, that would need to be fixed.

How do I use this "tip of tree"?

I'd also be interesting in seeing the fix. Could you link the relevant commit?

I will try it out with tip of the tree.

Before BumbleBee, you can try the latest version 3.1.16-dev by adding to your gradle script:

 repositories {
        maven {
            url 'https://storage.googleapis.com/r8-releases/raw'
        }
    }

    dependencies {
        classpath 'com.android.tools:r8:3.1.16-dev'      // Must be before the Gradle Plugin for Android.
        classpath 'com.android.tools.build:gradle:X.Y.Z' // Your current AGP version.
     }

Since BumbleBee, the syntax is

repositories {
        google()
        mavenCentral()
        maven {url 'https://storage.googleapis.com/r8-releases/raw'}
    }

    dependencies {
        classpath 'com.android.tools:r8:3.1.16-dev'
    }

You can also try ToT directly but using the latest dev version that we release once a week is safest (ToT may be quite unstable sometimes)

Thanks for sharing the instructions on how to testing with dev version of r8, Clement!

To reporter, could you try it out with dev version of r8 and see if you still see the difference? I didn't find where you enable core library desugaring in you sample project(debug_reproducible_builds branch) and you are using Android Gradle Plugin 4.2.2 instead of 7.0.

Looking at the diffoscope output from the APKs linked in the salsa issue by OP, that seems like a different issue than the "classes ending with $-EL"; I mostly see things like this there:

├── smali_classes4/de/rki/coronawarnapp/util/encoding/Base45Decoder.smali
│┄ Ordering differences only
│ @@ -50,17 +50,17 @@
│          "(Ljava/math/BigInteger;I)J",
│          "",
│          "encode",
│          "([B)Ljava/lang/String;",
│          "decode",
│          "(Ljava/lang/String;)[B",
│          "kotlin.jvm.PlatformType",
│ -        "int45",
│ -        "Ljava/math/BigInteger;",
│          "int256",
│ +        "Ljava/math/BigInteger;",
│ +        "int45",
│          "alphabet",
│          "Ljava/lang/String;",
│          "<init>",
│          "()V",
│          "Corona-Warn-App_deviceRelease"
│      }
│      k = 0x1

@bingran unless you've found a reason why the issue would come from gradle, I now suspect that this issue comes from R8 and you can assign it to me. I will investigate tomorrow.

Clement, I am going to re-assign it to you, if you need any additional info from AGP, I am more than happy to help. As Daniel mentioned(#10), upgrading r8 version fixes the problem, do you know which r8 version contains that fix? If so, we could let users know which AGP version they should upgrade to.

Re #10, you could try AGP 7.1 alpha08 which is latest one.(without using r8 dev version)

Re #6 the fix for $-EL dispatch seems to be https://r8-review.googlesource.com/q/I191acc3acb420676104a72ee027b22689a528b22 It's in 2.2.39, 2.1.101 and it seems it's in 3.0.3-dev? Maybe 3.0.5-dev? and higher.

We tried with latest r8 version 3.1.16-dev and the problem persists.

I've pushed a branch for this test, it can be found here: https://codeberg.org/corona-contact-tracing-germany/cwa-android/src/branch/new_r8_tests/Corona-Warn-App/build.gradle#L302

Also attaching 2 apks that were produced in runnign ./gradlew clean assembleDeviceRelease twice on that branch. It's hard to really get a feeling for the differences here as they are soooo many. diffoscope produces somewhat readable output but takes ages and the output size is huge. maybe it still helps someone figuring out what's going on.

Re #2: the issue with the ordering differences in "classes ending with $-EL" seems to go away when using classpath 'com.android.tools:r8:2.2.64'.

Re #11: the issue with the ordering differences in compiled kotlin code seems to go away when setting minSdkVersion to 24 (instead of 23 in this case); would be nice to know the cause, not just the fix (if that is indeed the fix).

Re #14: I seems to be 2.2.41 instead of 2.2.39.

NB: I use "seems", since it's easy to prove something is not deterministic and hard/impossible to prove something is deterministic; I've run 7 (#2) and 9 (#11) builds respectively, all identical, whereas before there would be a non-identical build within 3 builds (#2) and every build was different (#11).

NB: I've only tested the reproducibility of the builds, not whether the changes break anything (I do get some warnings during build when updating r8).

NB: both apps I tested use com.android.tools.build:gradle:4.1.3.

OK, I'm assuming the $-EL issues are gone on recent versions, we actually have an internal test for it, so I would be very surprised if tip of tree can reproduce.

It's unclear to me however if the kotlin code on API < 23 non determinism issue is present on tip of tree, so I will investigate that now from the apks provided in #15.