Status Update
Comments
im...@google.com <im...@google.com>
fl...@obfusk.net <fl...@obfusk.net> #2
fwiw: I've seen (variations of) this very small difference in various apps:
│ │ ├── j$/util/Collection$-EL.class
│ │ │ ├── procyon -ec {}
│ │ │ │ @@ -1,14 +1,14 @@
│ │ │ │
│ │ │ │ package j$.util;
│ │ │ │
│ │ │ │ import java.util.Comparator;
│ │ │ │ import java.util.Set;
│ │ │ │ -import java.util.List;
│ │ │ │ import java.util.SortedSet;
│ │ │ │ +import java.util.List;
│ │ │ │ import java.util.LinkedHashSet;
│ │ │ │ import java.util.Iterator;
│ │ │ │ import java.util.Objects;
│ │ │ │ import j$.util.function.Consumer;
│ │ │ │ import java.util.Collection;
│ │ │ │ import java.util.OptionalLong;
│ │ │ │ import java.util.OptionalInt;
│ │ │ │ @@ -142,20 +142,20 @@
│ │ │ │ return ((j$.util.Collection)collection).spliterator();
│ │ │ │ }
│ │ │ │ if (collection instanceof LinkedHashSet) {
│ │ │ │ final LinkedHashSet obj = (LinkedHashSet)collection;
│ │ │ │ Objects.requireNonNull(obj);
│ │ │ │ return (Spliterator)new Spliterators$IteratorSpliterator((Collection)obj, 17);
│ │ │ │ }
│ │ │ │ + if (collection instanceof List) {
│ │ │ │ + return List$-CC.$default$spliterator((List)collection);
│ │ │ │ + }
│ │ │ │ if (collection instanceof SortedSet) {
│ │ │ │ final SortedSet set = (SortedSet)collection;
│ │ │ │ return (Spliterator)new SortedSet$1(set, (Collection)set, 21);
│ │ │ │ }
│ │ │ │ - if (collection instanceof List) {
│ │ │ │ - return List$-CC.$default$spliterator((List)collection);
│ │ │ │ - }
│ │ │ │ if (collection instanceof Set) {
│ │ │ │ return Set$-CC.$default$spliterator((Set)(SortedSet)collection);
│ │ │ │ }
│ │ │ │ return Collection$-CC.$default$spliterator(collection);
│ │ │ │ }
│ │ │ │ }
je...@google.com <je...@google.com>
bi...@google.com <bi...@google.com> #3
Hey Clement, do you have any ideas on producing different dex files from build to build?
cl...@google.com <cl...@google.com> #4
The example shown above matches a non deterministic issue we had in L8 for classes ending with $-EL. This issue should be fixed on tip of tree. If you can reproduce on ToT let us know, I'll work on another fix. If the changes include something else than ordering inside the $-EL methods, let us know too, that would need to be fixed.
fl...@obfusk.net <fl...@obfusk.net> #5
How do I use this "tip of tree"?
fl...@obfusk.net <fl...@obfusk.net> #6
I'd also be interesting in seeing the fix. Could you link the relevant commit?
bi...@google.com <bi...@google.com> #7
I will try it out with tip of the tree.
cl...@google.com <cl...@google.com> #8
Before BumbleBee, you can try the latest version 3.1.16-dev by adding to your gradle script:
repositories {
maven {
url 'https://storage.googleapis.com/r8-releases/raw'
}
}
dependencies {
classpath 'com.android.tools:r8:3.1.16-dev' // Must be before the Gradle Plugin for Android.
classpath 'com.android.tools.build:gradle:X.Y.Z' // Your current AGP version.
}
Since BumbleBee, the syntax is
repositories {
google()
mavenCentral()
maven {url 'https://storage.googleapis.com/r8-releases/raw'}
}
dependencies {
classpath 'com.android.tools:r8:3.1.16-dev'
}
You can also try ToT directly but using the latest dev version that we release once a week is safest (ToT may be quite unstable sometimes)
bi...@google.com <bi...@google.com> #9
Thanks for sharing the instructions on how to testing with dev version of r8, Clement!
To reporter, could you try it out with dev version of r8 and see if you still see the difference? I didn't find where you enable core library desugaring in you sample project(debug_reproducible_builds branch) and you are using Android Gradle Plugin 4.2.2 instead of 7.0.
da...@gultsch.de <da...@gultsch.de> #10
In order to eventually remove the unstable version of r8 again. What software release should we looking out for? build-tools? AGP?
fl...@obfusk.net <fl...@obfusk.net> #11
Looking at the diffoscope output from the APKs linked in the salsa issue by OP, that seems like a different issue than the "classes ending with $-EL"; I mostly see things like this there:
├── smali_classes4/de/rki/coronawarnapp/util/encoding/Base45Decoder.smali
│┄ Ordering differences only
│ @@ -50,17 +50,17 @@
│ "(Ljava/math/BigInteger;I)J",
│ "",
│ "encode",
│ "([B)Ljava/lang/String;",
│ "decode",
│ "(Ljava/lang/String;)[B",
│ "kotlin.jvm.PlatformType",
│ - "int45",
│ - "Ljava/math/BigInteger;",
│ "int256",
│ + "Ljava/math/BigInteger;",
│ + "int45",
│ "alphabet",
│ "Ljava/lang/String;",
│ "<init>",
│ "()V",
│ "Corona-Warn-App_deviceRelease"
│ }
│ k = 0x1
cl...@google.com <cl...@google.com> #12
@bingran unless you've found a reason why the issue would come from gradle, I now suspect that this issue comes from R8 and you can assign it to me. I will investigate tomorrow.
bi...@google.com <bi...@google.com> #13
Clement, I am going to re-assign it to you, if you need any additional info from AGP, I am more than happy to help. As Daniel mentioned(#10), upgrading r8 version fixes the problem, do you know which r8 version contains that fix? If so, we could let users know which AGP version they should upgrade to.
Re #10, you could try AGP 7.1 alpha08 which is latest one.(without using r8 dev version)
cl...@google.com <cl...@google.com> #14
Re #6 the fix for $-EL dispatch seems to be
[Deleted User] <[Deleted User]> #15
We tried with latest r8 version 3.1.16-dev and the problem persists.
I've pushed a branch for this test, it can be found here:
Also attaching 2 apks that were produced in runnign ./gradlew clean assembleDeviceRelease
twice on that branch. It's hard to really get a feeling for the differences here as they are soooo many. diffoscope produces somewhat readable output but takes ages and the output size is huge. maybe it still helps someone figuring out what's going on.
fl...@obfusk.net <fl...@obfusk.net> #16
Re #2: the issue with the ordering differences in "classes ending with $-EL" seems to go away when using classpath 'com.android.tools:r8:2.2.64'
.
Re #11: the issue with the ordering differences in compiled kotlin code seems to go away when setting minSdkVersion
to 24 (instead of 23 in this case); would be nice to know the cause, not just the fix (if that is indeed the fix).
Re #14: I seems to be 2.2.41 instead of 2.2.39.
NB: I use "seems", since it's easy to prove something is not deterministic and hard/impossible to prove something is deterministic; I've run 7 (#2) and 9 (#11) builds respectively, all identical, whereas before there would be a non-identical build within 3 builds (#2) and every build was different (#11).
NB: I've only tested the reproducibility of the builds, not whether the changes break anything (I do get some warnings during build when updating r8).
NB: both apps I tested use com.android.tools.build:gradle:4.1.3
.
cl...@google.com <cl...@google.com> #17
OK, I'm assuming the $-EL issues are gone on recent versions, we actually have an internal test for it, so I would be very surprised if tip of tree can reproduce.
It's unclear to me however if the kotlin code on API < 23 non determinism issue is present on tip of tree, so I will investigate that now from the apks provided in #15.
cl...@google.com <cl...@google.com> #18
Looking at the smali from #15, the only difference is:
1/smali_classes3/de/rki/coronawarnapp/covidcertificate/validation/core/business/wrapper/MappingsKt.smali 2/smali_classes3/de/rki/coronawarnapp/covidcertificate/validation/core/business/wrapper/MappingsKt.smali
The first difference comes from annotation runtime Lkotlin/Metadata;
fl...@obfusk.net <fl...@obfusk.net> #19
Re #15: wow, that diff became a lot smaller though. It went from 204MB to 24K.
cl...@google.com <cl...@google.com> #20
Morten,
smali_classes3/de/rki/coronawarnapp/covidcertificate/validation/core/business/wrapper/MappingsKt.smali from
mk...@google.com <mk...@google.com> #21
This could be because we iterate over the keys of an IdentityHashMap() - I have changed it to LinkedHashMap and will do a cherry-pick.
mk...@google.com <mk...@google.com> #22
The kotlin metadata should be deterministic from version 3.0.69 and forward. You can try out that version by adding the following to your top-level build.gradle file:
buildscript {
repositories {
maven {
url 'https://storage.googleapis.com/r8-releases/raw'
}
}
dependencies {
classpath 'com.android.tools:r8:3.0.69' // Must be before the Gradle Plugin for Android.
classpath 'com.android.tools.build:gradle:X.Y.Z' // Your current AGP version.
}
}
Description
It seems that when enabling the coreLibraryDesugaring option, the resulting apk output varies wildly from build to build. One set of such differing apks with the resulting diffoscope html diff output can be found in this diffoscope issue:https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/272
STEPS TO REPRODUCE:
./gradlew clean assembleRelease
, saving the resulting apk outputsStudio Build: - (not studio related) Version of Gradle Plugin: 7.0.0 Version of Gradle: 7.0.2 Version of Java: openjdk-11 OS: Archlinux