differing DEX bytecode depending on the number of CPU cores breaks Reproducible Builds [366412380]

Fixed

Bug

Status Update

No update yet.

Description

fl...@obfusk.net

created issue #1

Sep 13, 2024 07:13PM

We've rebuilt https://github.com/TheLastProject/ShareToInputStick/releases/tag/v3.6.1 multiple times (each time in a fresh container): with 2, 4, 6, 8, and 16 cores available, respectively.

Depending on the number of cores, we get a different classes.dex file and thus a different APK, which breaks Reproducible Builds:

With 2 and 4 cores we always get an unsigned APK with SHA-256 14763d682c9286ef65323d9eb947240ac62941938ac6f9048bdf880019fb3ee9.
with 6, 8, and 16 cores we get an unsigned APK with SHA-256 35324ba4c4927607415b5d5a9c56bcf0d557aad285a3d78b72365b143b1cb60e instead.

The only difference is the classes.dex file (and of course the baseline.prof as it contains a checksum of the .dex); the dexdump diff is very small:

@@ -2,9 +2,9 @@
 Opened 'classes.dex', DEX version '035'
 DEX file header:
 magic               : 'dex\n035\0'
-checksum            : 5c51b34f
-signature           : f062...4a1d
-file_size           : 9889456
+checksum            : 7df369bb
+signature           : 155e...3e08
+file_size           : 9889452
 header_size         : 112
 link_size           : 0
 link_off            : 0 (0x000000)
@@ -20,7 +20,7 @@
 method_ids_off      : 808120 (0x0c54b8)
 class_defs_size     : 7247
 class_defs_off      : 1331088 (0x144f90)
-data_size           : 8326464
+data_size           : 8326460
 data_off            : 1562992 (0x17d970)
 
 access_flags        : 1537 (0x0601)
@@ -1309293,23 +1309293,22 @@
       registers     : 7
       ins           : 4
       outs          : 4
-      insns size    : 57 16-bit code units
+      insns size    : 55 16-bit code units
 | com.inputstick.api.BTConnectionManager.connect:(ZIZ)V
 |: invoke-virtual {v3}, Lcom/inputstick/api/BTConnectionManager;.resetErrorCode:()V
 |: iget-object v0, v3, Lcom/inputstick/api/BTConnectionManager;.mBTService:Lcom/inputstick/api/bluetooth/BTService;
-|: if-nez v0, 0022 // +001d
+|: if-nez v0, 0020 // +001b
 |: new-instance v0, Lcom/inputstick/api/bluetooth/BTService;
 |: iget-object v1, v3, Lcom/inputstick/api/BTConnectionManager;.mApp:Landroid/app/Application;
 |: iget-object v2, v3, Lcom/inputstick/api/BTConnectionManager;.mBTHandler:Lcom/inputstick/api/BTConnectionManager$BTHandler;
 |: invoke-direct {v0, v1, v2}, Lcom/inputstick/api/bluetooth/BTService;.<init>:(Landroid/app/Application;Landroid/os/Handler;)V
 |: iput-object v0, v3, Lcom/inputstick/api/BTConnectionManager;.mBTService:Lcom/inputstick/api/bluetooth/BTService;
-|: new-instance v0, Lcom/inputstick/api/PacketManager;
-|: iget-object v1, v3, Lcom/inputstick/api/BTConnectionManager;.mBTService:Lcom/inputstick/api/bluetooth/BTService;
+|: new-instance v1, Lcom/inputstick/api/PacketManager;
 |: iget-object v2, v3, Lcom/inputstick/api/BTConnectionManager;.mKey:[B
-|: invoke-direct {v0, v1, v2}, Lcom/inputstick/api/PacketManager;.<init>:(Lcom/inputstick/api/bluetooth/BTService;[B)V
-|: iput-object v0, v3, Lcom/inputstick/api/BTConnectionManager;.mPacketManager:Lcom/inputstick/api/PacketManager;
-|: iget-object v1, v3, Lcom/inputstick/api/BTConnectionManager;.mInitManager:Lcom/inputstick/api/init/InitManager;
-|: invoke-virtual {v1, v3, v0}, Lcom/inputstick/api/init/InitManager;.init:(Lcom/inputstick/api/init/InitManagerListener;Lcom/inputstick/api/PacketManager;)V
+|: invoke-direct {v1, v0, v2}, Lcom/inputstick/api/PacketManager;.<init>:(Lcom/inputstick/api/bluetooth/BTService;[B)V
+|: iput-object v1, v3, Lcom/inputstick/api/BTConnectionManager;.mPacketManager:Lcom/inputstick/api/PacketManager;
+|: iget-object v0, v3, Lcom/inputstick/api/BTConnectionManager;.mInitManager:Lcom/inputstick/api/init/InitManager;
+|: invoke-virtual {v0, v3, v1}, Lcom/inputstick/api/init/InitManager;.init:(Lcom/inputstick/api/init/InitManagerListener;Lcom/inputstick/api/PacketManager;)V
 |: iget-object v0, v3, Lcom/inputstick/api/BTConnectionManager;.mBTService:Lcom/inputstick/api/bluetooth/BTService;
 |: invoke-virtual {v0, v5}, Lcom/inputstick/api/bluetooth/BTService;.setConnectTimeout:(I)V
 |: iget-object v5, v3, Lcom/inputstick/api/BTConnectionManager;.mBTService:Lcom/inputstick/api/bluetooth/BTService;
@@ -1309323,10 +1309322,10 @@
       catches       : (none)
       positions     : 
       locals        : 
-        0x0000 - 0x0039 reg=3 this Lcom/inputstick/api/BTConnectionManager; 
-        0x0000 - 0x0039 reg=4 (null) Z 
-        0x0000 - 0x0039 reg=5 (null) I 
-        0x0000 - 0x0039 reg=6 (null) Z 
+        0x0000 - 0x0037 reg=3 this Lcom/inputstick/api/BTConnectionManager; 
+        0x0000 - 0x0037 reg=4 (null) Z 
+        0x0000 - 0x0037 reg=5 (null) I 
+        0x0000 - 0x0037 reg=6 (null) Z 
 
     #4              : (in Lcom/inputstick/api/BTConnectionManager;)
       name          : 'disconnect'

Version of Gradle Plugin: 8.6.0
Version of Gradle: 8.7
Version of Java: OpenJDK 17
OS: Debian bookworm

Comments

sg...@google.com <sg...@google.com> Sep 17, 2024 09:17AM

Assigned to ch...@google.com.

ch...@google.com <ch...@google.com> #2Sep 19, 2024 06:49AM

I remember this issue occurs on other similar cases, but I don't remember which exactly.
Also when it's enabled, many times the focus gets crazy, as I've written here:

https://issuetracker.google.com/issues/79671576

ch...@google.com <ch...@google.com> #3Sep 19, 2024 06:51AM

It seems there a lag, auto-scroll doesn't work if we move cursor rapidly.

fl...@obfusk.net <fl...@obfusk.net> #4Sep 19, 2024 03:35PM

ch...@google.com <ch...@google.com> Sep 25, 2024 04:14PM

Reassigned to ri...@google.com.

ri...@google.com <ri...@google.com> #5Sep 25, 2024 04:31PM

@4 Right, but note that it's not the same issue. Here it doesn't focus at all , and there it does too much (going to a lot of files).
It could all be related though. Here's another one that for some reason I decided to report on the youtrack website:

https://youtrack.jetbrains.com/issue/IDEA-208579

fl...@obfusk.net <fl...@obfusk.net> #6Oct 25, 2024 07:42PM

Thanks for the report! I've attempted to reproduce this with no avail. If it's still reproducible for you, please re-open and attach reproduction steps.

INFO-4-cores

41 KB

Download

INFO-8-cores

44 KB

Download

ri...@google.com <ri...@google.com> #7Oct 28, 2024 10:09AM

@6 How can I reopen? You are the one that can close and open...

Issue is still the same, and with exact same steps as I've shown on the video. Watch a new video. It's so easy to reproduce that it's in less than 5 seconds...

Reproduced on :

Android Studio Ladybug | 2024.2.1 Canary 8 Build #AI-242.20224.300.2421.12279409, built on August 27, 2024 Runtime version: 21.0.3+-12099254-b509.4 amd64 VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o. Toolkit: sun.awt.windows.WToolkit Windows 11.0 GC: G1 Young Generation, G1 Concurrent GC, G1 Old Generation Memory: 5120M Cores: 16 Registry: ide.instant.shutdown=false ide.experimental.ui=true ide.images.show.chessboard=true Non-Bundled Plugins: com.intellij.marketplace (242.20224.428) Show As ... (1.0.3) cn.jxzhang.plugin.json-formatter (1.4) String Manipulation (9.14.1) GenerateSerialVersionUID (3.0.3) idea.plugin.protoeditor (242.20224.155) com.drvector (1.2.1) com.steve.plugins.autoscroll (1.1) com.dethlex.numberconverter (2.1.1) izhangzhihao.rainbow.brackets (2024.2.6-241) com.ppismerov.ksvu (0.0.1) net.aquadc.mike.plugin (0.32) com.developerphil.adbidea (1.6.18) com.mistamek.drawablepreview.drawable-preview (1.1.6) GenerateSerializationHelpers (1.0.6) cn.yiiguxing.plugin.translate (3.6.4)

fl...@obfusk.net <fl...@obfusk.net> #8Oct 28, 2024 10:43PM

This looks to be the affected code from the dexdump: https://github.com/inputstick/InputStickAPI-Android/blob/6f2056be0f091b900e621c0222452a908f0fc9e7/InputStickAPI/src/com/inputstick/api/BTConnectionManager.java#L110

I've uploaded the 2 ZIP files (one from each build with 4/8 cores respectively) with a program.jar containing a .dex referencing that class.

I wrote this small python script to identify the correct files (maybe it'll be useful to someone):

import glob
import os
import tempfile
import zipfile

needle = b"BTConnectionManager"
for file in glob.glob("*.zip"):
    with tempfile.TemporaryDirectory() as tmpdir:
        jar = os.path.join(tmpdir, "program.jar")
        with zipfile.ZipFile(file) as zf:
            zf.extract("program.jar", tmpdir)
        with zipfile.ZipFile(jar) as zf:
            for info in zf.infolist():
                if not info.filename.endswith(".dex"):
                    continue
                data = zf.read(info.filename)
                if needle in data:
                    print(f"file={file} dex={info.filename}")

4core.zip

3.7 MB

Download

8core.zip

3.8 MB

Download

ri...@google.com <ri...@google.com> #9Oct 29, 2024 08:37AM

If we diff the inputs, then there is a large difference, likely because of differences in the order in the zip file that we get passed in

$ unzip -d /tmp/8core ~/Downloads/8core.zip 
$ unzip -d /tmp/4core ~/Downloads/4core.zip 
$ diff -rq  /tmp/{4,8}core
Files /tmp/4core/program.jar and /tmp/8core/program.jar differ
$ disasm.py --no-build --all /tmp/8core/program.jar > /tmp/8core/dis
$ disasm.py --no-build --all /tmp/4core/program.jar > /tmp/4core/dis
$ diff /tmp/{4,8}core/dis | wc -l
15575

I do see the diff that you saw:

1c1
< Running: /usr/local/google/home/ricow/src/r8/third_party/openjdk/jdk-11/linux/bin/java -jar /usr/local/google/home/ricow/src/r8/build/libs/r8.jar disasm --all /tmp/4core/program.jar
---
> Running: /usr/local/google/home/ricow/src/r8/third_party/openjdk/jdk-11/linux/bin/java -jar /usr/local/google/home/ricow/src/r8/build/libs/r8.jar disasm --all /tmp/8core/program.jar
512c512
<     2:   0x05: IfNez               v0, 0x20 (+27)
---
>     2:   0x05: IfNez               v0, 0x22 (+29)
520,540c520,541
<     8:   0x12: NewInstance         v1, Lcom/inputstick/api/PacketManager;
<     9:   0x14: IgetObject          v2, v3, Lcom/inputstick/api/BTConnectionManager;mKey:[B
<    10:   0x16: InvokeDirect        { v1 v0 v2 } Lcom/inputstick/api/PacketManager;<init>(Lcom/inputstick/api/bluetooth/BTService;[B)V
<    11:   0x19: IputObject          v1, v3, Lcom/inputstick/api/BTConnectionManager;mPacketManager:Lcom/inputstick/api/PacketManager;
<          0x1b, #115, locals: [3 -> this]
<    12:   0x1b: IgetObject          v0, v3, Lcom/inputstick/api/BTConnectionManager;mInitManager:Lcom/inputstick/api/init/InitManager;
<    13:   0x1d: InvokeVirtual       { v0 v3 v1 } Lcom/inputstick/api/init/InitManager;init(Lcom/inputstick/api/init/InitManagerListener;Lcom/inputstick/api/PacketManager;)V
<          0x20, #117, locals: [3 -> this]
<    14:   0x20: IgetObject          v0, v3, Lcom/inputstick/api/BTConnectionManager;mBTService:Lcom/inputstick/api/bluetooth/BTService;
<    15:   0x22: InvokeVirtual       { v0 v5 } Lcom/inputstick/api/bluetooth/BTService;setConnectTimeout(I)V
<          0x25, #118, locals: [3 -> this]
<    16:   0x25: IgetObject          v5, v3, Lcom/inputstick/api/BTConnectionManager;mBTService:Lcom/inputstick/api/bluetooth/BTService;
<    17:   0x27: InvokeVirtual       { v5 v4 } Lcom/inputstick/api/bluetooth/BTService;enableReflection(Z)V
<          0x2a, #119, locals: [3 -> this]
<    18:   0x2a: IgetObject          v4, v3, Lcom/inputstick/api/BTConnectionManager;mBTService:Lcom/inputstick/api/bluetooth/BTService;
<    19:   0x2c: IgetObject          v5, v3, Lcom/inputstick/api/BTConnectionManager;mMac:Ljava/lang/String;
<    20:   0x2e: IgetBoolean         v0, v3, Lcom/inputstick/api/BTConnectionManager;mIsBT40:Z
<    21:   0x30: InvokeVirtual       { v4 v5 v6 v0 } Lcom/inputstick/api/bluetooth/BTService;connect(Ljava/lang/String;ZZ)V
<          0x33, #120, locals: [3 -> this]
<    22:   0x33: InvokeDirect        { v3 } Lcom/inputstick/api/BTConnectionManager;onConnecting()V
<    23:   0x36: ReturnVoid          
---
>     8:   0x12: NewInstance         v0, Lcom/inputstick/api/PacketManager;
>     9:   0x14: IgetObject          v1, v3, Lcom/inputstick/api/BTConnectionManager;mBTService:Lcom/inputstick/api/bluetooth/BTService;
>    10:   0x16: IgetObject          v2, v3, Lcom/inputstick/api/BTConnectionManager;mKey:[B
>    11:   0x18: InvokeDirect        { v0 v1 v2 } Lcom/inputstick/api/PacketManager;<init>(Lcom/inputstick/api/bluetooth/BTService;[B)V
>    12:   0x1b: IputObject          v0, v3, Lcom/inputstick/api/BTConnectionManager;mPacketManager:Lcom/inputstick/api/PacketManager;
>          0x1d, #115, locals: [3 -> this]
>    13:   0x1d: IgetObject          v1, v3, Lcom/inputstick/api/BTConnectionManager;mInitManager:Lcom/inputstick/api/init/InitManager;
>    14:   0x1f: InvokeVirtual       { v1 v3 v0 } Lcom/inputstick/api/init/InitManager;init(Lcom/inputstick/api/init/InitManagerListener;Lcom/inputstick/api/PacketManager;)V
>          0x22, #117, locals: [3 -> this]
>    15:   0x22: IgetObject          v0, v3, Lcom/inputstick/api/BTConnectionManager;mBTService:Lcom/inputstick/api/bluetooth/BTService;
>    16:   0x24: InvokeVirtual       { v0 v5 } Lcom/inputstick/api/bluetooth/BTService;setConnectTimeout(I)V
>          0x27, #118, locals: [3 -> this]
>    17:   0x27: IgetObject          v5, v3, Lcom/inputstick/api/BTConnectionManager;mBTService:Lcom/inputstick/api/bluetooth/BTService;
>    18:   0x29: InvokeVirtual       { v5 v4 } Lcom/inputstick/api/bluetooth/BTService;enableReflection(Z)V
>          0x2c, #119, locals: [3 -> this]
>    19:   0x2c: IgetObject          v4, v3, Lcom/inputstick/api/BTConnectionManager;mBTService:Lcom/inputstick/api/bluetooth/BTService;
>    20:   0x2e: IgetObject          v5, v3, Lcom/inputstick/api/BTConnectionManager;mMac:Ljava/lang/String;
>    21:   0x30: IgetBoolean         v0, v3, Lcom/inputstick/api/BTConnectionManager;mIsBT40:Z
>    22:   0x32: InvokeVirtual       { v4 v5 v6 v0 } Lcom/inputstick/api/bluetooth/BTService;connect(Ljava/lang/String;ZZ)V
>          0x35, #120, locals: [3 -> this]
>    23:   0x35: InvokeDirect        { v3 } Lcom/inputstick/api/BTConnectionManager;onConnecting()V
>    24:   0x38: ReturnVoid          
2892899,2896595d2892899

Now, this is actually a merge step, we produced the code way before this (D8 will merge those files in this step) If I run D8 on it, I get exactly your diff:

$ tools/compiledump.py --temp /tmp/compare/4core/ -d ~/Downloads/4core.zip
$ tools/compiledump.py --temp /tmp/compare/8core/ -d ~/Downloads/8core.zip
$ disasm.py --no-build --all /tmp/compare/4core/out.jar > /tmp/compare/4core/dis_out
$ disasm.py --no-build --all /tmp/compare/8core/out.jar > /tmp/compare/8core/dis_out
$ diff /tmp/compare/{4,8}core/dis_out
1c1
< Running: /usr/local/google/home/ricow/src/r8/third_party/openjdk/jdk-11/linux/bin/java -jar /usr/local/google/home/ricow/src/r8/build/libs/r8.jar disasm --all /tmp/compare/4core/out.jar
---
> Running: /usr/local/google/home/ricow/src/r8/third_party/openjdk/jdk-11/linux/bin/java -jar /usr/local/google/home/ricow/src/r8/build/libs/r8.jar disasm --all /tmp/compare/8core/out.jar
1366153c1366153
<     2:   0x05: IfNez               v0, 0x20 (+27)
---
>     2:   0x05: IfNez               v0, 0x22 (+29)
1366161,1366181c1366161,1366182
<     8:   0x12: NewInstance         v1, Lcom/inputstick/api/PacketManager;
<     9:   0x14: IgetObject          v2, v3, Lcom/inputstick/api/BTConnectionManager;mKey:[B
<    10:   0x16: InvokeDirect        { v1 v0 v2 } Lcom/inputstick/api/PacketManager;<init>(Lcom/inputstick/api/bluetooth/BTService;[B)V
<    11:   0x19: IputObject          v1, v3, Lcom/inputstick/api/BTConnectionManager;mPacketManager:Lcom/inputstick/api/PacketManager;
<          0x1b, #115, locals: [3 -> this]
<    12:   0x1b: IgetObject          v0, v3, Lcom/inputstick/api/BTConnectionManager;mInitManager:Lcom/inputstick/api/init/InitManager;
<    13:   0x1d: InvokeVirtual       { v0 v3 v1 } Lcom/inputstick/api/init/InitManager;init(Lcom/inputstick/api/init/InitManagerListener;Lcom/inputstick/api/PacketManager;)V
<          0x20, #117, locals: [3 -> this]
<    14:   0x20: IgetObject          v0, v3, Lcom/inputstick/api/BTConnectionManager;mBTService:Lcom/inputstick/api/bluetooth/BTService;
<    15:   0x22: InvokeVirtual       { v0 v5 } Lcom/inputstick/api/bluetooth/BTService;setConnectTimeout(I)V
<          0x25, #118, locals: [3 -> this]
<    16:   0x25: IgetObject          v5, v3, Lcom/inputstick/api/BTConnectionManager;mBTService:Lcom/inputstick/api/bluetooth/BTService;
<    17:   0x27: InvokeVirtual       { v5 v4 } Lcom/inputstick/api/bluetooth/BTService;enableReflection(Z)V
<          0x2a, #119, locals: [3 -> this]
<    18:   0x2a: IgetObject          v4, v3, Lcom/inputstick/api/BTConnectionManager;mBTService:Lcom/inputstick/api/bluetooth/BTService;
<    19:   0x2c: IgetObject          v5, v3, Lcom/inputstick/api/BTConnectionManager;mMac:Ljava/lang/String;
<    20:   0x2e: IgetBoolean         v0, v3, Lcom/inputstick/api/BTConnectionManager;mIsBT40:Z
<    21:   0x30: InvokeVirtual       { v4 v5 v6 v0 } Lcom/inputstick/api/bluetooth/BTService;connect(Ljava/lang/String;ZZ)V
<          0x33, #120, locals: [3 -> this]
<    22:   0x33: InvokeDirect        { v3 } Lcom/inputstick/api/BTConnectionManager;onConnecting()V
<    23:   0x36: ReturnVoid          
---
>     8:   0x12: NewInstance         v0, Lcom/inputstick/api/PacketManager;
>     9:   0x14: IgetObject          v1, v3, Lcom/inputstick/api/BTConnectionManager;mBTService:Lcom/inputstick/api/bluetooth/BTService;
>    10:   0x16: IgetObject          v2, v3, Lcom/inputstick/api/BTConnectionManager;mKey:[B
>    11:   0x18: InvokeDirect        { v0 v1 v2 } Lcom/inputstick/api/PacketManager;<init>(Lcom/inputstick/api/bluetooth/BTService;[B)V
>    12:   0x1b: IputObject          v0, v3, Lcom/inputstick/api/BTConnectionManager;mPacketManager:Lcom/inputstick/api/PacketManager;
>          0x1d, #115, locals: [3 -> this]
>    13:   0x1d: IgetObject          v1, v3, Lcom/inputstick/api/BTConnectionManager;mInitManager:Lcom/inputstick/api/init/InitManager;
>    14:   0x1f: InvokeVirtual       { v1 v3 v0 } Lcom/inputstick/api/init/InitManager;init(Lcom/inputstick/api/init/InitManagerListener;Lcom/inputstick/api/PacketManager;)V
>          0x22, #117, locals: [3 -> this]
>    15:   0x22: IgetObject          v0, v3, Lcom/inputstick/api/BTConnectionManager;mBTService:Lcom/inputstick/api/bluetooth/BTService;
>    16:   0x24: InvokeVirtual       { v0 v5 } Lcom/inputstick/api/bluetooth/BTService;setConnectTimeout(I)V
>          0x27, #118, locals: [3 -> this]
>    17:   0x27: IgetObject          v5, v3, Lcom/inputstick/api/BTConnectionManager;mBTService:Lcom/inputstick/api/bluetooth/BTService;
>    18:   0x29: InvokeVirtual       { v5 v4 } Lcom/inputstick/api/bluetooth/BTService;enableReflection(Z)V
>          0x2c, #119, locals: [3 -> this]
>    19:   0x2c: IgetObject          v4, v3, Lcom/inputstick/api/BTConnectionManager;mBTService:Lcom/inputstick/api/bluetooth/BTService;
>    20:   0x2e: IgetObject          v5, v3, Lcom/inputstick/api/BTConnectionManager;mMac:Ljava/lang/String;
>    21:   0x30: IgetBoolean         v0, v3, Lcom/inputstick/api/BTConnectionManager;mIsBT40:Z
>    22:   0x32: InvokeVirtual       { v4 v5 v6 v0 } Lcom/inputstick/api/bluetooth/BTService;connect(Ljava/lang/String;ZZ)V
>          0x35, #120, locals: [3 -> this]
>    23:   0x35: InvokeDirect        { v3 } Lcom/inputstick/api/BTConnectionManager;onConnecting()V
>    24:   0x38: ReturnVoid

To figure out where the difference in compilation is coming from we need the dump where we compiled to dex in the first place, i.e., you need to find the zip with program.jar that contains com/inputstick/api/BTConnectionManager.class

fl...@obfusk.net <fl...@obfusk.net> #10Oct 29, 2024 04:40PM

When you do your dexdump you should take a look at what method it actually is that is changed, then you unzip all of your dumps, and zipinfo + grep for the class that contains the method in the progam.jar file in those unziped dumps The dump with this class is the dump that I need.

I am completely unfamiliar with the dump format. So I looked at a few of the ZIP files and found .dex files in the program.jar. So then I looked for a ZIP file containing a program.jar with a .dex file containing the BTConnectionManager class, which is what I assumed you meant based on those instructions.

you need to find the zip with program.jar that contains com/inputstick/api/BTConnectionManager.class

This makes it clear I misunderstood and should have been looking for a .class file instead of a .dex file. I've attached the ZIP files containing a file named com/inputstick/api/BTConnectionManager.class now.

Updated script: https://gist.github.com/obfusk/83ca1ada701a5d58458fff65c2ae74dd

4core-class.zip

15 MB

Download

8core-class.zip

15 MB

Download

ri...@google.com <ri...@google.com> #11Oct 30, 2024 03:57PM

The problem here is that we run side effect evaluation of class initializers when we do redundant field load elimination (we should only do that in R8)

This means, that dependent on if we have classes on class path or on program path we could choose to remove the field.

Apparently, AGP will do a different sharding of the inputs to D8 based on the number of cores, this is why you saw it in this case ( +@cmw to keep me honest here)

The fact that we eliminated the field load was a bug, I have a cl out to fix this (while still allowing this for clinit on the class itself and its immediate super type)

ap...@google.com <ap...@google.com> #12Oct 30, 2024 04:00PM

Project: r8
Branch: main
Author: Rico Wind <ricow@google.com>
Link: https://r8-review.googlesource.com/97562

Don't make assumptions about sideeffects of class initializer in D8

Expand for full commit details

Don't make assumptions about sideeffects of class initializer in D8 
 
Bug: b/366412380 
Change-Id: I8e2c0e31a1734da5574ae0bc98d863a9bbf6e58b

Files:

M src/main/java/com/android/tools/r8/graph/DexType.java
A src/test/java/com/android/tools/r8/regress/Regress366412380.java

Hash: bfb75f6984a6ba85eb41547c8ff2daa98aa03924
Date: Wed Oct 30 14:53:55 2024

fl...@obfusk.net <fl...@obfusk.net> #13Oct 31, 2024 02:51AM

Makes sense now, yeah. Thanks. Though I'm slightly confused why the comment says "return true" when the code is return false?

    // To ensure that we assume that there are no class initialization side effects directly on
    // the super class in D8 we always return true if this is a new instance of a direct super type.
    if (context.getContextClass().getSuperType().isIdenticalTo(this)
        || context.getContextType().isIdenticalTo(this)) {
      return false;
    }

ap...@google.com <ap...@google.com> #14Oct 31, 2024 07:11AM

Project: r8
Branch: 8.6
Author: Rico Wind <ricow@google.com>
Link: https://r8-review.googlesource.com/97620

Don't make assumptions about sideeffects of class initializer in D8

Expand for full commit details

Don't make assumptions about sideeffects of class initializer in D8 
 
Bug: b/366412380 
Change-Id: I8e2c0e31a1734da5574ae0bc98d863a9bbf6e58b

Files:

M src/main/java/com/android/tools/r8/graph/DexType.java
A src/test/java/com/android/tools/r8/regress/Regress366412380.java

Hash: 8c064e58b366697410a8c4955e78aab65b2bc803
Date: Thu Oct 31 08:04:21 2024

ap...@google.com <ap...@google.com> #15Oct 31, 2024 07:11AM

Project: r8
Branch: 8.7
Author: Rico Wind <ricow@google.com>
Link: https://r8-review.googlesource.com/97600

Don't make assumptions about sideeffects of class initializer in D8

Expand for full commit details

Don't make assumptions about sideeffects of class initializer in D8 
 
Bug: b/366412380 
Change-Id: I8e2c0e31a1734da5574ae0bc98d863a9bbf6e58b

Files:

M src/main/java/com/android/tools/r8/graph/DexType.java
A src/test/java/com/android/tools/r8/regress/Regress366412380.java

Hash: 057c83ea8dc3428b53f604d5bf9f710fae8435ff
Date: Thu Oct 31 08:02:07 2024

ap...@google.com <ap...@google.com> #16Oct 31, 2024 07:13AM

Project: r8
Branch: 8.6
Author: Rico Wind <ricow@google.com>
Link: https://r8-review.googlesource.com/97620

Don't make assumptions about sideeffects of class initializer in D8

Expand for full commit details

Don't make assumptions about sideeffects of class initializer in D8 
 
Bug: b/366412380 
Change-Id: I8e2c0e31a1734da5574ae0bc98d863a9bbf6e58b

Files:

M src/main/java/com/android/tools/r8/graph/DexType.java
A src/test/java/com/android/tools/r8/regress/Regress366412380.java

Hash: 8c064e58b366697410a8c4955e78aab65b2bc803
Date: Thu Oct 31 08:04:21 2024

ap...@google.com <ap...@google.com> #17Oct 31, 2024 07:15AM

Project: r8
Branch: 8.6
Author: Rico Wind <ricow@google.com>
Link: https://r8-review.googlesource.com/97602

Version 8.6.33

Expand for full commit details

Version 8.6.33 
 
Bug: b/366412380 
Change-Id: Ibc8517ffd3c5ab3ae37a36bb65c74cfe2c639e83

Files:

M src/main/java/com/android/tools/r8/Version.java

Hash: 3a252e519e08fab818dab46496b811078aacdd42
Date: Thu Oct 31 08:04:33 2024

ap...@google.com <ap...@google.com> #18Oct 31, 2024 07:15AM

Project: r8
Branch: 8.7
Author: Rico Wind <ricow@google.com>
Link: https://r8-review.googlesource.com/97601

Version 8.7.20

Expand for full commit details

Version 8.7.20 
 
Bug: b/366412380 
Change-Id: I4a408e7bb67517d6ee0c5d0f1999831dab0081f3

Files:

M src/main/java/com/android/tools/r8/Version.java

Hash: a499d4d8e9c559b178fb3099b21931e35cfbee49
Date: Thu Oct 31 08:02:20 2024

ri...@google.com <ri...@google.com> #19Oct 31, 2024 07:22AM

Marked as fixed.

re #13: good point on the comment :-) - fix landed

This has landed in the 8.6 branch, you can try this out to validate that it works correctly for you by doing something along the lines of

pluginManagement {
    buildscript {
        repositories {
            mavenCentral()
            maven {
                url = uri("https://storage.googleapis.com/r8-releases/raw")
            }
        }
        dependencies {
            classpath("com.android.tools.r8:8.6.33")
        }
    }
}

Remember to remove this again when you upgrade to a more recent AGP

fl...@obfusk.net <fl...@obfusk.net> #20Oct 31, 2024 07:53PM

Took me a while to spot your snipped said com.android.tools.r8:8.6.33 instead of com.android.tools:r8:8.6.33 and the error I got was really unhelpful. But I can confirm 4 and 8 cores now produce the same DEX file (and the only difference to the original DEX file seems to an absence of several redundant field load eliminations, as expected). So looks fixed indeed. Thanks!

fl...@obfusk.net <fl...@obfusk.net> #21Oct 31, 2024 07:54PM

Which AGP release can we expect the fix to be in?

ri...@google.com <ri...@google.com> #22Nov 1, 2024 10:20AM

8.8 should get this (maybe not the first beta, but after that) - I don't think we will see more releases of older versions, so you would need to stick with the hack I gave in #19 until then if you want this in (you can use the same to get a more recent R8 for 8.7, then pick version 8.7.20 which has the fix cherry picked)

ap...@google.com <ap...@google.com> #23Nov 3, 2024 09:08AM

Project: r8
Branch: main
Author: Rico Wind <ricow@google.com>
Link: https://r8-review.googlesource.com/97621

Fix comment in early return from classInitializationMayHaveSideEffectsInContext

Expand for full commit details

Fix comment in early return from classInitializationMayHaveSideEffectsInContext 
 
Bug: b/366412380 
Change-Id: I0dd245b6a1204fe5d8fa4f06ff78f2ca7380e4c9

Files:

M src/main/java/com/android/tools/r8/graph/DexType.java

Hash: 3e40335d0a887ef7b0daa6d0b780da67011b1661
Date: Thu Oct 31 08:11:28 2024

Issue 366412380

Description

Issue summary

Comments

sg...@google.com <sg...@google.com> Sep 17, 2024 09:17AM

ch...@google.com <ch...@google.com> #2Sep 19, 2024 06:49AM

ch...@google.com <ch...@google.com> #3Sep 19, 2024 06:51AM

fl...@obfusk.net <fl...@obfusk.net> #4Sep 19, 2024 03:35PM

ch...@google.com <ch...@google.com> Sep 25, 2024 04:14PM

ri...@google.com <ri...@google.com> #5Sep 25, 2024 04:31PM

fl...@obfusk.net <fl...@obfusk.net> #6Oct 25, 2024 07:42PM

ri...@google.com <ri...@google.com> #7Oct 28, 2024 10:09AM

fl...@obfusk.net <fl...@obfusk.net> #8Oct 28, 2024 10:43PM

ri...@google.com <ri...@google.com> #9Oct 29, 2024 08:37AM

fl...@obfusk.net <fl...@obfusk.net> #10Oct 29, 2024 04:40PM

ri...@google.com <ri...@google.com> #11Oct 30, 2024 03:57PM

ap...@google.com <ap...@google.com> #12Oct 30, 2024 04:00PM

fl...@obfusk.net <fl...@obfusk.net> #13Oct 31, 2024 02:51AM

ap...@google.com <ap...@google.com> #14Oct 31, 2024 07:11AM

ap...@google.com <ap...@google.com> #15Oct 31, 2024 07:11AM

ap...@google.com <ap...@google.com> #16Oct 31, 2024 07:13AM

ap...@google.com <ap...@google.com> #17Oct 31, 2024 07:15AM

ap...@google.com <ap...@google.com> #18Oct 31, 2024 07:15AM

ri...@google.com <ri...@google.com> #19Oct 31, 2024 07:22AM

fl...@obfusk.net <fl...@obfusk.net> #20Oct 31, 2024 07:53PM

fl...@obfusk.net <fl...@obfusk.net> #21Oct 31, 2024 07:54PM

ri...@google.com <ri...@google.com> #22Nov 1, 2024 10:20AM

ap...@google.com <ap...@google.com> #23Nov 3, 2024 09:08AM

Add comment

Issue metadata