Fixed
Status Update
No update yet.
Comments
to...@googlemail.com
As some people have already starred this issue, here's how to workaround it for now:
Instead of building the app in its usual REPO_DIR, create a second BUILD_DIR where the deterministic file system will be mounted into like so:
disorderfs --sort-dirents=yes --reverse-dirents=no REPO_DIR BUILD_DIR
Then just build your app inside BUILD_DIR to make the build reproducible.
What might help to track down this issue is that on old version of our app didn't have this issue. There it doesn't matter if it is build in a disorderfs or not, it always is reproducible. The latest version that worked was using com.android.tools.build:gradle:3.0.1, buildToolsVersion '26.0.2' and aaptOptions { cruncherEnabled = false }
Instead of building the app in its usual REPO_DIR, create a second BUILD_DIR where the deterministic file system will be mounted into like so:
disorderfs --sort-dirents=yes --reverse-dirents=no REPO_DIR BUILD_DIR
Then just build your app inside BUILD_DIR to make the build reproducible.
What might help to track down this issue is that on old version of our app didn't have this issue. There it doesn't matter if it is build in a disorderfs or not, it always is reproducible. The latest version that worked was using com.android.tools.build:gradle:3.0.1, buildToolsVersion '26.0.2' and aaptOptions { cruncherEnabled = false }
ha...@guardianproject.info
This should be a simple fix, I'm guessing it is just a classic "unsorted direntries" issue.
xa...@google.com
hi...@gmail.com
Facing the same issue. Any update on this?
cm...@google.com
rt...@google.com
ph...@googlemail.com
Is this issue currently being worked on by Google? It seems as if it was ignored, even though it is causing significant trouble to anyone trying to implement reproducible builds (important to ensure security and safety of open source apps)
rt...@google.com
Are these APKs built using aapt or aapt2? This could be an aapt issue or an issue with which order Gradle passes files to aapt.
ha...@guardianproject.info
How can you check? The builds where we are seeing this are pretty plain
Gradle/Android setups. So whatever that would call by default.
Gradle/Android setups. So whatever that would call by default.
go...@schildbach.de
Öffi is not affected, because it is using an older toolchain: `com.android.tools.build:gradle:2.3.3`. In contrast, Briar is using `com.android.tools.build:gradle:3.2.1`, and Android DNS `com.android.tools.build:gradle:3.3.0`.
The original reporter already made it clear that deterministic builds broke somewhere after `com.android.tools.build:gradle:3.0.1`.
The original reporter already made it clear that deterministic builds broke somewhere after `com.android.tools.build:gradle:3.0.1`.
dj...@riseup.net
>Are these APKs built using aapt or aapt2? This could be an aapt issue or an issue with which order Gradle passes files to aapt.
Since we use a version of the android gradle plugin >= 3.0.x aapt2 is used (by default).
Since we use a version of the android gradle plugin >= 3.0.x aapt2 is used (by default).
rt...@google.com
I put together a patch to attempt to fix determinism issues with aapt2 and I will run it on the examples you provided today.
rt...@google.com
While I was not able to reproduce the non-determinism between builds on Windows or Linux, I found that with my patch, Windows and Linux now produce the same exact resources.arsc. This goes along with my suspicion that the order in which files are passed to aapt2 is non-deterministic, not aapt2 itself. I will still merge the patch and we will also address this in Gradle.
to...@googlemail.com
Great to see this getting fixed! :)
Note that the non-determinism is not between different builds on the same machine. It is between different builds on different file-systems. As written above, using a special sorted and therefore deterministic file-system works around the issue. So do reproduce the problem on the same machine, you would need to use virtual machines or build on different partitions with different file systems.
Note that the non-determinism is not between different builds on the same machine. It is between different builds on different file-systems. As written above, using a special sorted and therefore deterministic file-system works around the issue. So do reproduce the problem on the same machine, you would need to use virtual machines or build on different partitions with different file systems.
to...@googlemail.com
Btw. while we (mis-)use disorderfs to work around this bug, you can also use it for its original purpose to provide a shuffled filesystem in a directory to reproduce this issue on one single machine. You might even want to add this to your test suite to catch reproducibility bugs due to differently ordered filesystems early on before they get shipped in production releases.
Once you fixed the issue, please let us know in which version of the gradle plugin or aapt2 the fix will appear. Thanks!
Once you fixed the issue, please let us know in which version of the gradle plugin or aapt2 the fix will appear. Thanks!
rt...@google.com
im...@google.com
I added sorting the input files in AGP in change list with Change-Id: I371304c8a6d244f8bb7833609f464eba000e608f - it should be available in android gradle plugin version 3.5 Canary 7
go...@schildbach.de
@im Thanks a lot for the fix! Any chance this can be backported to the current plugin version 3.3?
im...@google.com
Saurabh, is it too late to CP things into 3.3?
im...@google.com
We cherry picked it into 3.4 and it should be available in the next 3.4 canary. We'll see what we can do about 3.3.
ka...@gmail.com
oui N'ko
ka...@gmail.com
ߞߒߏ߫
Description
The issue is that recent build-tools (26.x, 27.x and even 28.0.0) introduce small differences into the resources.arsc file. All other APK contents can be build reproducibly as one would expect. Various tools for comparing APKs do not show differences at all. Even aapt2 dump or aapt2 diff don't show differences, but the resources.arsc file itself has differences.
The only clue I was able to find is using aapt dump resources file.apk. This gives a diff like this:
14128,14130c14128,14130
< resource 0x7f10001c org.package.name:string/abc_toolbar_collapse_description: t=0x03 d=0x000028e5 (s=0x0008 r=0x00)
< resource 0x7f100143 org.package.name:string/search_menu_title: t=0x03 d=0x000028e2 (s=0x0008 r=0x00)
< resource 0x7f10016a org.package.name:string/status_bar_notification_info_overflow: t=0x03 d=0x000024bc (s=0x0008 r=0x00)
---
> resource 0x7f10001c org.package.name:string/abc_toolbar_collapse_description: t=0x03 d=0x000028e7 (s=0x0008 r=0x00)
> resource 0x7f100143 org.package.name:string/search_menu_title: t=0x03 d=0x000028e4 (s=0x0008 r=0x00)
> resource 0x7f10016a org.package.name:string/status_bar_notification_info_overflow: t=0x03 d=0x000024e5 (s=0x0008 r=0x00)
Note how the resource IDs are all the same and the only difference is with the d value (whatever this is).
Trying many different things, we were finally able to prevent these differences from occurring when using a special file system that supports deterministic ordering (disorderfs). However, this is not a real solution. Ideally, the resources.arsc file should be created in a deterministic fashion. I am hoping for your help to achieve this. Thanks!
Further Information: We are using the following configuration in our build.gradle file:
android {
buildTypes {
release {
shrinkResources true
minifyEnabled true
crunchPngs false
}
}
}
We set the latter to false, because crunching PNGs is not always done deterministically as well.