Dexing task/transform generates non-deterministic classes.dex contents [359616078]

Fixed

Bug

Status Update

No update yet.

Description

iv...@gradle.com

created issue #1

Aug 14, 2024 03:09AM

Gradle 8.9 introduces an internal property that allows disabling artifact transform caching during build execution(-Dorg.gradle.internal.transform-caching-disabled). We measured the impact of disabling these transforms in a performance scenario where the same build was executed multiple times on a GitHub runner. We observed a significant regression(disabling cache transforms and cache transform enabled) with the following behavior in an Android project:

Disabling the cache transforms affects the task input dexDir of the DexMergingTask, resulting in non-deterministic output for each iteration. When comparing two build scans, the input differences in the task are found in the classes.dex files of the input artifacts. After analyzing the dex files from two different builds(attached), we observed the following differences in the strings within the dex file:

Build A
....
"Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda0;":"8825da7cd"
"Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda1;":"9dee08735"
"Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda2;":"a685b3805"
"Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda3;":"a0c078988"
...
Build B
...
"Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda0;":"4957a219a"
"Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda1;":"8d1fa282d"
"Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda2;":"3a518c7af"
"Landroidx/compose/ui/graphics/ColorSpaceVerificationHelper$$ExternalSyntheticLambda3;":"a1783c5de"

The diff report was generated by diffuse the following output:

OLD: classes.dex
NEW: classes2.dex


STRINGS:

   old  │ new  │ diff      
  ──────┼──────┼───────────
   5873 │ 5873 │ 0 (+1 -1)

However, if we exclude DexingWithClasspathTransform instead of disabling the cache for all transforms, the input dexDir of the DexMergingTask remains consistent, and the task hits the cache.

Attached the transform outputs (classes.dex) for androidx.compose.ui:ui-graphics-android:1.7.0-beta01 (instrumented_ui-graphics-release-runtime.jar) of two builds without caching for DexingWithClasspathTransform

Steps to reproduce:

Clone https://github.com/cdsap/CpuExperiment. Branch reproducing_issue_dex
Execute ./gradlew assembleDebug -Dorg.gradle.internal.transform-caching-disabled publishing build scans from two different runners/agents
Compare two build scans

Gradle 8.9

AGP 8.5.2

classes.dex

819 KB

Download

classes2.dex

819 KB

Download

Comments

iv...@gradle.com <iv...@gradle.com> #2Aug 16, 2024 12:00AM

This was broken by 3bca759a5ff08352de831bb1e9b61b1ec2b3362d.
Fix (pending) is I2c2dc7b600603ee430fd0d91b23d52ea8aa29ca9.

xa...@google.com <xa...@google.com> Aug 17, 2024 02:12AM

Assigned to an...@google.com.

hu...@google.com <hu...@google.com> #3Aug 19, 2024 02:39PM

Reassigned to sg...@google.com.

Almost 2 months later and this is still broken

hu...@google.com <hu...@google.com> #4Aug 19, 2024 03:20PM

Since there is no progression, I wanted to share our quick-fix for the issue.

#sdkmanager --package_file=${PATH_WORKSPACE}/packages
while read p; do echo "y" | sdkmanager "${p}"; done <${PATH_WORKSPACE}/packages

Build1_classes.dex

819 KB

Download

Build1_classes_dexdump_with_h_option.txt

2.7 MB

View

Download

Build1_instrumented_ui-graphics-release-runtime.jar

700 KB

Download

Build2_classes.dex

819 KB

Download

Build2_classes_dexdump_with_h_option.txt

2.7 MB

View

Download

Build2_instrumented_ui-graphics-release-runtime.jar

700 KB

Download

ri...@google.com <ri...@google.com> #5Aug 20, 2024 01:32PM

jb...@google.com What is the update on this?

ri...@google.com <ri...@google.com> #6Aug 20, 2024 01:34PM

What is the status of this item?

ri...@google.com <ri...@google.com> #7Aug 21, 2024 07:08AM

Reassigned to ri...@google.com.

This has been fixed on master today (internal ref: ag/2945015) and will be available in the next SDK release.

ri...@google.com <ri...@google.com> #8Aug 21, 2024 10:16AM

Any ETA on next release?

ap...@google.com <ap...@google.com> #9Aug 26, 2024 02:12PM

Still broken and not updated? --package_file argument is not usable in it's current form on 26.1.1 straight from the developer site.

xa...@google.com <xa...@google.com> #10Aug 26, 2024 05:21PM

Comfirmed that this seems to still be broken. Can we have an update please?

```
(15:58:11) C02W513SHTD8:files aso$ /opt/android-sdk-macosx/tools/bin/sdkmanager --version
26.1.1

(15:58:17) C02W513SHTD8:files aso$ /opt/android-sdk-macosx/tools/bin/sdkmanager --install --package_file=package_file
Warning: Unknown argument --package_file=package_file
```

ri...@google.com <ri...@google.com> #11Aug 26, 2024 05:26PM

Hi, is there any update to this issue? Thanks.

ap...@google.com <ap...@google.com> #12Aug 27, 2024 09:11AM

Hi Google. You claim it's been fixed on master, but we haven't had a new release since the broken version 26.1.1. Can you please release the fix?

ap...@google.com <ap...@google.com> #13Aug 27, 2024 09:11AM

Yeah, still not fixed --'

ap...@google.com <ap...@google.com> #14Aug 27, 2024 09:11AM

Can't believe this still isn't fixed 2 years later for a command line utility that sits on the main dev site.

ap...@google.com <ap...@google.com> #15Aug 27, 2024 09:11AM

Any updates on this? The help for this command clearly states this argument is supported.

ap...@google.com <ap...@google.com> #16Aug 27, 2024 09:13AM

Has anyone re-tried it? We switched back to RUN sdkmanager --package_file=$ANDROID_HOME/packages.txt in our Dockerfile back in March of 2021.

sg...@google.com <sg...@google.com> Aug 28, 2024 12:01PM

Marked as fixed.

an...@google.com <an...@google.com> #17Sep 17, 2024 09:18PM

For what it's worth, I did a quick test with the latest CLI: 11076708 (https://developer.android.com/studio#downloads) on OSX and tried it with a simple:

./sdkmanager --sdk_root="../sdk" --package_file=deps.txt

Deps.txt:

platform-tools
extras;google;instantapps
build-tools;35.0.0-rc3

So perhaps this is now resolved? I haven't tried it with more packages

Issue 359616078

Description

Issue summary

Comments

iv...@gradle.com <iv...@gradle.com> #2Aug 16, 2024 12:00AM

xa...@google.com <xa...@google.com> Aug 17, 2024 02:12AM

hu...@google.com <hu...@google.com> #3Aug 19, 2024 02:39PM

hu...@google.com <hu...@google.com> #4Aug 19, 2024 03:20PM

ri...@google.com <ri...@google.com> #5Aug 20, 2024 01:32PM

ri...@google.com <ri...@google.com> #6Aug 20, 2024 01:34PM

ri...@google.com <ri...@google.com> #7Aug 21, 2024 07:08AM

ri...@google.com <ri...@google.com> #8Aug 21, 2024 10:16AM

ap...@google.com <ap...@google.com> #9Aug 26, 2024 02:12PM

xa...@google.com <xa...@google.com> #10Aug 26, 2024 05:21PM

ri...@google.com <ri...@google.com> #11Aug 26, 2024 05:26PM

ap...@google.com <ap...@google.com> #12Aug 27, 2024 09:11AM

ap...@google.com <ap...@google.com> #13Aug 27, 2024 09:11AM

ap...@google.com <ap...@google.com> #14Aug 27, 2024 09:11AM

ap...@google.com <ap...@google.com> #15Aug 27, 2024 09:11AM

ap...@google.com <ap...@google.com> #16Aug 27, 2024 09:13AM

sg...@google.com <sg...@google.com> Aug 28, 2024 12:01PM

an...@google.com <an...@google.com> #17Sep 17, 2024 09:18PM

Add comment

Issue metadata