DexingNoClasspathTransform (minSdk >= 24) with Java 11 target fails due to missing nest members [266599585]

Fixed

Bug

Status Update

No update yet.

Description

hu...@google.com

created issue #1

Jan 24, 2023 08:17PM

(I'm creating this bug based on Internal Issue 234414461.)

Steps to reproduce

In app, set minSdk >= 24 (when minSdk < 24, dexing transforms are not incremental)
In lib, set compileOption.targetCompatibility = "11" (in lower Java versions, nest members do not exist)
In lib, write a Java class with nested classes
Copy this project to another location. Run ./gradlew :app:mergeLibDexDebug --build-cache at that location.
Come back to the current project, run ./gradlew :app:mergeLibDexDebug --build-cache. Many tasks/transforms should have cache hits.
Make a change to the Java class above or its nested classes.
Run ./gradlew :app:mergeLibDexDebug --build-cache again. It will fail with:

* What went wrong:
Execution failed for task ':app:mergeLibDexDebug'.
> Could not resolve all files for configuration ':app:debugRuntimeClasspath'.
   > Failed to transform debug (project :lib) to match attributes {artifactType=android-dex, asm-transformed-variant=NONE, com.android.build.api.attributes.AgpVersionAttr=8.1.0-dev, com.android.build.api.attributes.BuildTypeAttr=debug, com.android.build.gradle.internal.attributes.VariantAttr=debug, dexing-enable-desugaring=true, dexing-enable-jacoco-instrumentation=false, dexing-is-debuggable=true, dexing-min-sdk=24, org.gradle.category=library, org.gradle.jvm.environment=android, org.gradle.libraryelements=classes, org.gradle.usage=java-runtime, org.jetbrains.kotlin.platform.type=androidJvm}.
      > Execution failed for DexingNoClasspathTransform: MyApplication/lib/build/intermediates/runtime_library_classes_dir/debug.
         > Error while dexing.
...
Caused by: com.android.builder.dexing.DexArchiveBuilderException: Error while dexing.
        at com.android.builder.dexing.D8DexArchiveBuilder.getExceptionToRethrow(D8DexArchiveBuilder.java:186)
        at com.android.builder.dexing.D8DexArchiveBuilder.convert(D8DexArchiveBuilder.java:125)
        at com.android.build.gradle.internal.dependency.BaseDexingTransform.process(DexingTransform.kt:316)
        at com.android.build.gradle.internal.dependency.BaseDexingTransform.processIncrementally(DexingTransform.kt:227)
        at com.android.build.gradle.internal.dependency.BaseDexingTransform.doTransform(DexingTransform.kt:137)
        at com.android.build.gradle.internal.dependency.BaseDexingTransform.transform(DexingTransform.kt:107)
...
Caused by: com.android.tools.r8.CompilationFailedException: Compilation failed to complete, origin: MyApplication/lib/build/intermediates/runtime_library_classes_dir/debug/com/example/lib/JavaClass.class
        at Version.fakeStackEntry(Version_8.1.8-dev.java:0)
...
        at com.android.builder.dexing.D8DexArchiveBuilder.convert(D8DexArchiveBuilder.java:123)
        ... 125 more
Caused by: com.android.tools.r8.utils.b: Compilation of classes com.example.lib.JavaClass requires its nest mates com.example.lib.JavaClass$NestedClass (unavailable) to be on program or class path.
...

Root cause

Dexing transforms use a desugaring graph to store dependencies among class files for incremental dexing/desugaring.

The desugaring graph's nodes are currently absolute paths, which means after a remote cache hit, it will not be usable and will result in subsequent failed incremental builds.

We were aware of this issue when designing the graph and used a workaround, but (I think) at some point Gradle changed the behavior and it no longer worked.

Potential fix

Make the desugaring graph relocatable. It is a bit difficult because there isn't one single root path to normalize the absolute file paths (instead, there are multiple root paths).

(If we can't work out a solution soon, the temporary measure is to ensure in the first build after a remote cache hit, affected transforms will be non-incremental.)

Comments

sh...@gmail.com <sh...@gmail.com> #2Mar 10, 2023 04:45PM

Comment has been deleted.

Message last modified on Mar 10, 2023 04:45PM

sh...@pinterest.com <sh...@pinterest.com> #3Mar 10, 2023 04:46PM

(reposting under work account)

We have been hitting this issue constantly. Is it possible to configure those tasks to be non cacheable as a workaround or is that only possible inside of AGP?

hu...@google.com <hu...@google.com> #4Mar 10, 2023 06:40PM

Currently the only way to disable caching for dexing transforms is to patch AGP manually by providing a patched DexTransform.kt file in buildSrc. Please see https://issuetracker.google.com/169266011#comment27, section on "Workarounds" for an example.

sh...@pinterest.com <sh...@pinterest.com> #5Mar 10, 2023 08:39PM

Thanks. Ignoring potential build perf impacts, if we essentially commented out the processIncrementally block in DexingTransform.kt and always ran non-incrementally, would that workaround this issue?

hu...@google.com <hu...@google.com> #6Mar 13, 2023 05:02PM

Yes, always run the dex transform non-incrementally is another workaround.

as...@gmail.com <as...@gmail.com> #7Mar 15, 2023 05:23PM

Is it possible to do same patch with included build?

hu...@google.com <hu...@google.com> #8Mar 17, 2023 05:47PM

I guess so, could you give it a try?

as...@gmail.com <as...@gmail.com> #9Mar 20, 2023 07:57AM

I've tried(not very neat) but it only work with buildSrc

Anyway, what about issue, I've tried a lot of attempts to fix and helped only turn off jacoco transformation `useJacocoTransformInstrumentation = false`.

What was changed in agp 7.4.x from 7.3.1 that break this?

hu...@google.com <hu...@google.com> #10Mar 20, 2023 12:41PM

Are you referring to https://issuetracker.google.com/266109833#comment2? That issue is different from this issue.

(Regarding What was changed in agp 7.4.x from 7.3.1 that break this?: I don't know, my theory is that some of your code has changed too when updating AGP, which exposed the Jacoco bug. Please post your comments in that issue if you'd like to continue discussing this.)

sh...@pinterest.com <sh...@pinterest.com> #11Mar 21, 2023 10:47PM

Edit: we were actually not applying the patch correctly, so this was user error.

We've been running with a patched version of DexingTransformer.kt for about a week but are still seeing the issue. First I tried commenting out @CacheableTransform on DexingNoClasspathTransform, but after devs were still experiencing the error, I tried reverting that change and instead applying what I'd suggested in comment #5, but we're still running into the issue.

Is there any other workaround or additional information I can provide to help debug? At first I thought it's perhaps the patched version of DexingTransformer.kt wasn't being picked up, but I confirmed that it was by throwing an exception inside of doTransform.

Message last modified on Mar 22, 2023 08:18PM

hu...@google.com <hu...@google.com> #12Mar 22, 2023 01:19PM

Regarding the non-cacheable fix: Can you check if it actually takes effect (i.e., the transform outputs are not cached)? Maybe comment out @CacheableTransform in the BaseDexingTransform class too?

Regarding the non-incremental fix: Note that the stacktrace in comment #1 is:

at com.android.build.gradle.internal.dependency.BaseDexingTransform.processIncrementally(DexingTransform.kt:227)
at com.android.build.gradle.internal.dependency.BaseDexingTransform.doTransform(DexingTransform.kt:137)

Please check your stacktrace after applying the non-incremental fix:

If it still shows processIncrementally, that means the fix was not applied successfully.
If it shows processNonIncrementally, that is probably a different bug.

sh...@pinterest.com <sh...@pinterest.com> #13Mar 22, 2023 08:19PM

Thanks! I did confirm we were not applying the patch correctly (as you pointed out, processIncrementally was in the callstack) but we are now. I will report back if we're still having issues. Appreciate the support.

hu...@google.com <hu...@google.com> #14Apr 6, 2023 10:55AM

Marked as fixed.

Update: We have fixed this issue with this change: https://android.googlesource.com/platform/tools/base/+/03bc950504c4c64cef2b0de377175f02a62e70d3. It will be included AGP 8.1.0-beta01.

as...@gmail.com <as...@gmail.com> #15Apr 11, 2023 12:45PM

wow, thanks for fixing it so fast

Is it possible to cherry-pick changes to AGP 8.0?

hu...@google.com <hu...@google.com> #16Apr 11, 2023 01:42PM

Thanks, but I'm afraid that it's a bit late for AGP 8.0.

(Btw, as mentioned at comment #10, Issue 266109833 is different from this issue, so if you're looking for a fix for Issue 266109833, the above change will not fix it.)

as...@gmail.com <as...@gmail.com> #17Apr 18, 2023 04:34PM

Yeah, thank you, you was right, I've build jacoco from this branch(

https://github.com/jacoco/jacoco/pull/893) and push it to local maven and it's helped to fix my problem

Message last modified on Apr 18, 2023 04:35PM

an...@google.com <an...@google.com> #18Apr 18, 2024 09:05PM

Thank you for your patience while our engineering team worked to resolve this issue. A fix for this issue is now available in:

Android Studio Koala | 2024.1.1 Canary 6
Android Gradle Plugin 8.5.0-alpha06

We encourage you to try the latest update.

If you notice further issues or have questions, please file a new bug report.

Thank you for taking the time to submit feedback — we really appreciate it!

hu...@google.com <hu...@google.com> #19Apr 19, 2024 10:16AM

FYI: Comment #18 is an automated comment, referring to a small improvement to the fix in comment #14. It will be available in AGP 8.5.0-alpha06+: https://android.googlesource.com/platform/tools/base/+/8473e5fa2f36600947d29a49b6cce17ee46dbc45

Message last modified on Apr 19, 2024 10:17AM

ma...@gmail.com <ma...@gmail.com> #20Jun 22, 2024 04:09AM