IssueTracker

If it helps, I have a screencap attached that shows the build/.transforms dir of the module that has the mixed source set. You can see that the contents differ dramatically based on whether or not the ObserverFix.java file is present. I will note that the actual jar produced by normal javac/kotlinc has all the expected class files.

At this point, I don't know if the bug is in:

1. AGP
2. Gradle
3. D8

I think I know more or less where the issue is, but I haven't precisely pinpointed the problem just yet. Sorry of this comment is a bit messy, but I haven't delved this deep into AGP code in a while and I wanted to do a brain dump before the weekend.

In AGP 4.2.0, in SubtractingArtifactCollection, line 89, there is a function initArtifactResult(), which I believe to be buggy. Here's why, to summarize:

1. mainArtifacts contains our artifact at build/.transforms/51c03e... This is correct.
2. removedArtifacts does NOT contain our artifact. This is correct.
3. artifactResults does NOT contain our artifact. This is incorrect.

So the logic in initArtifactResult() is wrong: it is producing an incorrect artifactResults object, which is then used to filter mainArtifacts.

This buggy function is using the ComponentArtifactIdentifer (it.id) rather than the path, so maybe there's an ID collision?

On line 102, there is a lambda

mainArtifacts.artifactFiles.filter { f -> artifactFileSet.value.contains(f) }

when f = /Users/trobalik/Development/android-register/common/rx2-utilities-jvm/testing/build/.transforms/51c03e5924694fbadf87d9d7444d7bb0/transformed/main, that filter should be true, but is actually false. This f points to a file hierarchy that ultimately contains com/squareup/rx2/utilities/DeprecatedRxKillSwitchRule.dex. That class was written in Kotlin.

artifactFileSet does not contain the artifact it should, because artifactResults does not contain the artifact it should, because initArtifactResult() has a bug (I assume!).

I did a build comparison with Gradle Enterprise. Recall that the issue description pointed to a problem with mixed Java/Kotlin source.

Changing inputs for dexDirs
With Java and Kotlin:
  common/rx2-utilities-jvm/testing/build/.transforms/cce801b89890a0d5783032b8034a8667/transformed/main
With only Kotlin:
  common/rx2-utilities-jvm/testing/build/.transforms/51c03e5924694fbadf87d9d7444d7bb0/transformed/main
  common/rx2-utilities-jvm/testing/build/.transforms/d7b5653982cffdfe5eb48bf61ce9dd32/transformed/empty

Note those hashes. When I look in artifactFileSet, I see a hit for cce801b89890a0d5783032b8034a8667, but nothing for 51c03e5924694fbadf87d9d7444d7bb0. These transform artifacts come from different builds. I have been building with clean --no-build-cache, but it doesn't seem to matter, these transforms stay behind. I haven't manually deleted them yet because I'm still debugging.

Another round of debugging, and some deeper results:

  initArtifactResult()
    mainArtifacts.artifacts.filter { it.id.displayName.contains("rx2") }
      - main (project :common:rx2-utilities-jvm:testing)
        - This has a single file that points to ObserverFix.dex alone.
      - 5 other elements
    mainArtifacts.artifactFiles.files.filter { it.absolutePath.contains("common/rx2") }
      - [0] points to a dir that contains only ObserverFix.dex (the sole file from Java source).
        build/.transforms/cce801b89890a0d5783032b8034a8667/transformed/main
      - [1] points to a dir that contains all the Kotlin files.
        build/.transforms/51c03e5924694fbadf87d9d7444d7bb0/transformed/main
      - 3 other elements

The code snippets are what I ran in the AS debugger.

When I look at mainArtifacts.artifacts, I see only a single matching artifact, and it has a single file (based on Java source). When I instead look at mainArtifacts.artifactFiles.files, I see two files of interest, both directories. One points to the Java-based dex files, and the other to the Kotlin-based dex files.

In other words, artifacts and artifactFiles seems to be pointing to different sets of objects? That's weird?

I'm not sure if I mentioned this elsewhere, but the task I'm executing is :path:to:an:app:mergeLibDexDebugAndroidTest

I repeated the above, but this time after removing the Java source and having only Kotlin in the module in question.

  initArtifactResult()
    mainArtifacts.artifacts.filter { it.id.displayName.contains("rx2") }
      - main (project :common:rx2-utilities-jvm:testing)
        This has a single file that points to all the classes we expect.
        build/.transforms/51c03e5924694fbadf87d9d7444d7bb0/transformed/main
        This transform directory is identical to the one from the bad build with the Kotlin files.
      - 6 other elements
    mainArtifacts.artifactFiles.files.filter { it.absolutePath.contains("common/rx2") }
      - [1] points to a dir that contains all the classes we expect.
        build/.transforms/51c03e5924694fbadf87d9d7444d7bb0/transformed/main
        This transform directory is identical to the one from the bad build with the Kotlin files.
      - 4 other elements
    ImmutableSet.copyOf(mainArtifacts.artifacts.filter { !removed.contains(it!!.id) }).filter { it.file.absolutePath.contains("common/rx2") }
      - [1] points to a dir that contains all the classes we expect.
        build/.transforms/51c03e5924694fbadf87d9d7444d7bb0/transformed/main
        This transform directory is identical to the one from the bad build with the Kotlin files.
      - 1 other element

This seems to confirm that the transforms are stable, as they always produce the same hash, and produce separate outputs for each type of source (two outputs, one each for Kotlin and for Java). What's weird is that the final ArtifactCollection looks different depending on whether you look at it via mainArtifacts.artifacts vs mainArtifacts.artifactFiles. I'm not sure if this is an AGP bug or a Gradle bug.

I do note that the AGP source seems to assume they are basically identical. Consider these two functions:

// Uses `artifact.file
private fun initArtifactFileSet() = ImmutableSet.builder<File>().apply {
    for (artifact in artifactResults.value) {
        add(artifact.file)
    }
}.build()

// uses `ArtifactCollection.artifactFiles`
private fun initFileCollection() = objectFactory.fileCollection().from(
    mainArtifacts.artifactFiles.filter { f -> artifactFileSet.value.contains(f) }
).builtBy(mainArtifacts.artifactFiles, removedArtifacts.artifactFiles)

I executed :path:to:an:app:mergeLibDexDebugAndroidTest again, this time with AGP 4.1.3, and saw just a single classes.dex file in the build/.transforms directory. Inspecting this with dexdump showed me all the classes, both Java and Kotlin, were in this one dex file.

Thanks for filing this Tony. Did you manage to reduce this to a small sample project that you could share?

Update: I managed to reproduce. Setup requires:

• app with minSdkVersion 24+ (so that desugaring classpath is not needed)
• lib project without Android with 2 classes: 1 Kotlin and 1 Java
• app with androidTestImplementation project(":lib")
• android test apk will not have all classes from lib

Here is project that reproduces the issue. Running ./gradlew aDAT creates android test APK with missing classes.

Thanks for investigating and reproducing it so quickly! Let me know if I can help further. I very much want to understand this issue.

Running ./gradlew :lib:outgoingVariants there is

Secondary variants (*)
    - Variant : classes
       - Attributes
          - org.gradle.category                 = library
          - org.gradle.dependency.bundling      = external
          - org.gradle.jvm.version              = 8
          - org.gradle.libraryelements          = classes
          - org.gradle.usage                    = java-runtime
          - org.jetbrains.kotlin.localToProject = public
          - org.jetbrains.kotlin.platform.type  = jvm
       - Artifacts
          - build/classes/java/main (artifactType = java-classes-directory)
          - build/classes/kotlin/main (artifactType = java-classes-directory)

and this results in having ArtifactCollection.getArtifactFiles() return both files, but ArtifactCollection.getArtifacts() returns only 1 artifact. As already mentioned in previous comments, we can probably fix this in SubtractingArtifactCollection in AGP.

The issue is triggered when application consumes kotlin-only module and desugaring is not needed (either using java 7 lang level or deploying to device with API 24+).

Is it expected for ArtifactCollection.getArtifacts() to return only 1 artifact, when there are two? Meaning, is Gradle working as intended?

FWIW, I had a slightly different output from running outgoingVariants

--------------------------------------------------
Variant apiElements
--------------------------------------------------
Description = API elements for main.

Capabilities
    - register.common.rx2-utilities-jvm:testing:unspecified (default capability)
Attributes
    - org.gradle.category                 = library
    - org.gradle.dependency.bundling      = external
    - org.gradle.jvm.version              = 8
    - org.gradle.libraryelements          = jar
    - org.gradle.usage                    = java-api
    - org.jetbrains.kotlin.localToProject = public
    - org.jetbrains.kotlin.platform.type  = jvm

Artifacts
    - build/libs/testing.jar (artifactType = jar)

Secondary variants (*)
    - Variant : classes
       - Attributes
          - org.gradle.category                 = library
          - org.gradle.dependency.bundling      = external
          - org.gradle.jvm.version              = 8
          - org.gradle.libraryelements          = classes
          - org.gradle.usage                    = java-api
          - org.jetbrains.kotlin.localToProject = public
          - org.jetbrains.kotlin.platform.type  = jvm
       - Artifacts
          - build/classes/java/main (artifactType = java-classes-directory)
          - build/classes/kotlin/main (artifactType = java-classes-directory)
          - build/classes/kotlin/main (artifactType = java-classes-directory)

I'm not 100% sure why my build shows three artifacts vs your two, but I suspect it's because my build has two separate packages in the Kotlin source (i.e., more than one Kotlin source file, in different packages), whereas yours only has one Kotlin source file.

I've also filled https://github.com/gradle/gradle/issues/17213, to check if this Gralde behavior is expected.

FYI, we also see this issue in a slightly different context, which results in flaky builds, rather than deterministically-incorrect builds. Here's the module with multiple artifacts:

Secondary variants (*)
    - Variant : classes
       - Attributes                                                                                                                                            - org.gradle.category                 = library
          - org.gradle.dependency.bundling      = external
          - org.gradle.jvm.version              = 8
          - org.gradle.libraryelements          = classes
          - org.gradle.usage                    = java-api
          - org.jetbrains.kotlin.localToProject = public
          - org.jetbrains.kotlin.platform.type  = jvm
       - Artifacts
          - build/classes/java/main (artifactType = java-classes-directory)
          - build/classes/kotlin/main (artifactType = java-classes-directory)
          - build/classes/kotlin/main (artifactType = java-classes-directory)

This module only has Kotlin source, but it also uses Dagger, so there is generated code.

After one first build (./gradlew :path:to:module:assembleDebugAndroidTest), I inspected the resultant test APK:

$ find bad -type f -name "NoopTmn*"
bad/smali_classes2/com/squareup/tmn/NoopTmnTimings_Factory.smali
bad/smali_classes2/com/squareup/tmn/NoopTmnTimings_Factory$InstanceHolder.smali

I know the name is meaningless to you, but it's missing the class NoopTmnTimings.smali.

After re-running that build, I get a different artifact:

$ find new -type f -name "NoopTmn*"
new/smali_classes2/com/squareup/tmn/NoopTmnTimings_Factory.smali
new/smali_classes2/com/squareup/tmn/NoopTmnTimings.smali
new/smali_classes2/com/squareup/tmn/NoopTmnTimings_Factory$InstanceHolder.smali

So, this is flaky. Sometimes I get the expected file, other times I don't.

I can't explain why this is flaky, but we are hopeful that AGP 4.2.2 will also resolve this issue. We suspect/are irrationally worried it's related to the build cache somehow.

This is now fixed in studio-main and also cherry-picked to 4.2 and 7.0 branches.

You can also work around this issue by having your own version of SubtractingArtifactCollection in buildSrc which has this fix - https://cs.android.com/android-studio/platform/tools/base/+/447e6cb62d1520074572fb26c27f3d40e3ee78dd:build-system/gradle-core/src/main/java/com/android/build/gradle/internal/dependency/SubtractingArtifactCollection.kt;dlc=c499ed9775b4b7aa9d9e7942d385ae279065d0de.

Yup, we've already patched it locally :)

steamkar id,sappowt