Using d8 desugaring in Android - desugared classes are defined in multiple DEX files [171666278]

WAI

Bug

Status Update

No update yet.

Description

sg...@google.com

created issue #1

Oct 26, 2020 07:36AM

From Stackoverflow:

I'm attempting to use the new desugaring features in AGP, however I am getting the following error when trying to use ConcurrentHashMap.newKeySet() in my app:

10-23 21:17:49.471 5023-5023/uk.org.mattford.scoutlink E/AndroidRuntime: FATAL EXCEPTION: main
    Process: uk.org.mattford.scoutlink, PID: 5023
    java.lang.NoSuchMethodError: No static method newKeySet()Lj$/util/concurrent/ConcurrentHashMap$KeySetView; in class Lj$/util/concurrent/ConcurrentHashMap; or its super classes (declaration of 'j$.util.concurrent.ConcurrentHashMap' appears in /data/app/uk.org.mattford.scoutlink-1/base.apk:classes3.dex)
        at org.pircbotx.hooks.managers.ThreadedListenerManager.<init>(ThreadedListenerManager.java:49)
        at org.pircbotx.Configuration$Builder.getListenerManager(Configuration.java:884)
        at org.pircbotx.Configuration$Builder.addListener(Configuration.java:726)
        at uk.org.mattford.scoutlink.irc.IRCService.connect(IRCService.java:143)
        at uk.org.mattford.scoutlink.irc.IRCService.onStartCommand(IRCService.java:67)
        at android.app.ActivityThread.handleServiceArgs(ActivityThread.java:2864)
        at android.app.ActivityThread.access$2100(ActivityThread.java:144)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1376)
        at android.os.Handler.dispatchMessage(Handler.java:102)
        at android.os.Looper.loop(Looper.java:135)
        at android.app.ActivityThread.main(ActivityThread.java:5221)
        at java.lang.reflect.Method.invoke(Native Method)
        at java.lang.reflect.Method.invoke(Method.java:372)
        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:899)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:694)

I have the following in my top level build.gradle

buildscript {
    repositories {
        google()
        jcenter()
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:4.1.0'
        classpath 'com.google.gms:google-services:4.3.4'
        classpath 'com.google.firebase:firebase-crashlytics-gradle:2.3.0'
    }
}

allprojects {
    repositories {
        google()
        jcenter()
        maven {
            url 'https://oss.sonatype.org/content/repositories/snapshots'
        }
    }
}

and in my module build-gradle:

apply plugin: 'com.android.application'
apply plugin: 'com.google.gms.google-services'
apply plugin: 'com.google.firebase.crashlytics'

android {
    compileSdkVersion 29
    buildToolsVersion '29.0.3'
    defaultConfig {
        applicationId "uk.org.mattford.scoutlink"
        minSdkVersion 16
        targetSdkVersion 29
        multiDexEnabled true
    }
    buildTypes {
        release {}
    }
    packagingOptions {
        exclude 'META-INF/DEPENDENCIES'
        exclude 'META-INF/LICENSE'
        exclude 'META-INF/LICENSE.txt'
        exclude 'META-INF/license.txt'
        exclude 'META-INF/NOTICE'
        exclude 'META-INF/NOTICE.txt'
        exclude 'META-INF/notice.txt'
        exclude 'META-INF/ASL2.0'
    }
    compileOptions {
        coreLibraryDesugaringEnabled true
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }
    productFlavors {
    }
    buildFeatures {
        viewBinding = true
    }
}

dependencies {
    implementation 'org.slf4j:slf4j-android:1.7.30'
    implementation 'org.pircbotx:pircbotx:2.3-SNAPSHOT'
    implementation 'androidx.appcompat:appcompat:1.2.0'
    implementation 'com.google.firebase:firebase-analytics:17.6.0'
    implementation 'com.google.firebase:firebase-crashlytics:17.2.2'
    implementation 'com.google.android.material:material:1.2.1'

    implementation "androidx.room:room-runtime:2.2.5"
    implementation 'androidx.legacy:legacy-support-v4:1.0.0'
    implementation 'androidx.recyclerview:recyclerview:1.1.0'
    implementation 'androidx.lifecycle:lifecycle-extensions:2.2.0'
    implementation "androidx.lifecycle:lifecycle-viewmodel:2.2.0"
    annotationProcessor "androidx.room:room-compiler:2.2.5"

    implementation 'androidx.multidex:multidex:2.0.1'
    coreLibraryDesugaring 'com.android.tools:desugar_jdk_libs:1.0.10'
}

Looking at the APK analyser. I can see that j$.util.concurrent.ConcurrentHashMap is defined in classes3.dex but does not include the newKeySet method. However j$.util.concurrent.ConcurrentHashMap is also defined in classes4.dex and does have this method!

The ConcurrentHashMap.newKeySet call I am trying to work around is in the pircbotx dependency. However I have put a call to this method in my Application's onCreate method and this also crashes the app.

Thanks in advance for any help. I've been at this for hours now to no avail.

Comments

sg...@google.com <sg...@google.com> #2Oct 26, 2020 07:41AM

This was fixed in

https://android-review.googlesource.com/875694 and will be available in alpha10

cl...@google.com <cl...@google.com> #3Oct 26, 2020 07:44AM

Sgjesse asked the following questions on stack overflow:

Does both classes3.dex and classes4.dex only have classes with the j$ prefix?
What happens if you just remove classes3.dex (the one missing the method) from the apk?
Can you share a project with this issue, or a simple example project using the pricboxt dependency demonstrating the issue?
can you share the apk with the issue?

Waiting for the reporter answers.

It looks like for some reasons the application is building 2 desugared library dex files with different sets of keep rules, messing up the apk. This is a multidex application with minSDK 16, that may be related to the problem.

sg...@google.com <sg...@google.com> #4Oct 26, 2020 08:28AM

Just looked at the apk that the developer shared, and found:

Only classes3.dex have j$ classes, and j$.util.concurrent.ConcurrentHashMap does not define newKeySet.
In classes4.dex the following two invokes are present:

grep -r newKeySet app-debug/smali_classes4
app-debug/smali_classes4/org/pircbotx/hooks/managers/ThreadedListenerManager.smali:    invoke-static {}, Lj$/util/concurrent/ConcurrentHashMap;->newKeySet()Lj$/util/concurrent/ConcurrentHashMap$KeySetView;
app-debug/smali_classes4/org/pircbotx/hooks/managers/ThreadedListenerManager.smali:    invoke-static {}, Lj$/util/concurrent/ConcurrentHashMap;->newKeySet()Lj$/util/concurrent/ConcurrentHashMap$KeySetView;

Could be an issue with the generated keep rules.

cl...@google.com <cl...@google.com> #5Oct 26, 2020 08:49AM

Status: Won't Fix (Intended Behavior)

I don't think it's a keep rule issue.

I think it's a problem with covariant return type and ConcurrentHashMap APIs. ConcurrentHashMap in the desugared library does not expose the KeySetView type and keep Java 7 return types (Set).

In https://github.com/google/desugar_jdk_libs/blob/master/src/share/classes/java/util/concurrent/ConcurrentHashMap.java , the method newKeyset is not present. The method keySet is present, but as follows:

// For desugar: keep Java 7 API, don't expose KeySetView for now
// public KeySetView<K,V> keySet() {
public Set<K> keySet() {

Now when I look in the documentation (https://developer.android.com/studio/write/java8-support-table) I see:

public ConcurrentHashMap.KeySetView keySet(Object mappedValue)
public static ConcurrentHashMap.KeySetView newKeySet()
public static ConcurrentHashMap.KeySetView newKeySet(int initialCapacity)

So I understand someone got confused because the doc says it's supported while obviously it's not.

I think we should sync by visio on how we should proceed here.

cl...@google.com <cl...@google.com> #6Oct 26, 2020 09:41AM

At his point we don't support the 3 methods.

mr...@gmail.com <mr...@gmail.com> #7Oct 26, 2020 10:39AM

Hi, I'm the original reporter. Is that a final decision and are there any plans to support these methods in future?

cl...@google.com <cl...@google.com> #8Oct 26, 2020 12:28PM

Sorry I did not elaborate I did not think you would check.

I filed in a feature request (b/171682016) to support these 3 methods instead of this bug. The long term plan is to try to support them. It's however not easy to extend the desugared library, and I don't think anyone has time to look into this before next quarter, so don't expect any short-term fix.

At this point the 3 methods are not supported and we are going to update the documentation as soon as possible to strip them out.

mr...@gmail.com <mr...@gmail.com> #9Oct 26, 2020 12:51PM

Understood, thanks for looking into this, I'll have to find another way around this for now.