Language ID SIGSEGV crashes (LanguageIdentificationJni_nativeIdentifyPossibleLanguages) by R8 optimizer [203573699]

Fixed

Bug

adexe s nau

Status Update

No update yet.

Description

ga...@linecorp.com

created issue #1

Oct 20, 2021 01:32AM

Environment:
Android Gradle Plugin 7.0.2
R8 : 3.0.69

Original report is here : https://issuetracker.google.com/issues/194915686#comment11 (Comment includes reproducible project.)

When upgrade AGP7, app is starting to crash from liblanguage_id_jni.so which is included on Firebase MLKit Language Id library.

Workarounds are

Build with debugmode
- I tried to disable several optimization on R8, crashes also disappeared.
Downgrade AGP 4.2

Comments

cl...@google.com <cl...@google.com> Oct 20, 2021 09:07AM

Reassigned to cl...@google.com.

cl...@google.com <cl...@google.com> #2Oct 20, 2021 11:45AM

I have tried to generate a dump and got:

...
> Task :app:extractProductionReleaseNativeSymbolTables
Unable to extract native debug metadata from /Users/clementbera/Desktop/BUG/mlkit_agp7/app/build/intermediates/merged_native_libs/productionRelease/out/lib/armeabi-v7a/liblanguage_id_jni.so because unable to locate the objcopy executable for the armeabi-v7a ABI.
Unable to extract native debug metadata from /Users/clementbera/Desktop/BUG/mlkit_agp7/app/build/intermediates/merged_native_libs/productionRelease/out/lib/x86/liblanguage_id_jni.so because unable to locate the objcopy executable for the x86 ABI.
Unable to extract native debug metadata from /Users/clementbera/Desktop/BUG/mlkit_agp7/app/build/intermediates/merged_native_libs/productionRelease/out/lib/arm64-v8a/liblanguage_id_jni.so because unable to locate the objcopy executable for the arm64-v8a ABI.
Unable to extract native debug metadata from /Users/clementbera/Desktop/BUG/mlkit_agp7/app/build/intermediates/merged_native_libs/productionRelease/out/lib/x86_64/liblanguage_id_jni.so because unable to locate the objcopy executable for the x86_64 ABI.

...
[retraced]
Caused by: com.android.tools.r8.CompilationFailedException: Compilation failed to complete
        at Version.fakeStackEntry(Version_3.0.69.java:0)
        at com.android.tools.r8.utils.FeatureClassMapping$JarFileOrigin.a(FeatureClassMapping.java:68)
        at com.android.tools.r8.utils.FeatureClassMapping$JarFileOrigin.a(FeatureClassMapping.java:28)
        at com.android.tools.r8.utils.FeatureClassMapping$JarFileOrigin.a(FeatureClassMapping.java:27)
        at com.android.tools.r8.utils.FeatureClassMapping$JarFileOrigin.b(FeatureClassMapping.java:3)
        at com.android.tools.r8.D8.run(D8.java:123)
        at com.android.builder.dexing.D8DexArchiveBuilder.convert(D8DexArchiveBuilder.java:115)
        ... 37 more
Caused by: com.android.tools.r8.internal.b: Dumped compilation inputs to: mydump.zip
        at com.android.tools.r8.apimodel.AndroidApiDatabaseClassForClass_android_preference_PreferenceScreen.visitFields(AndroidApiDatabaseClassForClass_android_preference_PreferenceScreen.java:13)
        at com.android.tools.r8.apimodel.AndroidApiDatabaseClassForClass_android_preference_PreferenceScreen.visitFields(AndroidApiDatabaseClassForClass_android_preference_PreferenceScreen.java:21)
        at com.android.tools.r8.dex.ApplicationReader$ClassReader.a(ApplicationReader.java:61)
        at com.android.tools.r8.dex.ApplicationReader$ClassReader.buildClassProvider(ApplicationReader.java:428)
        at com.android.tools.r8.D8.readApp(D8.java:177)
        at com.android.tools.r8.D8.run(D8.java:192)
        at com.android.tools.r8.D8.lambda$run$1(D8.java:126)
        at com.android.tools.r8.utils.FeatureClassMapping$JarFileOrigin.a(FeatureClassMapping.java:24)
        ... 40 more

Is this build working on Linux only? I'm building from a Mac and I wonder if that causes the "Unable to extract native debug metadata" which then leads me to an invalid crash.

ga...@linecorp.com <ga...@linecorp.com> #3Oct 20, 2021 04:54PM

> Task :app:extractProductionReleaseNativeSymbolTables
Unable to extract native debug metadata from /Users/clementbera/Desktop/BUG/mlkit_agp7/app/build/intermediates/merged_native_libs/productionRelease/out/lib/armeabi-v7a/liblanguage_id_jni.so because unable to locate the objcopy executable for the armeabi-v7a ABI.
Unable to extract native debug metadata from /Users/clementbera/Desktop/BUG/mlkit_agp7/app/build/intermediates/merged_native_libs/productionRelease/out/lib/x86/liblanguage_id_jni.so because unable to locate the objcopy executable for the x86 ABI.
Unable to extract native debug metadata from /Users/clementbera/Desktop/BUG/mlkit_agp7/app/build/intermediates/merged_native_libs/productionRelease/out/lib/arm64-v8a/liblanguage_id_jni.so because unable to locate the objcopy executable for the arm64-v8a ABI.
Unable to extract native debug metadata from /Users/clementbera/Desktop/BUG/mlkit_agp7/app/build/intermediates/merged_native_libs/productionRelease/out/lib/x86_64/liblanguage_id_jni.so because unable to locate the objcopy executable for the x86_64 ABI.

This just means that NDK is not installed on your mac.
And this should not be related with R8 crash.
Please check default NDK version on https://developer.android.com/studio/releases/gradle-plugin#compatibility

Also following log shows same warnings. but build can be succeeded with mlkit_agp7 branch.
Linux build without NDK : https://scans.gradle.com/s/zdzekqelqdvpm/console-log#L58
OSX build without NDK : https://scans.gradle.com/s/isnpa77cqns2w/console-log#L58

And app crashed whether NDK is installed or not.

cl...@google.com <cl...@google.com> #4Oct 22, 2021 09:09AM

Reassigned to ri...@google.com.

Looks like a potential feature split issue Rico.

ri...@google.com <ri...@google.com> #5Oct 25, 2021 07:55AM

I don't think this has anything to do with feature splits
Based on the crash this seems like we are missing a keep rule for something that is called from native code:
yliuyliu@: would anything call into com.google.mlkit.nl.languageid.internal.LanguageIdentificationJni from native? (can you point me to the code, cs search did not find this for me but I am probably holding it wrong

A simple workaround for now is to keep LanguageIdentificationJni:
-keep class com.google.mlkit.nl.languageid.internal.LanguageIdentificationJni { *; }
fixes the issue for me

ri...@google.com <ri...@google.com> #6Oct 25, 2021 09:50AM

Doing a little more digging, I would bet that we grap the field:
private java.nio.MappedByteBuffer zzd
from native (this is the only real difference from 4.2 to 7.0 in what we actually remove in that class)
Keeping this field also solves the issue:
-keep class com.google.mlkit.nl.languageid.internal.LanguageIdentificationJni {
private java.nio.MappedByteBuffer zzd;
}
So R8 in 7.0 did some analysis that enables us to remove the field (since there is no keep on it) and it is used from native (which r8 can't see)

Message last modified on Oct 25, 2021 09:50AM

ba...@google.com <ba...@google.com> #7Oct 25, 2021 02:39PM

Ahh... The MappedByteBuffer member itself isn't accessed from native code, but the object keeps a reference around to prevent it from being garbage collected while the native code is accessing the contents of the MappedByteBuffer.

The code has since been removed at head, so this is mostly for my own education, but is there a better way to tell R8 that the member isn't dead that does not involve Proguard config?

sg...@google.com <sg...@google.com> #8Oct 25, 2021 03:47PM

Not sure how the native code gets to access the contents of the MappedByteBuffer without accessing the member. Is the MappedByteBuffer created in native code with a content reference stored in the native code?

As far as I can see this code is part of the GMSCore client API - isn't that true?

A proguard config is the way to specify that something is accessed which cannot be statically determined from analyzing the Java byte code for the application. If this (as I assume) is part of the GMSCore client SDK, then (as the client SDK is run through shrinking/obfuscation before shipping) there needs to be rules both for the build of the client SDK, and for the consumer of the client SDK. In this case the following could be used for both:

-keep class com.google.mlkit.nl.languageid.internal.LanguageIdentificationJni {
  private java.nio.MappedByteBuffer <original field name>;
}

In this case the <original field name> will not be renamed (e.g. to zzd).

Using annotations (e.g. b/203692302#comment2) is also an option, but that gets more complicated, as the annotations then needs to be preserved when building the client SDK.

yl...@google.com <yl...@google.com> #9Oct 25, 2021 03:53PM

Thanks for digging into the root cause!

The old code com.google.mlkit.nl.languageid.internal.LanguageIdentificationJni was deleted at head.

But similar com.google.mlkit.nl.languageid.internal.ThickLanguageIdentifier is at head that we can keep MappedByteBuffer field to do a patch fix?

ba...@google.com <ba...@google.com> #10Oct 25, 2021 04:00PM

@sgjesse: The Java code creates the MappedByteBuffer, stores it in the member (to prevent it from being garbage-collected), and passes it to native code, which then gets the memory address using GetDirectBufferAddress().

yl...@google.com <yl...@google.com> #11Nov 2, 2021 11:10PM

Marked as fixed.

Added workarounds in Known issues section in public docs.