Status Update
No update yet.
Comments
uc...@google.com
Thank you for your feedback. Team may reach out for more feedback in reproducing or triaging this issue.
[Deleted User] <[Deleted User]>
I wanted to check to see if there was any update on this issue. It is blocking us from updating AGP. Same error is also happening on AGP 3.2 beta05.
je...@google.com
ag...@google.com
Thanks for the report. This looks like a JNI issue in dex2oat where local references to DexCache objects are not deleted. It is interesting that this is only happening when upgrading AGP and happens with both DX and D8. As far as I know DX did not change between 3.1.2 and 3.1.3.
Nicolas, does this ring a bell for you or anyone else on the Art team? If we can understand why this happens we might be able to come up with a workaround.
Nicolas, does this ring a bell for you or anyone else on the Art team? If we can understand why this happens we might be able to come up with a workaround.
ag...@google.com
Adding Ivan as well. Ivan, do you have a good picture of what changed on the plugin side from 3.1.2 to 3.1.3? Since this is apparently happening with both DX and D8 it seems we should look outside the dexer?
ag...@google.com
Christopher and Chris, two questions:
1. are there more dex files in the APK with 3.1.3 compared to 3.1.2?
2. what exactly is 'dex2oatd-androidN' - is that something you built yourself or where did you get it (to make sure we are looking at the right thing).
1. are there more dex files in the APK with 3.1.3 compared to 3.1.2?
2. what exactly is 'dex2oatd-androidN' - is that something you built yourself or where did you get it (to make sure we are looking at the right thing).
ag...@google.com
Make that three questions:
3. does the apk install on an actual device or does this happen with dex2oat on a device as well?
3. does the apk install on an actual device or does this happen with dex2oat on a device as well?
ga...@google.com
Between the working AGP (3.1.1), and the failing one (3.1.2), there were no changes to DX.
The only change in the plugin ishttp://ag/3890736 . In 3.1.1, a single jar is passed to dexer (DX/D8), but in 3.1.2 we split the jar into "min(4, CPU / 2) + 1" parts, and pass that to dexer. However, this should not impact the output of dexing. Is everything working with 3.1.1 both with DX and D8?
The only change in the plugin is
[Deleted User] <[Deleted User]>
Some additional context, we produce custom Android based hardware and are running dex2oat preopt on the apk before including it into the OS image that is ultimately installed on to the device. The failure that we are seeing is essentially a dry run of dex2oat in our CI pipeline that we've included to make sure we don't break dex2oat in the actual Android OS build pipeline.
Answers to your questions:
1. Here are the dex counts with each version of AGP:
3.1.1 d8=false - 274
3.1.2 d8=false - 971
3.1.2 d8=true - 971
3.1.3 d8=false - 971
3.1.3 d8=true - 971
3.2b5 d8=true - 1113
2. dex2oatd-androidN - It is the `dex2oat` binary built from the 7.1.2 source as provided by Qualcomm for the msm8939 SOC and associated ART libraries.
3. I installed the app on a Pixel AVD emulator on API level 25 which matches the version of Android we are using for our custom hardware. Watched the logcat while the installation and it appeared that dex2oat ran successfully. I attached the relevant logcat output.
re: Is everything working with 3.1.1 both with DX and D8?
We can not run D8 with 3.1.1 due tohttps://issuetracker.google.com/issues/75985796 (RejectedExecutionException: Thread limit exceeded replacing blocked worker). 3.1.1 works fine with d8 disabled and is what we are currently using.
Answers to your questions:
1. Here are the dex counts with each version of AGP:
3.1.1 d8=false - 274
3.1.2 d8=false - 971
3.1.2 d8=true - 971
3.1.3 d8=false - 971
3.1.3 d8=true - 971
3.2b5 d8=true - 1113
2. dex2oatd-androidN - It is the `dex2oat` binary built from the 7.1.2 source as provided by Qualcomm for the msm8939 SOC and associated ART libraries.
3. I installed the app on a Pixel AVD emulator on API level 25 which matches the version of Android we are using for our custom hardware. Watched the logcat while the installation and it appeared that dex2oat ran successfully. I attached the relevant logcat output.
re: Is everything working with 3.1.1 both with DX and D8?
We can not run D8 with 3.1.1 due to
ag...@google.com
Wow, that is a massive amount of dex files - that doesn't seem right - not for 3.1.1 either.
I'm wondering if something in the project setup has gotten dex merging disabled. How many classes are in these dex files? In Android Studio, if you go to "build->analyze apk" and point to this apk, you can click on the dex files and it will show you have many classes and methods are in each dex file (and which classes and methods that is). If there are very few classes per dex file we need to have a look at your project setup to figure out why dex file merging is going wrong here.
Nicolas, I'm taking this bug back. With these numbers on dex files in this APK it makes sense that dex2oat would hit that limit by holding on to a JNI local reference to a DexCache per dex file.
I'm wondering if something in the project setup has gotten dex merging disabled. How many classes are in these dex files? In Android Studio, if you go to "build->analyze apk" and point to this apk, you can click on the dex files and it will show you have many classes and methods are in each dex file (and which classes and methods that is). If there are very few classes per dex file we need to have a look at your project setup to figure out why dex file merging is going wrong here.
Nicolas, I'm taking this bug back. With these numbers on dex files in this APK it makes sense that dex2oat would hit that limit by holding on to a JNI local reference to a DexCache per dex file.
ga...@google.com
That number of dex files does seem to be the cause, and most likely the build pipeline is the culprit. Chris, how many subprojects do you have in you build? Also, is your min sdk version 23 or above?
For dex merging of debuggable builds, the build pipeline is the following. External (Maven) libraries are merged as tightly as possible. However, if minSdkVersion >= 23, every sub-project will be merged separately, and each of those projects will be split into e.g. 4 shards. This means that for 200 projects, there could be 800 dex files. We do this so if a single library changes, we will dex and dex-merge only that library.
To confirm this is indeed the issue, can you please do one of the following:
- Pass "-Pandroid.dexingNumberOfBuckets=1" when running the build with 3.2.0 (flag was added only in 3.2.0-alpha10).
- Try dex2oat on a non-debuggable build. You can just add to your build.gradle: "android.buildTypes.debug.debuggable false", and build the debug variant.
Please let me know if this fixes the issue for you.
For dex merging of debuggable builds, the build pipeline is the following. External (Maven) libraries are merged as tightly as possible. However, if minSdkVersion >= 23, every sub-project will be merged separately, and each of those projects will be split into e.g. 4 shards. This means that for 200 projects, there could be 800 dex files. We do this so if a single library changes, we will dex and dex-merge only that library.
To confirm this is indeed the issue, can you please do one of the following:
- Pass "-Pandroid.dexingNumberOfBuckets=1" when running the build with 3.2.0 (flag was added only in 3.2.0-alpha10).
- Try dex2oat on a non-debuggable build. You can just add to your build.gradle: "android.buildTypes.debug.debuggable false", and build the debug variant.
Please let me know if this fixes the issue for you.
ng...@google.com
#10: Thanks Mads.
#9 and #10: you're probably using dex2oatd (the debug version of dex2oat) which may create more jni references than the release version (seen on device). Also, starting O MR1, we now do class unloading between dex files, which if we handle it properly, should reduce the problem.
#9 and #10: you're probably using dex2oatd (the debug version of dex2oat) which may create more jni references than the release version (seen on device). Also, starting O MR1, we now do class unloading between dex files, which if we handle it properly, should reduce the problem.
[Deleted User] <[Deleted User]>
> Chris, how many subprojects do you have in you build?
We use a monorepo and currently have 317 modules. This specific project doesn't use them all but it does use most of them. If I had to guess, probably in the range of ~220 of them. I could try and get an exact number if it is necessary.
> Also, is your min sdk version 23 or above?
Yes
> Pass "-Pandroid.dexingNumberOfBuckets=1" when running the build with 3.2.0 (flag was added only in 3.2.0-alpha10)
This looks like it has resolved the dex2oat exception. It reduced it down to 237 dexes. I also noticed that it reduced the apk size from 96.8 MB to 91 MB. It seems to me that this should only be used for debug builds to speed up dex merging, is that correct? Is there any reason why we would would want dex sharding in our release apk?
We use a monorepo and currently have 317 modules. This specific project doesn't use them all but it does use most of them. If I had to guess, probably in the range of ~220 of them. I could try and get an exact number if it is necessary.
> Also, is your min sdk version 23 or above?
Yes
> Pass "-Pandroid.dexingNumberOfBuckets=1" when running the build with 3.2.0 (flag was added only in 3.2.0-alpha10)
This looks like it has resolved the dex2oat exception. It reduced it down to 237 dexes. I also noticed that it reduced the apk size from 96.8 MB to 91 MB. It seems to me that this should only be used for debug builds to speed up dex merging, is that correct? Is there any reason why we would would want dex sharding in our release apk?
ga...@google.com
Thanks for trying that. To clarify, we only shard for debuggable variants. Non-debuggable builds will be merged into as few dex files as possible. By default, there are 2 build types, debug (which has debuggable = true), and release (with debuggable = false). For the variant that you build, how do you declare the bulid type?
[Deleted User] <[Deleted User]>
So we do use debug signing configs for our release build because only our CI has access to our real release signing key. We don't declare "android.buildTypes.debug.debuggable" true or false in any of our build scripts. Here is the relevant snippet from this app's build.gradle:
buildTypes {
debug {
versionNameSuffix "dev"
signingConfig signingConfigs.debug
}
if (!rootProject.ext.engBuild) {
release {
signingConfig signingConfigs.debug
}
}
}
signingConfigs {
debug {
storeFile file("../../../certs/platform.keystore")
storePassword "************"
keyAlias "key"
keyPassword "************"
}
}
engBuild is just a special flag that enables some build time improvements and we just use it here to make sure those are all turned off for release builds.
buildTypes {
debug {
versionNameSuffix "dev"
signingConfig signingConfigs.debug
}
if (!rootProject.ext.engBuild) {
release {
signingConfig signingConfigs.debug
}
}
}
signingConfigs {
debug {
storeFile file("../../../certs/platform.keystore")
storePassword "************"
keyAlias "key"
keyPassword "************"
}
}
engBuild is just a special flag that enables some build time improvements and we just use it here to make sure those are all turned off for release builds.
[Deleted User] <[Deleted User]>
Regarding "-Pandroid.dexingNumberOfBuckets=1", is there somewhere I can set this in either gradle.properties or our build.gradle? If not currently, can we get that in a future release of AGP?
ga...@google.com
As a workaround until we release the fix, you can add android.dexingNumberOfBuckets to gradle.properties file.
Regarding debuggable, signing config does not affect it. Here is an example:
android {
buildTypes {
debug {} // this build is debuggable by default
release {} // this build is not debuggable by default
devBuild.initWith(debug) // this build is debuggable because it is a copy of debug
nonDebuggableBuild { debuggable false } // this is explicitly non-debuggable build
debuggableBuild { debuggable true } // this is explicitly non-debuggable build
}
}
When running, "assembleDebug", "assembleDevBuild", "assembleDebuggableBuild" there will be "many" dex files. When running "assembleRelease", "assembleNonDebuggableBuild" we will merge as tightly as possible.
Regarding debuggable, signing config does not affect it. Here is an example:
android {
buildTypes {
debug {} // this build is debuggable by default
release {} // this build is not debuggable by default
devBuild.initWith(debug) // this build is debuggable because it is a copy of debug
nonDebuggableBuild { debuggable false } // this is explicitly non-debuggable build
debuggableBuild { debuggable true } // this is explicitly non-debuggable build
}
}
When running, "assembleDebug", "assembleDevBuild", "assembleDebuggableBuild" there will be "many" dex files. When running "assembleRelease", "assembleNonDebuggableBuild" we will merge as tightly as possible.
ga...@google.com
This got fixed while we were taking care of https://issuetracker.google.com/132840182 , I just forgot to update this issue. Sorry about that.
Description
Android Plugin Version: 3.1.3
Compile Sdk Version: 27
Build Tools Version: 27.0.3
We currently use AGP version 3.1.1 with no error. When switching to AGP 3.1.2 or 3.1.3, the following error occurs when trying to invoke dex2oat:
[2018-06-19 18:44:59] dex2oatd-androidN F 113882 113882 art/runtime/runtime.cc:427]JNI ERROR (app bug): local reference table overflow (max=512)
[2018-06-19 18:44:59] local reference table dump:
[2018-06-19 18:44:59] Last 10 entries (of 512):
[2018-06-19 18:44:59] 511: 0x74dc47d8 java.lang.DexCache
[2018-06-19 18:44:59] 510: 0x74dc4790 java.lang.DexCache
[2018-06-19 18:44:59] 509: 0x74dc4748 java.lang.DexCache
[2018-06-19 18:44:59] 508: 0x74dc4700 java.lang.DexCache
[2018-06-19 18:44:59] 507: 0x74dc46b8 java.lang.DexCache
[2018-06-19 18:44:59] 506: 0x74dc4670 java.lang.DexCache
[2018-06-19 18:44:59] 505: 0x74dc4628 java.lang.DexCache
[2018-06-19 18:44:59] 504: 0x74dc45e0 java.lang.DexCache
[2018-06-19 18:44:59] 503: 0x74dc4598 java.lang.DexCache
[2018-06-19 18:44:59] 502: 0x74dc4550 java.lang.DexCache
[2018-06-19 18:44:59] Summary:
[2018-06-19 18:44:59] 512 of java.lang.DexCache (512 unique instances)
[2018-06-19 18:44:59]
[2018-06-19 18:44:59] dex2oatd-androidN F 113882 113882 art/runtime/runtime_linux.cc:336]*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
[2018-06-19 18:44:59] Fatal signal 6 (SIGABRT), code -6 (SI_TKILL)
[2018-06-19 18:44:59] OS: Linux 3.10.0-514.10.2.el7.x86_64 (x86_64)
Here is what we are running:
dex2oatd-androidN --runtime-arg -Xms64m --runtime-arg -Xmx512m --runtime-arg -classpath --runtime-arg & --boot-image=androidN/dex_bootjars/system/framework/boot.art --dex-file=app-debug.apk --oat-file=app-debug.odex --android-root=androidN/system --instruction-set=arm64 --instruction-set-variant=generic --instruction-set-features=default --include-patch-information --runtime-arg -Xnorelocate --no-generate-debug-info --abort-on-hard-verifier-error --no-inline-from=core-oj.jar --compile-pic
This occurs when using either dex or D8.