IssueTracker

Thank you for this feedback. The team may reach out for more information on triaging or reproducing this issue.

I have tried the example project you've provided, it does not reproduce.

Could you please try running with -Pandroid.useDexArchive=false? This disables the new dexing pipeline, and fallbacks to the old one.

@Mario Did you get a chance to try the build with -Pandroid.useDexArchive=false flag?

I'm the college of @Mario and I make it work without issues and without using  -Pandroid.useDexArchive=false by using alpha6 and removing retrolambda

Sorry my comment from Apr 21, 2017 12:41PM was wrong.
With -Pandroid.useDexArchive=false it looks like this

:app:transformClassesWithDesugarForNonpayItalyCompatDebugAndroidTest
:app:transformClassesWithPreDexForNonpayItalyCompatDebugAndroidTest
Dex: warning: Ignoring InnerClasses attribute for an anonymous inner class
(org.ccil.cowan.tagsoup.Parser$1) that doesn't come with an
associated EnclosingMethod attribute. This class was probably produced by a
compiler that did not target the modern .class file format. The recommended
solution is to recompile the class from source, using an up-to-date compiler
and without specifying any "-target" type options. The consequence of ignoring
this warning is that reflective operations on this class will incorrectly
indicate that it is *not* an inner class.

:app:transformDexWithDexForNonpayItalyCompatDebugAndroidTest FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:transformDexWithDexForNonpayItalyCompatDebugAndroidTest'.
> com.android.build.api.transform.TransformException: com.android.ide.common.process.ProcessException: java.util.concurrent.ExecutionException: com.android.dex.DexIndexOverflowException: method ID not in [0, 0xffff]: 65536

We updated our example repo https://github.com/originx/Gradle_troubles_android_enteprise where you can see the issue.
Just run

./gradlew spoonGermanyDebugAndroidTest
or
./gradlew spoonGermanyDebugAndroidTest -Pandroid.useDexArchive=false

and you will see the issue

Your tests do not fit into a single dex file. I've tried plugin 2.3.0 (I've also had to enable retrolambda for app, core, and pay modules), and the number of references that are generated is higher than 64k.
Please note that if you are using plugin's internal or Retrolamba support for Java 8 language features, for each lambda expression you will generate 3 additional references, so it might affect your reference count quite significantly.

Hello and thank you for the reply. What is the expected fix for this the ticket status was moved to "verified (fixed)" it is rather confusing for me. Do you suggest we have too many tests and that's it?
Could you please clarify?
Thank you for your time

Your test sources and test-only dependencies are generating too many references. Java 8 language features are not related to this, as even when I remove compileOptions from all modules, the test dependencies cannot fit into a single dex file.

I suggest you take a look at Proguard rules that you can specify when running the tests - https://google.github.io/android-gradle-dsl/current/com.android.build.gradle.internal.dsl.ProductFlavor.html#com.android.build.gradle.internal.dsl.ProductFlavor:testProguardFile(java.lang.Object). I've also updated the bug status, not to be confusing.

Hello,

Thank you for the link and clarification.
Pardon my many questions I just want to understand the problem better.

We use proguard for the app and define proguard rules which classes to keep for the test runner that we need early on in the start of the test run.
Why did it work with older build tools?
Also why doesnt multidex work for the test app or proguard rules from the debug builds?

Thank you for the quick response

No worries Mario, I would also like to understand where the issue is coming from :)

The fact that it worked with 2.3.0 and not with 2.4.0-alpha4 is worrying. Because I am unable to reproduce with the test project you've provided, can you please compare the contents of build/intermediates/transforms/preDex/androidTest/ when building tests:
- from clean, with 2.3.0
- from clean, with 2.4.0-alpha4, with -Pandroid.useDexArchive=false
The content of these 2 directories should be the same. Please report any differences.

The support in the test runner is missing to be able to run test apk with multiple DEX files. I don't have much more info on it, sorry.

Hello,

sorry I am  a bit restricted in the work environment so I cannot build the demo project from this location but I executed a diff on a real project where the issue happens for dex folders (predex is not available in 2.3.0) its disabled for our project.

this is for 2.3.0 and 2.4.0-alpha4 in location build/intermediates/transforms/dex/androidTest/nonPayMexicoBleeding for following command:
./gradlew clean connectedNonpayMexicoBleedingDebugAndroidTest
I used flag -Pandroid.useDexArchive=false for 2.4.0 only.

diff -rq nonpayMexicoBleeding\ 2.3.0/ nonpayMexicoBleeding\ 2.4.0-alpha4/
Result:
Only in nonpayMexicoBleeding 2.3.0/debug/folders/1000: 1
Only in nonpayMexicoBleeding 2.3.0/debug/folders/1000: 10
Only in nonpayMexicoBleeding 2.4.0-alpha4/debug/folders/1000: 1f

Does this help?

I updated the demo project so you have reproducable example:
https://github.com/originx/Gradle_troubles_android_enteprise

./gradlew clean connectedNonpayMexicoBleedingDebugAndroidTest

Gradle 3.4.0-alpha7 breaks
Commit: ca9e7f7134e5418f9e27e3164efe25521a324a46

Gradle 3.1.2 works
Commit: cfa48cf0b25427d48b328ef6cf80d51fc4e5933b

Cheers

Thanks for creating a repro project. I'll try it asap.

It works with 2.3.1 because we do not merge the individual dex files, so every test dependency gets dexed individually, and packaged in the test APK. In 2.4.0-alpha7, we are merging all test dependencies to a DEX file, and in this case, they do not fit.

so we would need to turn on multidex on the test APK to make this work ?

I think we can do the following:
- If the tested app is mono-dex, test will be mono dex for < 21, and multidex for 21+.
- If the tested app is native multidex, test is as well.
- And if the tested app is legacy multidex, test is mono dex (because currently the test runner does not support pre 21 test multidex apks).
WDYT?

Hello,

Sounds like a solution that would make it work on gradle 3.4.0>= ?

What are the side effects from the architecture point of view, performance etc, depending on the effort there would it pay off to bring full multidex support to test runner and test apk both?


I always find it confusing what is the difference between:
-multidex test apk
-multidex test runner

I thought that multidex apk is already supported by MultidexInstrumentationTestRunner
https://developer.android.com/reference/com/android/test/runner/MultiDexTestRunner.html

so that should work, or are you suggesting to introduce the multidex support to the test runner also (what is less effort and more benefit from your side? )

Best regards & thanks for your time

I am not sure when the fix would land (hopefully in the next 1-2 canary releases).
For api levels 21+, ART supports multidex for both the test apk and tested apk, so nothing needs to be done there, and there is no performance penalty. However, multidex for < 21 requires patching the class loader, which is done by the test runner for the tested APK. The test runner is unable to patch the class loader for the test apk though, and we this change is not going to change that.  So if you are testing your application with min sdk below 21, you will still have to make sure that the test apk contains a single DEX file.

Hello,

thank you for the explanation. But I still dont understand why did it work before for api <21, what broke it?

Will that be fixed?

Thank you for your time.

You might have been able to build a multidex test apk for api < 21, but I am pretty sure you were not able to run all tests on a device < 21 unless you've written your own test runner that patches the class loader. I'd like to avoid this runtime failure scenario. We should break the build until the support for legacy multidex test apk is added.

Hello,

If this is a decision from the Android team, could you at least give us instructions how to do this patching on our own. I am amazed that we did not see so much breaking problems up until now, but you are telling us that technically we had a bug/s that is now exposed with the change you guys did now.
Is there a way to add this manual patching by ourselves then, and how to do it?

Thx for the support.

There is a work in progress to add the support for the legacy multidex test apks, but I am not sure when it might land - https://android-review.googlesource.com/#/c/276561/.

I understand I will take a look at the commits and try to understand what was done there.
Is there a way to connect this bug to that commit and somehow get an ETA, we rely heavily on our instrumentation for a safety net and this is a show stopper for us so I need to relay and prioritize this information accordingly in the company.

To be honest the biggest worry for me here is that some of the classes you have there are package protected e.g. MultidexExtractor (from the first glance) so I would have to find a way to find proper list of dex files created from test apk, instrumentation and the app and sum them together in the test runner, not sure can I duplicate this behaviour but I will take a look in it in more detail starting next work week.

Hello, the merge strategy for commits you sent is: "Strategy    Merge if Necessary"
Could we link this bug to the commits and merge them in 2.4.0?

Thanks

ag/I6425e061a2d5e97e36c56fc01c009475af62c086 allows native multidex test apks. Legacy multidex tested app still requires mono-dex test apk. It should be released in the next canary release of the plugin.

@Mario The change is part of the multidex support library, and that is released as part of the SDK (not plugin nor Android Studio). Once the change is in, I will work on adding the support in the plugin. Unfortunately, I cannot provide any ETA. The owner of that change is aware of this bug (it is referenced from an internal bug report that is assigned to him).

So the TL;DR is:

1. using  native multidex, the fix will be available in the next gradle plugin releases (after 2.4.0-alpha7)
2. using legacy (i.e. support) multidex, we'll have to wait until a new release of com.android.support:multidex:1.0.x library

Do I have that right?

Yes, that is correct.

"We should break the build until the support for legacy multidex test apk is added."

This causes quite a bit of pain for my team. We currently have a test apk with more than 65k referenced methods (we're already running proguard on the test apk), and we're depending on it building properly. Our minSdkVersion is 18, but we only run our instrumentation tests on API 21+, so the old behavior allowed us to continue running our tests.

With the new behavior in AGP 3.0, our test apk fails to even build. It seems like my only options are:
1) Reduce the number of referenced methods
2) Create a new variant or flavor with minSdk 21+ and use that for running tests
3) Use minSdk 21 for all debug builds

The first option isn't really feasible, and the second is massively painful because it requires us to retrain 50+ devs on which gradle commands to use to run tests (since the commands change when adding a variant or flavor). The third option also isn't ideal because then we lose lint validation for our real minSdkVersion and lose the ability to easily run debug builds on older android versions manually.

Is there any way to keep the behavior from AGP 2.3.x for building test apks until a proper solution for legacy multidex is available?

@dr... for your specific situation, why not just stay on 2.3 until 2.4 (or 3.0) is out of alpha? Having 50 devs on alpha anything seems really iffy.

Yeah, definitely. We're not planning to upgrade until 3.0 is stable, but I'm worried that it will get to the stable channel without this being fixed. If legacy multidex support for test apks is ready before 3.0 is stable, then I don't have a problem :)

Hello,
since the multidex 1.0.2 is live I would expect that multidex works fine on api <21 for test apks, but this is not the case.
Did I misunderstand something when you said this will be backported to legacy multidex so test apk can contain multiple dex files also or?

classpath 'com.android.tools.build:gradle:3.0.0-alpha7'
multidex : '1.0.2',
Android Studio 3.0 Canary 7
Build #AI-171.4182969, built on July 14, 2017
JRE: 1.8.0_152-release-884-b01 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Mac OS X 10.12.4

Hello,
Thank you for the answer but I don't understand what is inside multidex 1.0.2 then.

in support lib Revision 25.4.0 it is stated:
Concurrent with this support library release, we are also releasing multidex version 1.0.2. This version includes the following important changes:
Allows multidexing of instrumentation APK.
Deprecates MultiDexTestRunner (AndroidJUnitRunner should be used instead).

We use multidex 1.0.2. Does this mean this fix will come out in 1.0.3, do you have any eta window on this topic?

Cheers