IssueTracker

Thanks, Hung! I'll include this in the release notes for Canary 11, as C10 already released early last week :)

Also, I moved this out of a public bug component.

Several users from reddit (https://www.reddit.com/r/androiddev/comments/9h76nu/android_studio_33_canary_11_available/e6abpzh/, https://www.reddit.com/r/androiddev/comments/9h76nu/android_studio_33_canary_11_available/e6aydtj/) would like to know exactly how the feature works and whether it takes effect in their projects. Below I will try to give a more comprehensive description of this feature:

Background
----------------
Conventionally, in Java-only projects, Java compilation and annotation processing is done by the AndroidJavaCompile task (which appears in the users' console as ":module:compile[Variant]JavaWithJavac").

If the user starts using the kotlin-android plugin to write Kotlin code, then the KotlinCompile task will compile Kotlin code in addition to AndroidJavaCompile compiling Java code, and AndroidJavaCompile still takes care of annotation processing.

If both cases above, because AndroidJavaCompile needs to do annotation processing, it will not be incremental if non-incremental annotation processors are present. Note that KotlinCompile, if used, is always incremental.


If the user starts using the kotlin-kapt plugin, then Kapt takes over annotation processing, and AndroidJavaCompile compiles Java code only. AndroidJavaCompile will therefore be incremental as it does not have to do annotation processing. (If it is observed to be non-incremental, then it is either a bug that has been fixed or should be fixed soon.) Kapt, however, is currently not incremental and will require a longer-term effort to be made incremental.


Goal
----------------
The goal of the new feature is to make AndroidJavaCompile incremental when Kapt is not used and non-incremental annotation processors are present. (If Kapt is used or all of the annotation processors used are incremental, then AndroidJavaCompile is already incremental without the need for the new feature.)


The new android.enableSeparateAnnotationProcessing feature
------------------------------------------------------------------------------------
In the new feature, the Android Gradle plugin first determines whether a new separate annotation processing task is needed (see below). If so, it creates a ProcessAnnotationsTask that does annotation processing only (this task appears in the users' console as ":module:process[Variant]AnnotationsWithJavac"). And because AndroidJavaCompile now does not have to do annotation processing, it will be incremental.

ProcessAnnotationsTask is used (i.e., the feature takes effect) if all of the following conditions are met:
    1. Kapt is not used
    2. Incremental compilation is not disabled by the user through the DSL
    3. Not all of the annotation processors are incremental
    4. The android.enableSeparateAnnotationProcessing flag is enabled


Incrementality of tasks
------------------------------
When Kapt is used (e.g., in most Kotlin-only or hybrid Kotlin-Java projects):
    + ProcessAnnotationsTask is not created.
    + KaptTask performs annotation processing only, without compiling. KaptTask is not incremental (yet).
    + AndroidJavaCompile and KotlinCompile perform compilation only, without annotation processing. Both tasks are incremental.

When Kapt is not used, (e.g., in Java-only projects):
    + If ProcessAnnotationsTask is needed (see above for when it is needed), ProcessAnnotationsTask first performs annotation processing only, and [AndroidJavaCompile] then performs compilation only. ProcessAnnotationsTask is not incremental (yet), and AndroidJavaCompile will be incremental.
    + Otherwise, ProcessAnnotationsTask is either not created or skipped. AndroidJavaCompile performs both annotation processing and compilation. It will be incremental or not depending on whether all of the annotation processors used are incremental or not.

Known Issue with AGP 3.3.0-alpha11
--------------------------------------------------
(This was first reported at https://www.reddit.com/r/androiddev/comments/9h76nu/android_studio_33_canary_11_available/e69psq5/, Internal issue 116170968.)

If the users enable the android.enableSeparateAnnotationProcessing flag in cases when ProcessAnnotationsTask is not needed (i.e., when the feature wouldn't have any real effects---see comment #4), then currently the build may fail with this error:
        > lateinit property processorListFile has not been initialized```

In the next release, we will make sure that the build doesn't fail in such cases.

In AGP 3.3.0-alpha12 I'm having a problem now where processAnnotationsWithJavac trips over not seeing generated code (from the protobuf plugin) when android.enableSeparateAnnotationProcessing is set.

Could you share a small project that reproduces this issue?

I've been unable to reproduce this with a small project, I suspect it's due to build timing/order. Maybe it helps to look at the build output? I've attached the outputs from the failing build, and the output from the small project that does not fail.

The passing test project doesn't have the flag enabled, can you retry with the flag enabled?

Oops. Turns out the very last thing I had tried before I gave up actually did trigger the problem, I just forgot to flip the switch. Here's the test project, I minimized it some more since I saved the logs earlier: https://github.com/bubenheimer/sepannprocprotobug

The problem is triggered by having both the protobuf plugin and the Dagger annotation processor in the same module, even without having any Dagger-related code in the module.

Thank you for providing a project!

Since the project doesn't have any Dagger-related code, Dagger does not actually run at all. Therefore, the processAnnotationsWithJavac task immediately proceeds to compilation, whereas it was meant to do annotation processing only. This is a known issue of the JDK Java compiler (https://bugs.openjdk.java.net/browse/JDK-6378728). To work around this, you will need to comment out Dagger from the project:
        //annotationProcessor 'com.google.dagger:dagger-compiler:2.17'

Does your actual project need annotation processing at all (besides Protobuf)? If so, could you provide another project having code that requires annotation processing?

The actual project uses plenty of Dagger. I've added a bit of Dagger code to the test project and the error remains the same.

The main thing that stands about the module that has errors is the protobuf plugin. Virtually all my code uses Dagger, but this is the only one with protobuf.

Thanks, I can reproduce the error now. I'll work on a fix.

I've filed Issue 117161463 for the error you're seeing. The fix will be available in AGP 3.3.0-beta01 and 3.4.0-alpha01 (Change-Id: Ief66555be7a74af50c896d11071c081c5b29acd1).

Thanks again for the report!

Thank you!

I noticed that android.enableSeparateAnnotationProcessing=true appears to have the glorious side effect of eliminating compiler crashes for errorprone: https://github.com/google/error-prone/issues/771

@17: I'm not sure how that works, but wouldn't recommend this feature as a fix for the errorprone bug.

Update: Starting with 3.3.0-beta04 and 3.4.0-alpha04, ProcessAnnotationsTask is always executed even if all annotation processors are incremental. (Before, if all annotation processors are incremental, we would skip ProcessAnnotationsTask and let the AndroidJavaCompile task perform compilation and annotation processing incrementally in the same task.)

Change-Id: Icafd28c2e8dadce44cb27cd27b253d480bd27c04

I am now facing the issue that annotations are not (re-) processed in a module if the build for that module failed previously. I see this happening specifically with Dagger 2.19. So, if the build for my project failed in module A, and then I do an incremental build later, there is a good chance that the Dagger annotations in module A are not (re-)processed, and the build fails due to missing generated Dagger classes. I don't know if that's happening because Dagger recently started supporting incremental annotation processing (and I'm enabling it), or because of the recent changes described in this issue, or because of how these two interact. I started seeing this issue with AGP 3.3.0-beta02, and it's still present in 3.3.0-beta03. I'm still using android.enableSeparateAnnotationProcessing on all modules due to other libraries that do not yet support incremental annotation processing.

Sorry if this is too vague, I'm hoping that it's enough information to get a handle on it. I tried to narrow it down earlier, but ran into trouble reproducing it predictably.

If you can reproduce it consistently, please attach a project and I will take a look.

Btw, I am curious how the feature is working for you. Could you share some numbers with and without the feature?

You're special-casing Lombok, but should also special-case javac compiler plugins (such as Error Prone): https://github.com/tbroyer/gradle-errorprone-plugin/issues/19

When Lombok is not used, and you pass -proc:none, you should still configure the annotationProcessorPath so that -Xplugin plugins can be found (this would require Gradle 5.0 at a minimum to work correctly though: https://github.com/gradle/gradle/issues/6573).
When Lombok is used, I would suggest using `-processor lombok.launch.AnnotationProcessorHider$AnnotationProcessor` (https://projectlombok.org/contributing/lombok-execution-path) rather than filtering the processorpath.

Disclaimer: I'm not an Android developer, the above information comes from reading the sources of the Android Plugin for Gradle 3.3.0-rc03.

+1 to #22. I have a repro example here:

https://github.com/uber/AutoDispose/pull/330

Add `android.enableSeparateAnnotationProcessing=true` to the root gradle.properties and should repro with `./gradlew build`

I've been unable to use Error Prone since the start of the year due to the issue in comment#22

Thanks a lot for the suggestions! I've filed Issue 130531986 and I'm working on a fix.

FYI, we have just fixed Issue 130531986, please see that bug for more details.

Update: We have decided to remove the android.enableSeparateAnnotationProcessing feature in AGP 4.0.0-alpha04+ (internal Issue 142515190) as the most popular annotation processors are now incremental, and we encourage authors of any non-incremental APs to make theirs APs incremental.

Release notes: https://androidstudio.googleblog.com/2019/11/android-studio-40-canary-4-available.html

Thank you for all the inputs in this bug!

I wish this could be re-added as it's the only way I know to work around this errorprone-related crash bug in the Java compiler (as I mentioned in #17); cannot use errorprone without enableSeparateAnnotationProcessing: https://github.com/google/error-prone/issues/771

Can you try applying the Kotlin Kapt plugin, it should provide a similar effect of separating annotation processing from Java compilation (with some performance overhead though, I'm not sure how much for your project)?

The project does not use Kotlin; Kotlin dependencies are externalized for performance and stability reasons.

You can still use Kapt for Java-only project. It is not efficient, but it is the only equivalence of this feature that I can think of.

I guess the root cause is still the error-prone bug, so perhaps the better way out is for that one to get fixed?

Thanks, the error-prone bug looks unlikely to be fixed as they blame the Java compiler, which also looks unlikely to be fixed.

Does it add much complexity to the AGP code to keep the flag around? I'd imagine the flag would be helpful for troubleshooting and working around other, future build problems. Android builds are complex, so having the ability to remove complexity when things break seems a good idea to me. I have to do it constantly, because I don't work on toy projects.

This feature was introduced to help with the transition when annotation processors are updated to be incremental. Now that the most common ones are incremental, we need to remove the feature to avoid cluttering our code base and having to maintain and test multiple code paths, allowing room for other new features.

If this feature is a bug fix for the AGP, then sure we'll have to keep it. But here it is an accidental workaround for another tool's bug, so there isn't a strong enough justification to keep it.

Maybe you can ask the error-prone authors for a workaround at least? E.g., would swallowing the NPE be an acceptable first step?