IssueTracker

Our builds do contain native libs from 3rd party libs. Sorry about that. They do not contain native code from us though.

We also tested the same benchmarks with AGP 4.2.0-beta01 and see the same issue.

It looks like using --rerun-tasks cannot be used to reproduce the issue: then the problem goes away. To reproduce, I've used ./gradlew clean <task> --no-build-cache. But the issue showed up with normal cached executions.

A quick question about the diff shown in image Thu Dec 10 2020 15:49:18 GMT-0800 (Pacific Standard Time).png, what changed in the build? Was it build without any changes?

Also, just to clarify do you expect mergeDebugNativeLibs to be FROM-CACHE rather than up-to-date? Because you are running clean task, its outputs will be removed and it cannot be up-to-date.

If you manually invoke the tasks and you clean, then we'd expect the result to be FROM-CACHE. However, in the benchmarks we don't clean anything and there I'd expect the task to be UP-TO-DATE and in some instances it is. The benchmark scenario looks like this:

incremental_build_abi_change_in_account_module_spos {
    tasks = [":..:assembleDebug"]
    apply-abi-change-to = "common/account-service/public/src/main/java/com../CreateBody.java"
    apply-abi-change-to = "common/account-service/public/src/main/java/com..CreateResponse.java"
    apply-abi-change-to = "common/account-service/public/src/main/java/com/..AccountStatusStandardResponse.kt"
    show-build-cache-size = true
    warm-ups = 4
    gradle-args = ["--offline", "--no-build-cache"]
}

The screenshot is from the benchmark scenario.

Note 2: when running with --rerun-tasks we can see that the task always run fast.

Roughly how long does the task take with --rerun-tasks?

We can see that the task mergeDebugNativeLibs has a random up-to-date status and when it runs, it takes around 160 seconds.

Roughly how long does the task take when it's UP-TO-DATE?

My suspicion is that the random UP-TO-DATE status is because the externalNativeLibs task input is annotated with @Classpath which is sensitive to the order of files, and perhaps the order isn't deterministic. In Thu Dec 10 2020 15:49:18 GMT-0800 (Pacific Standard Time).png, it says some files were removed and added, but I suspect that all of those files were both removed and added (looks like Gradle only lists the first 3 reasons), because that is how Gradle reports changes in the order of files (see here for example).

Roughly how long does the task take with --rerun-tasks?

Usually only a few seconds, around 3-5 I think.

Roughly how long does the task take when it's UP-TO-DATE?

It's very fast. The last build scan reported 0.032s.

I think we have 2 separate issues here:

• if classpath input for mergeDebugNativeLibs changes order, this task may take too long. We should look into that. I've asked Stéphane to share some CPU snapshots if possible.
• classpath order changes between builds. This impacts mergeDebugNativeLibs, but Stéphane shared that this also impacts other tasks e.g kaptGenerateStubsDebugKotlin (classpath file input property), desugarDebugFileDependencies (classpath property), compileDebugJavaWithJavac (classpath input property), dexBuilderDebug (dexParam.desugarClasspath input property).

I've asked Stéphane to share some CPU snapshots if possible.

Yes, some CPU snapshots would be very helpful

Here is a snapshot obtained via Gradle Profiler for the same scenario as the screenshots (incremental_build_abi_change_in_account_module_spos). I will provide the whole folder to @gavra via slack.

Thanks! Looks like the issue is FileUtils.isFileInDirectory. I have a WIP change from a couple months ago (ChangeId Ia88494c0b353545bf1ea3035ef8f165c1f5923e4) to improve the performance of this method, but I never merged it :/

Is there any chance this could be ported to a 4.1 fix release or 4.2 ? I would be happy to test a branch / build of AGP for this (through Ivan ?)

I was able to repro this issue by creating an app project with dependencies on 500 different AARs with .so files.

When I shuffled the order of the AAR dependencies in the build.gradle file to simulate the nondeterministic ordering of the externalLibNativeLibs task input, the mergeDebugNativeLibs task takes about 50 seconds.

One possible workaround for this issue would be to use @InputFiles instead of @Classpath in MergeNativeLibsTask.

Re #9, is there a bug filed for the 2nd issue, Ivan?

Note that a bug report has been open on the Gradle repo as well as it seems that the cache key computation issue is more general than the mergeDebugNativeLibs task : https://github.com/gradle/gradle/issues/15626

To summarize: mergeDebugNativeLibs seems to have a performance issue in incremental mode and this is the focus of this report, while the gradle ticket focuses on the reproducibility of the cache key computation for a few classes that see their inputs order shuffled.

Scott, can we try to optimize an incremental scenario with many changes, and if the cost is still too high, we can fallback to clean run if number of changed files is greater than some threshold? WRT other issue (classpath ordering), I've just filled https://issuetracker.google.com/176077101.

Discussed offline. Current plan is the following:

• In 4.2, run the task non-incrementally if the number of changed inputs is above a certain threshold
• In 7.0, fix the performance issue for incremental runs

4.2 beta 4 has the fix (workaround) for this. It's slated for release on Monday, Jan 11.

Thank you very much for the update @sp, we're eager to try it out.

Confirmed, it seems to be fixed in AGP 4.2 beta 04. Thx you very much for all your work!