IssueTracker

Luke - am I right in thinking this already fixed in AGP at head?

Yes, all Android resource inputs can now only use relative path sensitivity, apart from custom resource directories located outside the project when data binding is enabled. The mentioned inputs in the stacktrace such as resourcesComputer.resources.fastdev.sourceDirectoriesAbsolute have been removed at head.

Is "head" == AGP 7.3.1? or 7.3.0?
Could you please look at git history to see which version the fix is in?

This change will be present in AGP 8.0.0-alpha01, since this changed happened very recently.

any chance it could be cherrypicked to 7.3.1 or 7.4? Would be nice to see better remote cache hit rates.

After making more upgrades to simplify-build-failure.sh it tells me that with Gradle 6.6.1, some other tasks can be not UP-TO-DATE too, include :publicDocsTask and :room:integration-tests:room-testapp-kotlin:mergeExtDexDebugAndroidTest

Potentially even more interesting is that with Gradle 6.6.1, when I run ./cleanBuild.sh -y :room:integration-tests:room-testapp-kotlin:mergeExtDexDebugAndroidTest, I get this failure:

Execution failed for task ':room:integration-tests:room-testapp-kotlin:mergeExtDexDebugAndroidTest'.
> Could not resolve all files for configuration ':room:integration-tests:room-testapp-kotlin:debugAndroidTestRuntimeClasspath'.
   > Failed to transform core-testing-2.0.1.aar (androidx.arch.core:core-testing:2.0.1) to match attributes {artifactType=android-dex, dexing-enable-desugaring=true, dexing-incremental-transform=false, dexing-is-debuggable=true, dexing-min-sdk=14, org.gradle.category=library, org.gradle.libraryelements=jar, org.gradle.status=release, org.gradle.usage=java-runtime}.
      > Could not resolve all files for configuration ':room:integration-tests:room-testapp-kotlin:debugAndroidTestRuntimeClasspath'.
         > Failed to transform core-common.jar (project :arch:core-common) to match attributes {artifactType=android-classes-jar, org.gradle.category=library, org.gradle.dependency.bundling=external, org.gradle.jvm.version=7, org.gradle.libraryelements=jar, org.gradle.usage=java-runtime}.
            > Execution failed for IdentityTransform: /usr/local/google/workspace/aosp-androidx-git/out/androidx/arch/core-common/build/libs/core-common-2.2.0-alpha01.jar.
               > File/directory does not exist: /usr/local/google/workspace/aosp-androidx-git/out/androidx/arch/core-common/build/libs/core-common-2.2.0-alpha01.jar

The stacktrace in comment #7 looks like https://b.corp.google.com/issues/149681812#comment14 , which seems to have been fixed by aosp/1245945 .

I don't think that :room:integration-tests:room-testapp-kotlin is doing anything weird but I'll see if I can find anything suspicious anyway.

If anybody else has any ideas about this, that would be super cool

From #7, it is interesting that during dexing of androidx.arch.core:core-testing:2.0.1 we need a project artifact project :arch:core-common (probably due to dependency resolution).

I'll try to create the same setup to reproduce.

Update: Filed this as https://issuetracker.google.com/168685872. This is an issue in AGP.

Thanks a bunch!

simplify-build-failure.sh made some more discoveries last night which seem to suggest that the clean failure in comment #7 and comment #9 is related to the out-of-date error in #1 :

When I run:

./cleanBuild.sh -y :room:integration-tests:room-testapp-kotlin:packageDebugAndroidTest :room:integration-tests:room-testapp-kotlin:createReleaseCompatibleScreenManifests :room:room-testing:processDebugAndroidTestResources -PverifyUpToDate

I get

Execution failed for task ':room:integration-tests:room-testapp-kotlin:mergeExtDexDebugAndroidTest'.
> Ran two consecutive builds of the same tasks, and in the second build, observed task ':room:integration-tests:room-testapp-kotlin:mergeExtDexDebugAndroidTest' to be not UP-TO-DATE. This indicates that task ':room:integration-tests:room-testapp-kotlin:mergeExtDexDebugAndroidTest' does not declare inputs and/or outputs correctly.
  
  You can check why Gradle executed :room:integration-tests:room-testapp-kotlin:mergeExtDexDebugAndroidTest by passing the '--info' flag to Gradle and then searching stdout for output generated immediately before the task began to execute.
  Our best guess for the reason that :room:integration-tests:room-testapp-kotlin:mergeExtDexDebugAndroidTest executed is below.
  :room:integration-tests:room-testapp-kotlin:mergeExtDexDebugAndroidTest declares 1 output files. This seems fine.
  :room:integration-tests:room-testapp-kotlin:mergeExtDexDebugAndroidTest declares 31 input files. The last modified input file is
  /usr/local/google/workspace/aosp-androidx-git/out/.gradle/caches/transforms-2/files-2.1/a4dc446a21cc97f68d8151240995e702/core-testing-2.0.1-runtime
  modified at Wed Sep 16 09:15:55 EDT 2020 (this build started at about Wed Sep 16 09:16:52 EDT 2020). This file is not declared as the output of any task in this build.
  To reproduce this error you can try running `./gradlew :room:integration-tests:room-testapp-kotlin:mergeExtDexDebugAndroidTest -PverifyUpToDate`

This is the upstream issue https://github.com/gradle/gradle/issues/14529

It is a regression in Gradle.

I tried gradle-6.7-20200923130742+0000 that gradle said has a fix for both https://github.com/gradle/gradle/issues/14529 and android lint hanging builds.

Sadly it still fails with:

Caused by: org.gradle.api.GradleException: Ran two consecutive builds of the same tasks, and in the second build, observed task ':docs-runner:dokkaKotlinTipOfTreeDocs' to be not UP-TO-DATE. This indicates that task ':docs-runner:dokkaKotlinTipOfTreeDocs' does not declare inputs and/or outputs correctly.

You can check why Gradle executed :docs-runner:dokkaKotlinTipOfTreeDocs by passing the '--info' flag to Gradle and then searching stdout for output generated immediately before the task began to execute.
Our best guess for the reason that :docs-runner:dokkaKotlinTipOfTreeDocs executed is below.
:docs-runner:dokkaKotlinTipOfTreeDocs declares 1 output files. This seems fine.
Added these 1 inputs: $CHECKOUT/prebuilts/androidx/internal/androidx/paging/paging-common-ktx/3.0.0-alpha02/paging-common-ktx-3.0.0-alpha02.jar
Removed these 1 inputs: $CHECKOUT/prebuilts/androidx/internal/androidx/paging/paging-common-ktx/3.0.0-alpha06/paging-common-ktx-3.0.0-alpha06.jar

To reproduce this error you can try running `./gradlew :docs-runner:dokkaKotlinTipOfTreeDocs -PverifyUpToDate`

	at androidx.build.uptodatedness.TaskUpToDateValidator$Companion$setup$1.execute(TaskUpToDateValidator.kt:143)
	at androidx.build.uptodatedness.TaskUpToDateValidator$Companion$setup$1.execute(TaskUpToDateValidator.kt:107)
	at org.gradle...
	at jdk.internal.reflect.GeneratedMethodAccessor744.invoke(Unknown Source)
	at java.base...	at com.sun.proxy.$Proxy74.afterExecute(Unknown Source)
	at org.gradle...

I've started running ./development/simplify-build-failure/simplify-build-failure.sh --task "buildOnServer -PverifyUpToDate" "to be not UP-TO-DATE" --num-jobs auto again with gradle-6.7-20200923130742+0000 from comment #12 to see if it makes any discoveries about which tasks or files are required

Progress so far from simplify-build-failure.sh is that with gradle-6.7-20200923130742+0000 from comment #12 , the not UP-TO-DATE error can reproduce by running only these tasks:

./gradlew --no-daemon -PverifyUpToDate :docs-fake:javaPreCompilePublicRelease :distpublicDocs :createDiffArchiveForAndroidxLeanback

Interestingly enough, when I add some logging into the simplification from #17, I find that the prebuiltSources function is getting called ( https://android.googlesource.com/platform/frameworks/support/+/1af52deacc1797cafa8ed5374cb597d74d037ed9/buildSrc/src/main/kotlin/androidx/build/DiffAndDocs.kt#311 ) more often in the second build than in the first build

/usr/local/google/workspace/2-aosp-androidx-git/jeff-work/work (androidx-master-dev)$ diff log.pre.proc log.post.proc
56a57,64
>  prebuiltSources searching for androidx.leanback:leanback:1.1.0-alpha04
>  prebuiltSources searching for androidx.leanback:leanback-paging:1.1.0-alpha04
>  prebuiltSources searching for androidx.leanback:leanback-tab:1.1.0-alpha04
>  prebuiltSources searching for androidx.paging:paging-rxjava2-ktx:3.0.0-alpha06
>  prebuiltSources searching for androidx.recyclerview:recyclerview:1.2.0-alpha05
>  prebuiltSources searching for androidx.room:room-common:2.3.0-alpha02
>  prebuiltSources searching for androidx.room:room-runtime:2.3.0-alpha02
>  prebuiltSources searching for androidx.room:room-rxjava2:2.3.0-alpha02

Also interesting is that if I remove the docs.provider({ wrapper around the call to prebuiltSources, then I (based on a sample size of about 2) get a successful build with gradle-6.7-20200923130742+0000-bin.zip, which could be a possible workaround if we need to our Gradle version soonish.

I haven't been able to narrow the cause down more closely than comment #18 yet (build seems to pass with nightly version of Gradle when we remove the docs.provider call in DiffAndDocs.kt )

Should I assume that the priority on this bug isn't high enough to warrant upgrading to a nightly version of Gradle but is high enough to keep investigating the root cause?

Do we know which nightly version was it fixed in? Does it work with 6.7-rc-3? That might be good enough in terms of stability.

Good idea, 6.7-rc-3 probably does have a fix for comment #9

Hmm, the intermittent failures to copy lint.jar have started blocking some other potential fixes for other things: in b/170117185 I encountered the lint.jar error when trying to increase memory: https://android-review.googlesource.com/c/platform/frameworks/support/+/1449135/1#message-26705f9d706ffaa5419a48eb7208c0e6025e0e24

Unfortunately we’re also encountering Gradle to hang sometimes with 6.7: https://dynamite-preprod.sandbox.google.com/room/AAAAW8qmCIs/ZbBCj1VlCak

I did end up building my own AGP for local testing that includes ag/12344143 and that was easy enough.

If the biggest reason for upgrading Gradle is to unblock upgrading AGP to get the fix for b/163039193 , I wonder if we should check in a locally built AGP in the meanwhile to help unblock other fixes

We want to have a variety of fixes from AGP (including studio fixes for plugin download). I don't like locally build prebuilts.

Hmm after doing git bisect run inside Gradle itself to identify the first commit in Gradle that causes our build to fail, it seems that the first commit that reproduces the ‘not UP-TO-DATE’ error in comment #12 is:

https://github.com/gradle/gradle/commit/167cc0186ff92b5428275788387c885c0752f54f

Detangle artifact transform nodes from `ResolveableArtifact` so that the initial set of files can be represented in some other way.

I haven’t yet figured out how this relates to the error that we’re seeing. I do notice that at https://github.com/gradle/gradle/commit/167cc0186ff92b5428275788387c885c0752f54f#diff-87e4b73f25d509e83ebc149351253c32R124 it looks like there can be a case where an external dependency might no longer be visited

gavra@ any thoughts on #25 and why https://github.com/gradle/gradle/commit/167cc0186ff92b5428275788387c885c0752f54f might cause us to have inconsistent jar classpaths

Interesting, removing the if in if (artifact.getId().getComponentIdentifier() instanceof ProjectComponentIdentifier) at https://github.com/gradle/gradle/commit/167cc0186ff92b5428275788387c885c0752f54f#diff-87e4b73f25d509e83ebc149351253c32R124

does seem to make our "not UP-TO-DATE" failure from comment #12 stop happening

This could be related to https://github.com/gradle/gradle/issues/14529 and the way project substitution works with artifact transforms. I'd add logging breakpoint to ArtifactBackedResolvedVariant.visitLocalArtifacts and collect some more data. That may provide us with more details to create a sample repro project.

It seems that 6.7-rc-5 is still having the same issue.

https://android-build.googleplex.com/builds/pending/P17231263/androidx/latest/view/logs/build_error.log

I think at this point, we should suppress docs tasks in our up to date check and just land the upgrade. We've been stuck on this for too long.

Have we double-checked that the generated documentation contains the relevant files? Another option could be to explicitly add the relevant dependency to make sure it shows up deterministically in the docs

Re Comment #28 and collecting more data

I found that in our build, (artifact.getId().getComponentIdentifier() instanceof ProjectComponentIdentifier) is always false, although I haven’t yet made progress on finding what other data to collect.

Re making a simpler reproduction test case, I’ve made a couple more improvements to simplify-build-failure.sh and it’s binary searching individual lines in buildSrc in hopes of identifying the problem more specifically

Unfortunately, the way that comment #35 seems to get around the problem is by preventing the documentation from being generated at all, so I guess we can't do that

I started to look into whether we can just exempt :publicDocsTask from out task-up-to-date validation, but then I found that lots of generateApi tasks were also having similar not-up-to-date problems

Uploaded a small reproduction case to https://github.com/mathjeff/gradle-samples/tree/master/inconsistent-dependency-resolution, if anybody has any ideas about it, that would be super cool

Also interesting is that while still on Gradle 6.5.1, we've managed to reproduce what looks like a similar error in the (new) docs-public project: https://android-build.googleplex.com/builds/submitted/6932771/androidx_multiplatform/latest/view/logs/build_error.log

Execution failed for task ':docs-public:dokkaKotlinDocs'.
> Ran two consecutive builds of the same tasks, and in the second build, observed task ':docs-public:dokkaKotlinDocs' to be not UP-TO-DATE. This indicates that task ':docs-public:dokkaKotlinDocs' does not declare inputs and/or outputs correctly.
  
  You can check why Gradle executed :docs-public:dokkaKotlinDocs by passing the '--info' flag to Gradle and then searching stdout for output generated immediately before the task began to execute.
  Our best guess for the reason that :docs-public:dokkaKotlinDocs executed is below.
  :docs-public:dokkaKotlinDocs declares 1 output files. This seems fine.
  Added these 1 inputs: $OUT_DIR/.gradle/caches/transforms-2/files-2.1/101df836280a896f8f52be7764015985/lifecycle-viewmodel-2.3.0-alpha06-runtime.jar
  Removed these 1 inputs: $OUT_DIR/.gradle/caches/transforms-2/files-2.1/638cf4bde7dce693a823d4a83475bac3/lifecycle-viewmodel-2.3.0-beta01-runtime.jar

which seems to support the idea that the new version of Gradle may just exacerbate some existing, infrequent problem

Interestingly enough, presubmit for what looks like an unrelated change: aosp/1478907 failed with a similar error:

https://android-build.googleplex.com/builds/pending/P17510639/androidx/latest/view/logs/build_error.log:

> Ran two consecutive builds of the same tasks, and in the second build, observed task ':docs-public:dokkaKotlinDocs' to be not UP-TO-DATE. This indicates that task ':docs-public:dokkaKotlinDocs' does not declare inputs and/or outputs correctly.
  
  You can check why Gradle executed :docs-public:dokkaKotlinDocs by passing the '--info' flag to Gradle and then searching stdout for output generated immediately before the task began to execute.
  Our best guess for the reason that :docs-public:dokkaKotlinDocs executed is below.
  :docs-public:dokkaKotlinDocs declares 1 output files. This seems fine.
  Added these 1 inputs: $OUT_DIR/.gradle/caches/transforms-2/files-2.1/1d60826cd92f73dde6c3a293aefb5e2f/paging-runtime-ktx-3.0.0-alpha08-runtime.jar
  Removed these 1 inputs: $OUT_DIR/.gradle/caches/transforms-2/files-2.1/4ec147f8f8ddd0a8b1e88e31fd38b4ce/paging-rxjava2-ktx-3.0.0-alpha08-runtime.jar

also interesting is that the artifact ids for the added and removed artifacts are different in this case: paging-runtime-ktx-3.0.0-alpha08-runtime.jar vs paging-rxjava2-ktx-3.0.0-alpha08-runtime.jar

It seems like the issue is still there. Should we except these docs tasks from up to date validation while we investigate?

Hmm it looks like comment #45 , aosp/1503711 , might have decreased the failure rate but not fully fixed the problem because https://android-build.googleplex.com/builds/submitted/7036988/androidx/latest/view/logs/build_error.log seems to have failed with the same error:

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':docs-public:doclavaDocs'.
> Ran two consecutive builds of the same tasks, and in the second build, observed task ':docs-public:doclavaDocs' to be not  UP-TO-DATE. This indicates that task ':docs-public:doclavaDocs' does not declare inputs and/or outputs correctly.
  
  You can check why Gradle executed :docs-public:doclavaDocs by passing the '--info' flag to Gradle and then searching stdout for output generated immediately before the task began to execute.
  Our best guess for the reason that :docs-public:doclavaDocs executed is below.
  :docs-public:doclavaDocs declares 1 output files. This seems fine.
  Added these 1 inputs: $GRADLE_USER_HOME/caches/transforms-2/files-2.1/4726cf7e6f295447a9ff3d934342d810/paging-rxjava2-ktx-3.0.0-alpha11-runtime.jar
  Removed these 1 inputs: $GRADLE_USER_HOME/caches/transforms-2/files-2.1/e978e225a2665a17da44820ffa23604a/paging-runtime-ktx-3.0.0-alpha11-runtime.jar
  
  To reproduce this error you can try running `./gradlew :docs-public:doclavaDocs -PverifyUpToDate`

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':docs-public:doclavaDocs'.
	at org.gradle...
Caused by: org.gradle.api.GradleException: Ran two consecutive builds of the same tasks, and in the second build, observed task ':docs-public:doclavaDocs' to be not  UP-TO-DATE. This indicates that task ':docs-public:doclavaDocs' does not declare inputs and/or outputs correctly.

You can check why Gradle executed :docs-public:doclavaDocs by passing the '--info' flag to Gradle and then searching stdout for output generated immediately before the task began to execute.
Our best guess for the reason that :docs-public:doclavaDocs executed is below.
:docs-public:doclavaDocs declares 1 output files. This seems fine.
Added these 1 inputs: $GRADLE_USER_HOME/caches/transforms-2/files-2.1/4726cf7e6f295447a9ff3d934342d810/paging-rxjava2-ktx-3.0.0-alpha11-runtime.jar
Removed these 1 inputs: $GRADLE_USER_HOME/caches/transforms-2/files-2.1/e978e225a2665a17da44820ffa23604a/paging-runtime-ktx-3.0.0-alpha11-runtime.jar

To reproduce this error you can try running `./gradlew :docs-public:doclavaDocs -PverifyUpToDate`

	at androidx.build.uptodatedness.TaskUpToDateValidator$Companion$setup$2.execute(TaskUpToDateValidator.kt:191)
	at androidx.build.uptodatedness.TaskUpToDateValidator$Companion$setup$2.execute(TaskUpToDateValidator.kt:138)
	at org.gradle...
	at jdk.internal.reflect.GeneratedMethodAccessor818.invoke(Unknown Source)
	at java.base...
	at com.sun.proxy.$Proxy77.afterExecute(Unknown Source)
	at org.gradle...

==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':docs-tip-of-tree:doclavaDocs'.
> Ran two consecutive builds of the same tasks, and in the second build, observed task ':docs-tip-of-tree:doclavaDocs' to be not  UP-TO-DATE. This indicates that task ':docs-tip-of-tree:doclavaDocs' does not declare inputs and/or outputs correctly.
  
  You can check why Gradle executed :docs-tip-of-tree:doclavaDocs by passing the '--info' flag to Gradle and then searching stdout for output generated immediately before the task began to execute.
  Our best guess for the reason that :docs-tip-of-tree:doclavaDocs executed is below.
  :docs-tip-of-tree:doclavaDocs declares 1 output files. This seems fine.
  Added these 1 inputs: $GRADLE_USER_HOME/caches/transforms-2/files-2.1/4726cf7e6f295447a9ff3d934342d810/paging-rxjava2-ktx-3.0.0-alpha11-runtime.jar
  Removed these 1 inputs: $GRADLE_USER_HOME/caches/transforms-2/files-2.1/e978e225a2665a17da44820ffa23604a/paging-runtime-ktx-3.0.0-alpha11-runtime.jar
  
  To reproduce this error you can try running `./gradlew :docs-tip-of-tree:doclavaDocs -PverifyUpToDate`

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':docs-tip-of-tree:doclavaDocs'.
	at org.gradle...
Caused by: org.gradle.api.GradleException: Ran two consecutive builds of the same tasks, and in the second build, observed task ':docs-tip-of-tree:doclavaDocs' to be not  UP-TO-DATE. This indicates that task ':docs-tip-of-tree:doclavaDocs' does not declare inputs and/or outputs correctly.

You can check why Gradle executed :docs-tip-of-tree:doclavaDocs by passing the '--info' flag to Gradle and then searching stdout for output generated immediately before the task began to execute.
Our best guess for the reason that :docs-tip-of-tree:doclavaDocs executed is below.
:docs-tip-of-tree:doclavaDocs declares 1 output files. This seems fine.
Added these 1 inputs: $GRADLE_USER_HOME/caches/transforms-2/files-2.1/4726cf7e6f295447a9ff3d934342d810/paging-rxjava2-ktx-3.0.0-alpha11-runtime.jar
Removed these 1 inputs: $GRADLE_USER_HOME/caches/transforms-2/files-2.1/e978e225a2665a17da44820ffa23604a/paging-runtime-ktx-3.0.0-alpha11-runtime.jar

It’s interesting that I noticed this (about 5 days) after aosp/1524922 was submitted , as if some parallelism within a project is a prerequisite for reproducing the problem