Outputs of CompileLibraryResourcesTask and AarResourcesCompilerTransform are not relocatable [278255388]

Assigned

Bug

Status Update

No update yet.

Description

hu...@google.com

created issue #1

Apr 14, 2023 04:05PM

This issue came up when we investigated another issue (https://issuetracker.google.com/277978832 - Cache misses for tasks PackageForUnitTest and AndroidUnitTest).

Tested Version

AGP 7.4.1 and AGP @ HEAD

Steps to Reproduce

Download and build the sample project from one location:

cd ~/Desktop
mkdir ProjectCopy1 && cd ProjectCopy1
git clone https://github.com/cdsap/ReproducerIssueAGP712TestTask
cd ReproducerIssueAGP712TestTask
git checkout 9674abe95bbd8d5b016a1e5fd6968dfd67d10db0 // to use AGP 7.4.1
./gradlew clean :library:compileDebugLibraryResources

Copy the project to another location to test its relocatability:

cd ~/Desktop
cp -r ProjectCopy1 ProjectCopy2 && cd ProjectCopy2/ReproducerIssueAGP712TestTask
./gradlew clean :library:compileDebugLibraryResources

Compare the 2 outputs:

cd ~/Desktop
diff -r -q ProjectCopy1 ProjectCopy2

We get the following differences:

Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/drawable_ic_launcher_background.xml.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/drawable_ic_launcher_background.xml.flat differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/drawable-v24_ic_launcher_foreground.xml.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/drawable-v24_ic_launcher_foreground.xml.flat differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/layout_activity_main.xml.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/layout_activity_main.xml.flat differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-anydpi-v26_ic_launcher_round.xml.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-anydpi-v26_ic_launcher_round.xml.flat differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-anydpi-v26_ic_launcher.xml.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-anydpi-v26_ic_launcher.xml.flat differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-anydpi-v33_ic_launcher.xml.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-anydpi-v33_ic_launcher.xml.flat differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-hdpi-v4_ic_launcher_round.webp.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-hdpi-v4_ic_launcher_round.webp.flat differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-hdpi-v4_ic_launcher.webp.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-hdpi-v4_ic_launcher.webp.flat differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-mdpi-v4_ic_launcher_round.webp.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-mdpi-v4_ic_launcher_round.webp.flat differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-mdpi-v4_ic_launcher.webp.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-mdpi-v4_ic_launcher.webp.flat differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-xhdpi-v4_ic_launcher_round.webp.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-xhdpi-v4_ic_launcher_round.webp.flat differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-xhdpi-v4_ic_launcher.webp.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-xhdpi-v4_ic_launcher.webp.flat differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-xxhdpi-v4_ic_launcher_round.webp.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-xxhdpi-v4_ic_launcher_round.webp.flat differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-xxhdpi-v4_ic_launcher.webp.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-xxhdpi-v4_ic_launcher.webp.flat differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-xxxhdpi-v4_ic_launcher_round.webp.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-xxxhdpi-v4_ic_launcher_round.webp.flat differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-xxxhdpi-v4_ic_launcher.webp.flat and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/compiled_local_resources/debug/out/mipmap-xxxhdpi-v4_ic_launcher.webp.flat differ

Side note: The following files also differ (they contain absolute paths), but I'm guessing that it's intended (pls correct me otherwise).

Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/incremental/debug/packageDebugResources/compile-file-map.properties and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/incremental/debug/packageDebugResources/compile-file-map.properties differ
Files ProjectCopy1/ReproducerIssueAGP712TestTask/library/build/intermediates/incremental/debug/packageDebugResources/merger.xml and ProjectCopy2/ReproducerIssueAGP712TestTask/library/build/intermediates/incremental/debug/packageDebugResources/merger.xml differ

This shows that the outputs of :library:compileDebugLibraryResources (CompileLibraryResourcesTask) can change depending on the project location, which means that downstream tasks consuming these outputs will have cache misses.

Note: The AarResourcesCompilerTransform to compile resources from external dependencies has this issue too, as shown in the build scan provided at https://issuetracker.google.com/277978832#comment3, bullet 1. It probably has the same root cause. To trigger this transform, delete ~/.gradle/caches/transforms-3 and run ./gradlew :library:processDebugUnitTestResources.

Potential Root Cause

com.android.aaptcompiler.ResourceFile contains an absolute path (com.android.aaptcompiler.Source), when tested with the above sample project. This absolute path will probably impact the output (see serializeCompiledFileToPb in com.android.aaptcompiler.Container#addFileEntryImpl).

Comments

lu...@google.com <lu...@google.com> #2Apr 14, 2023 04:15PM

Weird "? extends" declarations floating in the middle of nowhere?

hu...@google.com <hu...@google.com> #3Apr 14, 2023 06:55PM

Sure, I'm attaching 2 of the .flat files.

ProjectCopy1 - drawable_ic_launcher_background.xml.flat

11 KB

Download

ProjectCopy2 - drawable_ic_launcher_background.xml.flat

11 KB

Download

lu...@google.com <lu...@google.com> #4Apr 17, 2023 08:57AM

Thanks Hung! Taking a look at the two compiled resources, they both encode the resource source file as absolute path. The difference being a/b/c/d/ProjectCopy1/my/resource.xml and a/b/c/d/ProjectCopy2/my/resource.xml.

While, relative path support in compiled resources covers most cases, there were some where we fallback to the absolute path e.g for resource processing, dynamic features. So this behaviour isn't entirely unexpected.

However, I'll take a look into why the absolute path is being used for these resources and see how much effort is wil be to use relative resource paths.

hu...@google.com <hu...@google.com> #5Apr 17, 2023 11:10AM

(Updating to P1 to match Issue 277978832, as discussed in the standup.)

lu...@google.com <lu...@google.com> #6Oct 18, 2023 11:13AM

Moving this issue to P2 as this was a known fallback case from enabling relative resources and is more performance related than an issue.

hu...@google.com <hu...@google.com> #7Oct 18, 2023 11:33AM

Just a note: Files containing absolute paths will likely cause correctness issues (not just performance issues), because when these files are copied from the Gradle build cache over to a different project, these paths can't be used on that project.

st...@warting.se <st...@warting.se> #8Apr 20, 2024 09:11PM

This is still an issue in agp 8.3.2.
I have reproduced it here:

https://github.com/warting/relocatable-build-cacheble
And here is the result:

https://github.com/warting/relocatable-build-cacheble/actions/runs/8767748521/job/24061358184#step:6:14

lu...@google.com <lu...@google.com> Apr 23, 2024 04:34PM

Assigned to lu...@google.com.