Find a way to determine whether a third-party library should be jetified [119135578]

Fixed

Bug

Reviewed (L1)

Status Update

No update yet.

Description

hu...@google.com

created issue #1

Nov 6, 2018 10:45PM

Jetifier is supposed to work on third-party libraries that do not yet support AndroidX. However, if a library has already been updated to support AndroidX, or if it is not using AndroidX/support libraries at all, then it is not necessary to run Jetifier to convert it, and doing so may even result in incorrect rewritten binaries.

Example failures:

(1)

http://issuetracker.google.com/118658603#comment1
Execution failed for task ':app:kaptGenerateStubsDebugKotlin'.
> Could not resolve all files for configuration ':app:kapt'.
> Failed to transform file 'butterknife-compiler-9.0.0-SNAPSHOT.jar' to match attributes {artifactType=processed-jar} using transform JetifyTransform
> Failed to transform '...\.gradle\caches\modules-2\files-2.1\com.jakewharton\butterknife-compiler\9.0.0-SNAPSHOT\d5efd0795737d4523044335a7faa82b0e3d984c9\butterknife-compiler-9.0.0-SNAPSHOT.jar' using Jetifier.
Reason: The given artifact contains a string literal with a package reference 'android.support.v4.content' that cannot be safely rewritten. Libraries using reflection such as annotation processors need to be updated manually to add support for androidx.

(2)

http://issuetracker.google.com/113224601#comment13
Execution failed for task ':app:kaptGenerateStubsDebugKotlin'.
> Could not resolve all files for configuration ':app:kapt'.
> Failed to transform file 'jetifier-core-1.0.0-alpha10.jar' to match attributes {artifactType=processed-jar} using transform JetifyTransform
> Failed to transform '/Users/nkotula/.gradle/caches/modules-2/files-2.1/com.android.tools.build.jetifier/jetifier-core/1.0.0-alpha10/9eb7027c383061de12f93aae7a22cbeb97832d2a/jetifier-core-1.0.0-alpha10.jar' using Jetifier. Reason: The given artifact contains a string literal with a package reference 'android/support/v4' that cannot be safely rewritten. Libraries using reflection such as annotation processors need to be updated manually to add support for androidx.

(In this example, Jetifier even tries to jetify itself!)

(3)

http://issuetracker.google.com/113880420
Execution failed for task ':app:kaptGenerateStubsDebugKotlin'.
> Could not resolve all files for configuration ':app:kapt'.
> Failed to transform file 'kotlin-compiler-embeddable-1.2.21.jar' to match attributes {artifactType=processed-jar} using transform JetifyTransform
> Failed to transform '/root/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-compiler-embeddable/1.2.21/39456b64a42dc359e385697e2e93b5cba52e197f/kotlin-compiler-embeddable-1.2.21.jar' using Jetifier.
Reason: Malformed input or input contains unmappable characters: javaslang/?.class.

(In this example, kotlin-compiler-embeddable-1.2.21.jar does not need to be jetified but was jetified and then failed.)

We need a way to determine whether a third-party library should be jetified or not. There are a couple of approaches:
1. The library adds some metadata stating that it is AndroidX-ready and therefore should not be jetified. (The Android Gradle plugin can also help to add this information automatically for libraries that are built using the AGP.)
2. Jetifier automatically detects the presence of AndroidX (e.g., import statements and strings containing references to AndroidX), and if AndroidX is present, assumes that the library is AndroidX-ready and skips converting it.
3. The users need to specify a blacklist of libraries that should not be jetified.

We'll need to weigh the pros and cons of these approaches and select the best one (or a combination of them).

Comments

xa...@google.com <xa...@google.com> #2Nov 7, 2018 12:04AM

There are a few more changes to DataStoreImpl's logic which makes a reevaluation necessary.

Now with Yigit's changes, datastore's cold flow always tries to do a dirty read by readState which calls readAndInitOrPropagateAndThrowFailure that invokes readDataOrHandleCorruption only in InitDataStore's first run. Any following read is triggered by IPC's listener which calls readDataAndUpdateCache which updates the cache to ReadException returned from attemptRead.

A ReadException is not automatically handled and stays in cache until a updateData call which revisits readAndInitOrPropagateAndThrowFailure, but it doesn't handle the corruption exception because it's already not the first run. In such case, the handleUpdate call proceeds to transformAndWrite which does not catch the exceptions thrown from readDataFromFileOrDefault.

pa...@google.com <pa...@google.com> #3Nov 8, 2018 06:55PM

can we create a test failure for this? I'm not fully sure what is going wrong here, it is a bit hard to track w/o a repro case.

Message last modified on Nov 8, 2018 06:56PM

hu...@google.com <hu...@google.com> #4Nov 8, 2018 06:59PM

can we create a test failure for this?

Sure I'm working on the code changes.

In the meantime I think this bug might makes the errors e.g. in b/309382591 harder to recover because it doesn't try to handle the corruption until the app restarts so the DS can readAndInit again (with corruption handler logic).

hu...@google.com <hu...@google.com> #5Nov 18, 2018 05:31AM

Project: platform/frameworks/support
Branch: androidx-main

commit 383a67d0e5a9b35e394b212db35bdee892b167fb
Author: Zhiyuan Wang <zhiyuanwang@google.com>
Date: Wed Feb 21 17:04:40 2024

Add tests to verify that CorruptionException is not properly handled after DataStore's first disk read. If the internal state is [ReadException], it will not recover for the rest of the lifecycle.

Fix in the follow up change.

Bug: 289582516
Test: ./gradlew :datastore:datastore-core:jvmTest
Change-Id: Ie77900b5c2cb8fb8e55d4c40364491dea9250784

M datastore/datastore-core/src/commonTest/kotlin/androidx/datastore/core/SingleProcessDataStoreTest.kt

https://android-review.googlesource.com/2972402

[Deleted User] <[Deleted User]> #6Nov 22, 2018 11:47PM

Project: platform/frameworks/support
Branch: androidx-main

commit 70085fee2af278bb8c3e45e1acfcf3f4cbb05c5a
Author: Zhiyuan Wang <zhiyuanwang@google.com>
Date: Thu Feb 22 15:08:34 2024

DataStore handles CorruptionException for all reads and writes.

Before this change, DataStore didn't handle the CorruptionException for both the reads and writes after initialization.

Fixes: 289582516
Test: ./gradlew :datastore:datastore-core:jvmTest
Change-Id: Icb07b140017b62a905d3626f82aeb993f636c053

M datastore/datastore-core/src/commonMain/kotlin/androidx/datastore/core/DataStoreImpl.kt
M datastore/datastore-core/src/commonTest/kotlin/androidx/datastore/core/SingleProcessDataStoreTest.kt

https://android-review.googlesource.com/2974198

hu...@google.com <hu...@google.com> #7Dec 4, 2018 04:49PM

The following release(s) address this bug.It is possible this bug has only been partially addressed:

androidx.datastore:datastore-core:1.1.0-beta02
androidx.datastore:datastore-core-android:1.1.0-beta02
androidx.datastore:datastore-core-iosarm64:1.1.0-beta02
androidx.datastore:datastore-core-iossimulatorarm64:1.1.0-beta02
androidx.datastore:datastore-core-iosx64:1.1.0-beta02
androidx.datastore:datastore-core-jvm:1.1.0-beta02
androidx.datastore:datastore-core-linuxx64:1.1.0-beta02
androidx.datastore:datastore-core-macosarm64:1.1.0-beta02
androidx.datastore:datastore-core-macosx64:1.1.0-beta02

jt...@temp.thunderhead.com <jt...@temp.thunderhead.com> #8Dec 7, 2018 03:30PM

I am interested in the ability to blacklist dependencies from being jettified as well. The binary rewrite messes with AspectJ aspect classes. It re-introduces this problem:

https://issuetracker.google.com/issues/119153999

ms...@gmail.com <ms...@gmail.com> #9Dec 14, 2018 06:58PM

I've succesfully tested the blacklist config with:
* AGP 3.3.0-rc01
* Gradle 5.1-rc-1

I just wanted to add a comment about what I had to blacklist, so that perhaps jetifier can handle this automatically without a rule.

TL;DR: jetifier should probably not even try to open *.so files.

Details:
My project depends (transitively) on a jar which contains only native *.so files.
On one particular machine (docker container) with apparently not enough memory, I was getting this error:

Execution failed for task ':libon-lite-client:checkDebugClasspath'.
> Could not resolve all files for configuration ':libon-lite-client:debugCompileClasspath'.
> Failed to transform artifact 'liblinphone-debug-symbols.jar (org.liblinphone:liblinphone:4.0.1)' to match attributes {artifactType=android-classes, org.gradle.usage=java-runtime-jars}
> Execution failed for JetifyTransform: /root/.gradle/caches/modules-2/files-2.1/org.liblinphone/liblinphone/4.0.1/871c6e76e93a7fa1bbafb5eb450e45c2330e866/liblinphone-4.0.1-debug-symbols.jar.
> Java heap space

(Note that with gradle 4.10.2/AGP 3.2.1, I didn't have this detailed info, but just simply "Transformation hasn’t been executed yet" (

https://issuetracker.google.com/issues/119951660). I got this additional error message with gradle 5.1-rc-1/AGP 3.2.1.)

Now with gradle 5.1-rc1/AGP-3.3.0-rc01, I have this rule:
android.jetifier.blacklist=liblinphone-4.0.1-debug-symbols.jar

And I no longer get the error.

I suppose jetifier shouldn't even try to look into binary *.so files.

pa...@google.com <pa...@google.com> #10Dec 18, 2018 12:17PM

#comment9

:
Jetifier ignores *.so files. However it has to extract each archive to the memory and check each file and recreate a new zip file to the filesystem. So loading the archive to the memory has to be done no matter if there are .so files or not.

hu...@google.com <hu...@google.com> #11Dec 18, 2018 02:05PM

@Filip: I wonder if Jetifier should check the zip entries' meta data first (without loading their contents) and return immediately if there are no entries that need jetifying. (Returning is good enough, Jetifier doesn't need to copy the file to somewhere else. And even when it needs to copy some large entries from one zip file to another, we can probably use streams/Zip64 so that it doesn't load the entire zip file/zip entry into memory---see

Issue 113559507

st...@gmail.com <st...@gmail.com> #12Jan 3, 2019 10:46PM

Is anyone else having issues with AGP 3.3.0-rc01+ or 3.4.0-alpha04+ where the project compiles and runs fine, but cannot resolve the R file? I tried invalidating android studio caches, cleaning, deleting .idea, deleting .gradle, and nothing works except reverting to android plugin 3.2.1...which I can't do because I need the jetifier blacklist workaround.

im...@google.com <im...@google.com> #13Jan 4, 2019 11:26AM

@12 Sounds like a different issue, possibly around light R classes handling in the IDE, would you mind filing a bug for it and describing the problem in more detail or provide a repro project?

st...@gmail.com <st...@gmail.com> #14Jan 4, 2019 03:31PM

It seems to be working fine in Android Studio 3.3 so I think it might not be an issue in the future. Thanks for the response!

jt...@temp.thunderhead.com <jt...@temp.thunderhead.com> #15Jan 22, 2019 05:31PM

Hi I was wondering the status of the blacklist? Is this official or on the roadmap for 3.4?

hu...@google.com <hu...@google.com> #16Jan 23, 2019 11:33AM

@15: The android.jetifier.blacklist property is available to use on AGP 3.3.0 and above. However, please note that it is only a workaround.

We are working a on better solution that does not require developers to use the workaround. This bug is left open to keep track of that work.

ne...@gmail.com <ne...@gmail.com> #17Feb 3, 2019 12:46PM

Hi. could you backport blacklist option to 3.2.x release?

hu...@google.com <hu...@google.com> #18Feb 4, 2019 10:37AM

Since the blacklist property is already available on 3.3.0 stable, I'm afraid that we won't be releasing 3.2.2 only to include that. I would recommend updating to 3.3.0 stable instead, and file a bug with us if you think something is blocking you from updating.

ne...@gmail.com <ne...@gmail.com> #19Feb 4, 2019 02:05PM

I have some gradle plugin that doesn't work with 3.3.0.
The problem is that Robolectric can't be used with 3.2.1 when Jetifier is enabled because it tries to re-write 'org.ow2.asm:asm' jar which has class files of version > 52.
So I would like to blacklist this jar from being processed.

hu...@google.com <hu...@google.com> #20Feb 5, 2019 03:47PM

I see, but again I'm afraid it is too late for 3.2 at this point. One thing you could try is to disable Jetifier (set android.enableJetifier=false), and run the Jetifier standalone tool (

https://developer.android.com/studio/command-line/jetifier) on individual libraries and import the jetified ones instead of the original ones. I know it's not pretty, but hopefully it will work for you.

ri...@gmail.com <ri...@gmail.com> #21Feb 21, 2019 02:14PM

Any ideas about this one -
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':app:kaptGenerateStubsDebugKotlin'
...
Caused by: java.lang.RuntimeException: Failed to transform '/Users/user/.gradle/caches/modules-2/files-2.1/org.xerial/sqlite-jdbc/3.20.1/df3068e837e9490a9554212fcab40a2c55faf0a3/sqlite-jdbc-3.20.1.jar' using Jetifier. Reason: invalid entry size (expected 618164 but got 636977 bytes). (Run with --stacktrace for more details.) To disable Jetifier, set android.enableJetifier=false in your gradle.properties file.
at com.android.build.gradle.internal.dependency.JetifyTransform.transform(JetifyTransform.kt:204)

hu...@google.com <hu...@google.com> #22Feb 21, 2019 02:21PM

Can you please file a new bug with us and link it here so I can follow up?

ri...@gmail.com <ri...@gmail.com> #23Feb 21, 2019 02:26PM

Sure

ri...@gmail.com <ri...@gmail.com> #24Feb 22, 2019 06:29AM

Hey, deleting and cloning the repo again fixes this issue, do you still want me to file this?

hu...@google.com <hu...@google.com> #25Feb 22, 2019 11:37AM

No need for now (as there is too little information for it to be actionable), but we'll keep that in mind and please do file a bug if you see it again. Thanks!

em...@gmail.com <em...@gmail.com> #26Mar 12, 2019 04:12PM

Hi,

I have this error but I cannot fix if with blacklist, is it the same issue ? Could `android.jetifier.blacklist` property fix it ?

09:08:56 > Could not resolve all files for configuration ':app:releaseUnitTestCompileClasspath'.
09:08:56 > Failed to transform artifact 'full.jar (project :api)' to match attributes {artifactType=processed-jar, com.android.build.api.attributes.BuildTypeAttr=release, com.android.build.api.attributes.VariantAttr=release, com.android.build.gradle.internal.dependency.AndroidTypeAttr=Aar, org.gradle.usage=java-api, org.jetbrains.kotlin.platform.type=androidJvm}

Thanks

hu...@google.com <hu...@google.com> #27Mar 13, 2019 11:50AM

This is a different issue. Please file a new bug and let me know the bug ID. Thanks!

eb...@netflix.com <eb...@netflix.com> #28Mar 13, 2019 03:46PM

Done here :

https://issuetracker.google.com/issues/128469625

me...@thomaskeller.biz <me...@thomaskeller.biz> #29Nov 25, 2019 06:07PM

Jetifier really brings up some headaches. We upgraded AGP from 3.5.2 to 3.6.0-beta04 and found our Cucumber tests failing afterwards. The issue Cucumber reports was

cucumber.runtime.CucumberException: No backends were found. Please make sure you have a backend module on your CLASSPATH.

Cucumber reports that when an artifact like `cucumber-java` (or any other language backend) is not present. Cucumber tries to service-load this automatically when present in the classpath (see

https://github.com/cucumber/cucumber-jvm/search?q=JavaBackendProviderService&unscoped_q=JavaBackendProviderService for references), but apparently fails.

Interestingly, the tests continued to run without issues in Android Studio, they were just failing when executed from command line. A test run with --debug showed me that the Gradle run however used jetified versions of the cucumber libraries in the application classpath. Adding `.*cucumber.*` to the jetifier blacklist solved the issue.

I'm completely unaware of the black magic that Jetifier involves, but I see that it has some very serious consequences when applied broadly on a project, things that are rather hard to track down because subtle things like service locators suddenly stop working.

hu...@google.com <hu...@google.com> #30Nov 28, 2019 12:36PM

Thank you for the feedback! We are looking to remove edge cases like this one, and bug reports like this help us do that.

If possible, could you attach the original version of the cucumber library and the jetified version, so that we can investigate what was actually modified by Jetifier?

hu...@google.com <hu...@google.com> #31Dec 6, 2019 03:13PM

Marked as fixed.

Starting with AGP 4.0.0-alpha07 (using Jetifier 1.0.0-beta06+), Jetifier will not jetify libraries that already support AndroidX:

https://android-review.googlesource.com/c/platform/frameworks/support/+/952620/. The blacklist workaround shouldn't be needed anymore.

After updating to AGP 4.0.0-alpha07+, please remove the android.jetifier.blacklist property if you have been using it. And if you still run into issues after doing that, please file a bug with us at

http://issuetracker.google.com/issues/new?component=460323.

To close the loop on the 3 different approaches outlined at

comment #1

:

> We need a way to determine whether a third-party library should be jetified or not. There are a couple of approaches:
> 1. The library adds some metadata stating that it is AndroidX-ready and therefore should not be jetified.
=> Infeasible

> 2. Jetifier automatically detects the presence of AndroidX (e.g., import statements and strings containing references to AndroidX), and if AndroidX is present, assumes that the library is AndroidX-ready and skips converting it.
=> Implemented

> 3. The users need to specify a blacklist of libraries that should not be jetified.
=> Used previously as a workaround (see

comment #5

), but not recommended now, as approach #2 should remove the need for this workaround. If you still find yourself having to use the blacklist, it is most likely a bug, please tell us at

http://issuetracker.google.com/issues/new?component=460323.

Message last modified on Dec 13, 2019 10:50AM