IssueTracker

Is there another exception logged earlier on? This looks like an exception was thrown from a class initializer at some point, but we need to original exception to debug it.

https://android-build.googleplex.com/builds/submitted/8919297/androidx_incremental/latest/view/logs/build.log

Is it possible to run the build such that stderr from each task is logged? Or is it doing that already?

It should already be printing stderr. My guess is that there was nothing extra.

We hit 14 failures in 2 hours that this CL was landed.

https://dashboards.corp.google.com/_d7c29bbb_d22c_4d60_833b_98f096f089e7?p=SEARCH:Could%20not%20initialize%20class%20org.jetbrains.uast.UastFacade

Did we land anything big in Lint between these two versions?

Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.jetbrains.uast.UastFacade
	at org.jetbrains.uast.UastContext.<init>(UastContext.kt:30)
	at com.android.tools.lint.Fe10UastEnvironmentKt.configureFe10ProjectEnvironment(Fe10UastEnvironment.kt:248)
	at com.android.tools.lint.Fe10UastEnvironmentKt.createKotlinCompilerEnv(Fe10UastEnvironment.kt:237)
	at com.android.tools.lint.Fe10UastEnvironmentKt.access$createKotlinCompilerEnv(Fe10UastEnvironment.kt:1)
	at com.android.tools.lint.Fe10UastEnvironment$Companion.create(Fe10UastEnvironment.kt:199)
	at com.android.tools.lint.UastEnvironment$Companion.create(UastEnvironment.kt:132)
	at com.android.tools.lint.LintCliClient.initializeProjects(LintCliClient.kt:1499)
	at com.android.tools.lint.client.api.LintClient.performInitializeProjects$lint_api(LintClient.kt:1039)
	at com.android.tools.lint.client.api.LintDriver.initializeProjectRoots(LintDriver.kt:589)
	at com.android.tools.lint.client.api.LintDriver.doAnalyze(LintDriver.kt:502)
	at com.android.tools.lint.client.api.LintDriver.analyzeOnly(LintDriver.kt:453)
	at com.android.tools.lint.LintCliClient$analyzeOnly$1.invoke(LintCliClient.kt:250)
	at com.android.tools.lint.LintCliClient$analyzeOnly$1.invoke(LintCliClient.kt:250)
	at com.android.tools.lint.LintCliClient.run(LintCliClient.kt:292)
	at com.android.tools.lint.LintCliClient.run$default(LintCliClient.kt:275)
	at com.android.tools.lint.LintCliClient.analyzeOnly(LintCliClient.kt:250)
	at com.android.tools.lint.Main.run(Main.java:1673)
	at com.android.tools.lint.Main.run(Main.java:279)
	... 34 more

Note that it starts from analyzeOnly (if it matters).

As per b/237807241#comment10, I thought "unnecessary UAST initialization" filtered out by https://googleplex-android.git.corp.google.com/platform/tools/base/+/8665f2647d1d9eb041b336e5bc0288efdd0d03c3%5E%21/#F2 may be still required. But, it's actually filtering out for DriverMode.MERGE I guess, while the error starts from analyzeOnly. :confused:

(Assigned as part of triage)

This is going to block androidx from upgrading AGP, it would be awesome if someone investigated this. It is definitely something that changed between the two versions.

But, it's actually filtering out for DriverMode.MERGE I guess, while the error starts from analyzeOnly. :confused:

IIUC the NoClassDefFoundError means that a previous attempt at class initialization failed. So it is possible that the exception originally occurred during mergeOnly, not analyzeOnly. (And, somehow we are missing the original exception in the logs, which makes this hard to debug.)

I only have one theory so far. I noticed that UastFacade has a static class initializer that does this:

init {
  UastLanguagePlugin.extensionPointName.addChangeListener(...);
}

In particular, addChangeListener queries Extensions.getRootArea() and assumes the result is non-null. However, the result could be null if the Application environment is not initialized. In that case, an NPE would be thrown from UastFacade.<clinit> and thus any subsequent accesses to UastFacade would throw NoClassDefFoundError.

If the theory is correct, then it implies that at some point during LintDriver.mergeOnly() we accidentally trigger class initialization for UastFacade.

This is only a theory for now. I'll continue digging.

Hm, I wonder if the original exception is being swallowed in LintDriver.doAnalyze here:

try {
    analysis(roots)
} catch (throwable: Throwable) {
    handleDetectorError(null, this, throwable)
}

(this catch-block reports most exceptions as a special Lint incident in the Lint report—which is unhelpful when the final Lint report task is broken).

@aurimas can you set either the Java system property lint.print-stacktrace=true and the environment variable LINT_PRINT_STACKTRACE=true? That should force the original exception to be logged to stderr.

We already set this env variable.

I was afraid you'd say that (but thanks for confirming).

I'll keep digging tomorrow.

OK, the root cause turned out to be quite interesting.

Root cause:

• During a lintReport task, Lint still loads custom Lint checks and runs LintJarVerifier on them to verify binary compatibility with the Lint API. To verify binary compatibility, LintJarVerifier scans Lint-check bytecode and makes sure that all class/method/field references would resolve correctly in the current classloader. E.g., to verify a class reference, Lint looks up the class via Class.forName(className).

• Unfortunately, Class.forName(className) also initializes the given class by default. So, if a custom Lint check references UastFacade, then the static initializer for UastFacade will run at this point.

• As predicted in comment comment#10, if the static initializer for UastFacade runs when the UAST environment is not initialized, then an NPE is thrown. This scenario is possible after the recent change http://ag/19152690 which disables UAST initialization for lintReport tasks.

• The NPE causes an ExceptionInInitializerError, and thus all subsequent accesses to UastFacade in the same Gradle daemon throw NoClassDefFoundError. Also, the original ExceptionInInitializerError is never logged to stderr is because it is swallowed by LintJarVerifier in one of its catch-Throwable blocks.

• The reason the issue is flaky is because it requires that some lintReport task executes before any other lintAnalyze task in a given Gradle daemon.

Repro steps:

• Make sure the project is using a custom Lint check which references the UastFacade class. One example is UnsafeFragmentLifecycleObserverDetector from the library androidx.fragment:fragment.

• Run a series of Gradle commands such that some lintReport task runs before any other lintAnalyze tasks within a given Gradle daemon. Example:

  ./gradlew clean
  ./gradlew :app:lintAnalyzeDebug
  ./gradlew --stop
  ./gradlew :app:lintReportDebug
  # <edit some source file so that Lint is no longer up-to-date>
  ./gradlew :app:lintAnalyzeDebug
  

• This results in the :app:lintAnalyzeDebug task failing with NoClassDefFoundError.

Next steps:

I'll change LintJarVerifier so that it no longer triggers class initialization. This should fix the issue. If it does not—and there are other places where the UastFacade static initializer can be triggered during a lintReport task—then I think we should revert http://ag/19152690 until we have a better understanding of how to avoid this class of issues.

Wow, that's super interesting, nice analysis!

So I'm assuming you're replacing Class.forName(className) with Class.forName(className, false, this.javaClass.classLoader) ? That seems like a good idea. I guess this would technically also catch cases where lint issue registries referenced classes that had errors (e.g. referencing some other class that hasn't been properly packaged) but that's probably fine; those codepaths would be touched eventually when the lint check runs -- and the goal here is to see whether you're referencing a lint API that doesn't exist due to version mismatches, and those shouldn't have initialization errors.

So I'm assuming you're replacing Class.forName(className) with Class.forName(className, false, this.javaClass.classLoader) ?

Yup! That's the plan.

(I realized afterwards that my comment about checking the bytecode for the lint check isn't important -- because here we're talking about the bytecode verifier loading the platform APIs that are referenced. When the JarFileIssueRegistry runs, it will initialize the classes in the jar file itself, so those errors should be surfaced either way.)

This should be fixed by Change I13dea197b, which should land in Canary 11 (or Canary 10 if my cherry-pick request goes through).

p.s. I still consider it a bug that UastFacade has a static initializer which can fail depending on some global state, so I may look into that some more too.

Thanks you for the quick fix!