IssueTracker

there are some image for help

hello?

ugh that makes sense :( Is it just flat out error if two tasks write to the same output even if they end up writing the same value? We can even make room check contents before writing to ensure file is not overwritten if contents are the same.

It is very unlikely for two configurations to write different schemas but it won't be possible to pick one as the right owner automatically.

Is this a problem still with AGP latest?

@#8-10: Thanks for your inputs! I think the best solution/workaround at this point is for users to mark the property as @Internal, so that it doesn't participate in Gradle's UP-TO-DATE check or task output caching.

In comment #1, I suggested using @OutputDirectory but because of the overlapping output issue mentioned at comment #8, I think we will need to work around that by using @Internal for now.

To do that, users can put the following script in their build.gradle file:

class RoomSchemaDirArgProvider implements CommandLineArgumentProvider{

    @Internal // <===== IMPORTANT
    File schemaDir

    RoomSchemaDirArgProvider(File schemaDir) {
        this.schemaDir = schemaDir
    }

    @Override
    Iterable<String> asArguments() {
        ["-Aroom.schemaLocation=${schemaDir.path}"]
    }
}

android.defaultConfig.javaCompileOptions.annotationProcessorOptions {
    arguments = [
        "room.incremental":"true",
        ... // other arguments
    ]
    compilerArgumentProvider(
        new RoomSchemaDirArgProvider(new File('/path/to/schemaDir'))
    )
}

Could you provide a similar snippet but using kapt instead?

I don't think compilerArgumentProvider exists within a kapt gradle block, so alternatively, moving "schemaLocation" from the kapt block into something like

kapt {
  arguments {
    arg("room.incremental", "true")
  }
}

android.defaultConfig.javaCompileOptions.annotationProcessorOptions {
  compilerArgumentProvider(
          new RoomSchemaDirArgProvider(new File("$projectDir/schemas".toString()))
  )
}

gives me org.codehaus.groovy.runtime.GStringImpl cannot be cast to java.lang.String, I am certainly missing something.

@12: The code snippet in comment #11 works for Kapt as well so you don't need to do anything extra (the kapt block in comment #12 is not necessary).

One issue with marking the output directory @Internal is that build cache entries will not include the generated schemas, so if the output of the kapt task is restored from the build cache the schemas will be missing.

We have code in our debug build that relies on those schemas being present, and I could easily imagine other similar use cases (e.g. CI jobs that do some kind of processing or validation of the schemas).

Yes, to fix this issue properly, I think schemaLocation needs to be marked as @OutputDirectory, and its value must be set per variant (to avoid overlapping outputs among tasks).

To do that, users can put the following script in the build.gradle file:

class RoomSchemaDirArgProvider implements CommandLineArgumentProvider{

    @OutputDirectory // <===== IMPORTANT
    File schemaDir

    RoomSchemaDirArgProvider(File schemaDir) {
        this.schemaDir = schemaDir
    }

    @Override
    Iterable<String> asArguments() {
        ["-Aroom.schemaLocation=${schemaDir.path}"]
    }
}

android.defaultConfig.javaCompileOptions.annotationProcessorOptions {
    arguments = [
        "room.incremental":"true",
        ... // other arguments
    ]
}

android.applicationVariants.all { variant -> // Or android.libraryVariants.all for libraries
    variant.javaCompileOptions.annotationProcessorOptions.compilerArgumentProvider(
            new RoomSchemaDirArgProvider(new File("/path/to/schemaDir-" + variant.name)))
}

For AGP 7.1+, the last block should be:

androidComponents {
    onVariants(selector().all(), { variant ->
        variant.javaCompilation.annotationProcessor.argumentProviders.add(
                new RoomSchemaDirArgProvider(new File('/path/to/schemaDir-' + variant.name)))
    })
}

To summarize:

• Currently, schemaLocation is effectively an @Input, so it's not cached at all, and it actually causes cache misses as that input contains absolute paths.
• schemaLocation can be set as @Internal (see the code snippet in comment #11). It will address the cache miss issue above, but the schemas are still not cached. This is an acceptable fix if you don't need the schemas to be cached.
• schemaLocation can be set as @OutputDirectory per variant (see comment #15); it must be set per variant to avoid overlapping outputs among tasks. This is the recommended fix (addresses all the issues above).

danysantiago@ are there any more details missing for us to address this?

Re comment#17: I don't think there are any more details, Hung would know best. As I understand things this is something users have to setup (see comment#16) since there is no built-in support for incremental processor consuming files and hooking those files properly as inputs, at the same time there is no support for incremental processor to writes files other than using the Filer API, so Room's schema outputs are also not properly hooked up as outputs. So in order to not break cacheability the user has to declare a CommandLineArgumentProvider for Room's schema folder.

At one point we did consider creating a Room Gradle Plugin to set this up for the user, but we didn't went that route since we thought it was not worth it.

Is there a sample available for using the CommandLineArgumentProvider in conjunction with KSP?

I think KSP does not support that yet. Feature request: https://github.com/google/ksp/issues/847

Thanks. Please also thumbs up this related issue which must be solved as well in order variant specific outputs: https://github.com/google/ksp/issues/861

For KAPT and AGP 7.1.2 I do not see schemas placed in the output folder when using the workaround described in https://issuetracker.google.com/issues/132245929#comment15

Here is a sample reproducer: https://github.com/runningcode/kapt-ksp-playground/blob/main/list/build.gradle.kts#L26

Thanks for the sample project!

I see that you've tried both the new and old AGP APIs to set the annotation processor arguments:

New AGP API:

androidComponents {
    onVariants { variant ->
        variant.javaCompilation.annotationProcessor.argumentProviders.add(DeepLinkDispatchDirProvider(File(buildDir.path + "/deeplink/${variant.name}")))
        variant.javaCompilation.annotationProcessor.argumentProviders.add(RoomSchemaDirProvider(File(buildDir.path + "/schemas/${variant.name}")))
    }
}

This is correct, but currently it doesn't work with Kapt because Kapt doesn't understand the new API yet. Specifically, Kapt still tries to read the arguments from javaCompileOptions.annotationProcessorOptions.getCompilerArgumentProviders (link), but with the new AGP API, Kapt will need to read from JavaCompilation.AnnotationProcessor.argumentProviders (link).

Would you mind filing a bug on https://youtrack.jetbrains.com/issues/KT with your sample project?

Old AGP API:

android.libraryVariants.all { variant ->
        variant.javaCompileOptions.annotationProcessorOptions.compilerArgumentProviders.add(DeepLinkDispatchDirProvider(File(buildDir.path + "/" + variant.name)))
        variant.javaCompileOptions.annotationProcessorOptions.compilerArgumentProviders.add(RoomSchemaDirProvider(File(projectDir.parent + "/schemas")))
}

This API is deprecated but not removed yet, so it is still possible to use it. However, the issue is that android.libraryVariants has type DefaultDomainObjectSet<LibraryVariant> and android.libraryVariants.all normally would resolve to org.gradle.api.DomainObjectCollection#all(org.gradle.api.Action<? super T>), but in a Kotlin DSL script it will resolve to Iterable<T>.all provided by kotlin-stdlib, which has entirely different semantics.

To force the code to resolve to the intended method, please try the following workaround and see if it helps:

val action = object : org.gradle.api.Action <com.android.build.gradle.api.LibraryVariant> {
    override fun execute(variant: com.android.build.gradle.api.LibraryVariant) {
        variant.javaCompileOptions.annotationProcessorOptions.compilerArgumentProviders.add(DeepLinkDispatchDirProvider(File(buildDir.path + "/" + variant.name)))
        variant.javaCompileOptions.annotationProcessorOptions.compilerArgumentProviders.add(RoomSchemaDirProvider(File(projectDir.parent + "/schemas")))
    }
}
android.libraryVariants.all(action)

Here you go: https://youtrack.jetbrains.com/issue/KT-51510

I spent some time looking at this problem in more depth recently while migrating our app to KSP. Ignoring the fact that KSP doesn't yet support CommandLineArgumentProvider (coming soon, and there are other workarounds), none of the suggestions in comment#16 are sufficient if you use a remote build cache:

1. Cacheable Gradle tasks need to exclusively own their output directory - if Gradle doesn't know how some of the files in a tasks's output directory were created it will disable caching for that task. room.schemaLocation contains version controlled copies of past versions of the schema, so it can't be exclusively owned by the kapt/KSP task.
2. kapt/KSP tasks fully delete any declared output directories when they run non-incrementally (this behaviour is inherited from AbstractKotlinCompile, and is mentioned in the issue that was closed as a dupe of this one). This is reasonable behaviour for a task that exclusively owns its output directories, but can't work for room.schemaLocation for the same reason as 1.

The only thing that actually works as far as I'm aware is the android-cache-fix-gradle-plugin, which has a clever workaround that involves copying the schemas around to multiple temporary directories and eventually back to room.schemaLocation.

Auto migrations are another spanner in the works. Now the schemas directory is both an input and output of the kapt/KSP task, because past versions of the schema are required to generate the auto migrations. So if the schema directory changes, Room should re-run... but Room also writes the current schemas into that directory, effectively invalidating its own inputs for the next build.

@25: Thanks for the detailed analysis! I think there are 3 issues here:

1. [TODO-Gradle] Gradle doesn't clean up the annotation processors' @OutputDirectorys (in this case, the schema directory): https://github.com/gradle/gradle/issues/9401. When Kapt is used, it does that on behalf of Gradle, so this is only an issue if Kapt is not used.
2. [TODO-Room] Mixing inputs and outputs in one directory makes the build likely incorrect/not cacheable. IMO, Room should provide two annotation processor options: (1) The schema output directory which will contain only 1 schema file, and (2) the schema input directory which contain multiple schema files. Room must read from the input directory only and write to the output directory only.

Workarounds for users:

• During regular development, set the schemaLocation option to the output schema directory (comment #16).

• During database migration, set the schemaLocation option to the input schema directory.

3. [TODO-Room or Room users] During regular development, users will need to collect the output schema files into the input schema directory to be used later (either manually or with the help of a Room script/plugin). The generated schema file probably doesn't change often, so devs may choose to do that only when the database version has changed.

Regarding the priority: #2 and #3 are not blocking issues because users can still do that manually. It seems to me that #1 should be the first to get fixed.

Thanks for the update. The vast majority of Android builds use Kapt or Ksp which would make your point 1 negligible.

However, let's assume that there is an Android build which is not using Kapt or Ksp. As soon as this fix is implemented, Gradle will start deleting user's existing old versioned schemas in that directory, correct?

As soon as the outputs and inputs are separate directories, the above is not an issue, correct? The output directory's single output would overwrite any existing output. As soon as the database version is upgraded, the existing schema is moved to the input directory by the user?

Either way, why is it not possible for the Room annotation processor to delete that directory?

@27: Yes to all your questions above.

Either way, why is it not possible for the Room annotation processor to delete that directory?

It is possible, but it will be a change in behavior for users because suddenly all existing schemas will be deleted.

Note that if Gradle does the cleaning, Room users will also observe the exact the same change in behavior, but at least the new behavior will be correct from Gradle's point of view, and Room will need to update its code/guide its users to work with the new behavior in Gradle.

Disclaimer: I'm a dev in the Android Gradle plugin who filed this bug. I'm just stating my proposals. @da...@google.com (owner of this bug/Room): Please feel free to interrupt me anytime. :)

Small update: One concern for the proposal in comment #26 is that it will degrade the user's experience, so let me clarify it a bit:

Key Idea

The key to fixing this issue is to (1) split the existing room.schemaLocation into room.outputSchemaDir (@OutputDirectory) and room.pastSchemasDir (@InputDirectory) and (2) ensure they are set per Android variant.

Details

The user will define 2 locations in build.gradle (Room will provide a code snippet similar to comment #15, so the user will just need to copy that snippet):

• outputSchemaDir = File(buildDir.path + "/schemas/${variant.name}/output"). Mark this as @OutputDirectory.
• pastSchemasDir = File(buildDir.path + "/schemas/${variant.name}/past-schemas"). Mark this as @InputDirectory.

During a build, Room will do the following steps:

1. Read from pastSchemasDir and perform its auto-migration logic
2. Clean outputSchemaDir
   • Side note: At this step, either Gradle, Kapt, or Room needs to clean the output. Because Gradle is not doing it yet (https://github.com/gradle/gradle/issues/9401), Room should do it (esp. when Kapt is not used).
3. Write the generated schema to outputSchemaDir
4. Also copy the generated schema to pastSchemasDir

In short, the work required for the user is really just copying a code snippet to build.gradle. As shown above, the actual code changes in Room will also be minimal.

Istead of user declaring 2 input directories, why can't we do it? (in the to be implemented plugin)

as in, just use the current schema dir as pastSchemaDir and use a buildDir as outputDir.

But as far as I understand, step 4 (whatever copies it back) is not necessarily declare poastSchemaDir as output (to avoid circular dependency)? So at that step we'll be "hacking" gradle? (e.g. maybe do a finalizedBy for the compilation task or something like that?)

Istead of user declaring 2 input directories, why can't we do it? (in the to be implemented plugin)

Yes, a Room plugin will simplify the user's build script significantly. (I was only proposing the bare minimum to make this work, anything more than this will be even better.)

as in, just use the current schema dir as pastSchemaDir and use a buildDir as outputDir.

Yeah we can refine the details (the options' names & paths), but the key is (1) separating pastSchemasDir from outputSchemaDir, and (2) outputSchemaDir needs to be per variant. (pastSchemasDir should ideally be per variant too, but it's not a strict requirement.)

But as far as I understand, step 4 (whatever copies it back) is not necessarily declare poastSchemaDir as output (to avoid circular dependency)? So at that step we'll be "hacking" gradle? (e.g. maybe do a finalizedBy for the compilation task or something like that?)

You're right, copying the generated schema from outputSchemaDir to pastSchemasDir is the hacky part of all the steps, but I think it's highly likely that we can do it in the annotation processing step (in the unlikely event that it's not possible, we can do it in a post-compilation task).

Ok i can see that working, albeit not great but I'm really not sure how to fix this unless Gradle has support for this use case (or we degrade the UX in another way by asking people to manually copy files, which they'll forget so not feasible)

As for the plugin solution, how do we envision that it will work? Before we add the arguments, we need to know whether particular compilation has the room annotation processor (or symbol processor for ksp), which would require resolving the classpath. Maybe we can declare the compilation's classpath as an input to the command line argument provider, then check if the classpath has room and if so, add the arguments and the code to copy back ?

Variants won't be an issue if we are the ones creating the workingDir but each of them will be copying back into the same schema dir.

But than, I'm not even sure that temporary "outputDir" has any value. Why don't we just use the snipped above (#1) but instead declare it as InputDir instead of OutputDir ? In either case, we are writing back to an InputDir without declaring it.

Ok i can see that working, albeit not great but I'm really not sure how to fix this unless Gradle has support for this use case (or we degrade the UX in another way by asking people to manually copy files, which they'll forget so not feasible)

I think Gradle doesn't have support for storing multiple versions of outputs across different builds. Each build should just be: Inputs -> Task -> Outputs, and happens in isolation.

In between builds, the filesystem can be updated. So Room users are free to move schema files around however they want. At step 4 - Also copy the generated schema to pastSchemasDir simply means Room is doing that on behalf of the users; if viewed that way, there are no violations to the Gradle rules at all even though it appears that way.

As for the plugin solution, how do we envision that it will work? Before we add the arguments, we need to know whether particular compilation has the room annotation processor (or symbol processor for ksp), which would require resolving the classpath. Maybe we can declare the compilation's classpath as an input to the command line argument provider, then check if the classpath has room and if so, add the arguments and the code to copy back ?

I believe all those details can be figured out, it will be similar to RoomSchemaLocationWorkaround in the Android Cache Fix Plugin.

Variants won't be an issue if we are the ones creating the workingDir but each of them will be copying back into the same schema dir.

Each compilation is per variant, so the outputs need to be per variant, and because the outputs are copied back to the inputs, the inputs need to be per variant too.

I think it's best to view the compilation for each build and each variant as independent of the others.

But than, I'm not even sure that temporary "outputDir" has any value. Why don't we just use the snipped above (#1) but instead declare it as InputDir instead of OutputDir ? In either case, we are writing back to an InputDir without declaring it.

That will be similar to marking it as @Internal, which leads to the issue in comment #14 (the output schemas are not cached), so @Internal is not yet the final solution.

That said, @Internal will be much better than the current situation. If we just do that alone, that will already be a big help to the users, so please do consider it as an intermediate fix while we're working on the final solution.

Btw, one possible albeit not great solution is to make compilation fail if an expected schema file is not there.

So if user change the schema version, compilation would fail but still write the output. Then user has to recompile to pass.

Note that, this is never the case in CI, in fact, we should likely consider this an error in CI (if the schema file is not committed along with a schema version upgrade).

Room will still be writing into its output directory but as long as it doesn't get cleaned up, that would be OK (we'll also fail the compilation).

I think we still need to update the documentation to provide the schema location w/ an argument provider (so it is tracked as input). Not sure if we need a plugin just to do that, it might be better to make this case easier in AGP (e.g. maybe a helper to provide these arguments).

You can also add a file, but with the wrong name, wrong location, or with something to remove manually from its content, so people don't have it pass just by trying to compile again without looking.

I think we still need to update the documentation to provide the schema location w/ an argument provider (so it is tracked as input)

I also agree this is the next step.

Maybe we haven't figured out all the details yet, but we can discuss that in the change. (Anything that's better than the current situation is already progress.)

Not sure if we need a plugin just to do that, it might be better to make this case easier in AGP (e.g. maybe a helper to provide these arguments).

I think a code snippet is enough to solve the issue for now. After that, we can think about a Room plugin / helper API from AGP to shorten the amount of build script that the user needs to add.

#35; I've thought about that but that is just another manual step for the user and might be harder to discover for junior developers. I think it is better to just write to the output directory if gradle is fine with it.

@Hung; wdyt about the crashing idea?

• set schemaDir as input directory
• if schema for the current version is missing, write it out and fail compilation
• next compile will work just fine

The difference between failing compilation or not just comes to the CI setup. If someone increments the schema, currently it compiles fine even if they don't commit the latest schema. This new behavior would fail the CI build if they forgot to upload the schema.

Yes, we can set schemaDir as @InputDirectory @PathSensitive(PathSensitivity.RELATIVE) (the RELATIVE part is important for cache relocatability). This is better than @Internal, and either one of them is already a big improvement to the current situation.

Regarding failing when there are no schemas, it looks like the first build will always fail, which isn't great. However, that seems like a separate issue -- the current issue is about schemaDir's absolute path being implicitly marked as @Input, so using a more suitable annotation is already a win.

ok so to wrap up:

• P0: change documentation to use the provider with the right annotations
• P1: figure out whether we want to fail the compilation
  • pro: consistent build w/ local (if someone forgets to commit the file)
  • con: annoying failure that works on a second run.
  • con2: need to double check it but failing might also have a negative impact on iteration speed (each schema change, even if you don't want to bump the version again, might become annoying to get through).

Please, don't give up reproducible builds if possible.

Having the thing fail on first build abd work on second build without any developer work/acknowledgement is a case of non-reproducible build.

Also keep in mind of bunch of developers don't have any CI (especially juniors, which you mentioned).

Besides that, all good I guess 🙂

So this is where things get a bit tricky. As in, having it fail first then succeed is actually making it reproducible. I know it sounds counter-intuitive but technically, reproducible build means same output comes for the same set of inputs.

Just the "questionable behavior" part is that, by writing the missing file when build fails, Room is "changing the inputs for the next compilation".

In another perspective, existence of the latest schema json in inputs does not necessarily mean it has to change the output. So we could still write the file and let the compilation continue (which is what we do right now).

So i think the most most critical part is to declare this as a relative input directory and update the docs hence it plays nice with gradle cache.

What's the latest direction on this?

I find the approach of failing the build when the schema is missing to be quite confusing for developers. Developers struggle with even the most basic build failures and almost never read the error messages no matter how clear they are.

I think making a Room Gradle plugin to be the only approach that would address all cacheability issues and lead to the best build performance. No matter what non-plugin option we pick, we are tying cache misses to the entire JavaCompile task.

We are about to publish updated documentation regarding the schema location setup to get the users on a better stop than right now.

Meanwhile we are also looking into creating a Room Gradle plugin to be able to provide the best experience and to more precisely deal with the cache and reproducible build aspects of the issue.

The updated schema location configuration documentation is live: https://developer.android.com/training/data-storage/room/migrating-db-versions#export-schemas

We are still working on the Gradle plugin, target release is Room 2.6.0 early next year. I'll circle back once we have made more definitive progress on the plugin.

The next alpha release of Room will contain a Gradle plugin that solves the various issues regarding having inputs and outputs of schemas via Gradle annotation processor options. More info will be available in the release notes.

The following release(s) address this bug.It is possible this bug has only been partially addressed:

• androidx.room:room-compiler:2.6.0-alpha02