IssueTracker

In our case, we have b2b app running on dedicated devices.

After 1.1.0 DataStore update we had 6 terminals reported 30k androidx.datastore.core.CorruptionException fatal crashes on app start.

We use DataStore in WorkManager jobs and on regular Activity starts. Seems that after the application gets this state it will never will succeed to start successfully again. I am afraid that data is being corrupted.

Can't reproduce. But same as OP, we have very vanilla usage of DataStore Preferences. No multiple processes, just simple read/write key-values in a single process single Activity.

Hi, thanks for the feedback, noticed the number of +1s, I'll prioritize an investigation.

To facilitate the process, could you please share how you create the DataStore instance?

Here is how we are creating our DataStore<Preferences> instances:

PreferenceDataStoreFactory.create(
    produceFile = { context.preferencesDataStoreFile(filename) }, // filename is [a-zA-Z]
    scope = scope, // CoroutineScope(Dispatchers.IO + SupervisorJob())
    migrations = listOf(
        SharedPreferencesMigration(
            context = context,
            sharedPreferencesName = sharedPrefFile, // we are using our package name here
            keysToMigrate = keysToMigrate, // a Set<String>, sometimes empty if nothing to be migrated
        ),
    ),
    corruptionHandler = ReplaceFileCorruptionHandler {
        // logging step omitted
        emptyPreferences()
    },
)

And these methods are called in Hilt @Provides methods in a @InstallIn(SingletonComponent::class) @Module for each preferences we use in our app.

@Module
@InstallIn(SingletonComponent::class)
class PreferencesDataStoreModule {

    @Provides
    @Singleton
    fun provideDataStore(@ApplicationContext context: Context): DataStore<Preferences> =
        PreferenceDataStoreFactory.create(
            produceFile = {
                context.preferencesDataStoreFile(PREFERENCES_FILE_NAME)
            },
        )
}

Re #comment8 and #comment10, you'll need to configure a corruptionHandler to handle the corrupted data file, like in #comment7, otherwise the library doesn't proactively fix the data corruption.

Re #comment11

Question1:

Why these corruptions started appearing only after 1.1.0 version? We were using DataStore forever with such config and everything worked fine.

Question2:

the library doesn't proactively fix the data corruption.

Why is that? Our expectation was that datastore-preferences would work as drop-in replacement for shared-preferences. And users of shared-preferences never had to care about such issues...

Hi I'm looking into it again, I think b/365889758 could be from the same root cause.

Re #comment12,

Why is that? Our expectation was that datastore-preferences would work as drop-in replacement for shared-preferences. And users of shared-preferences never had to care about such issues...

You didn't have to care such issues of shared preferences because SP just silently fails in such cases, so you won't know the failures. And that's one of the reason why we encouraged people to use Jetpack DataStore than SharedPreferences.

I understand that you would prefer to have the default corruption handler here to replace the file automatically, and it's reasonable to some extent. Unfortunately, we designed the API following a slightly different principle, where people need to "opt-in" this behavior, because their data could be sensitive and they would want to have some ad-hoc logic before the file just disappears and gets replaced automatically. Also you'll need to provide the value for the replacement anyways, right? As far as I can tell, that's not too much different from providing the corruption handler specifications, because you need to pass a param anyways.

Small update. We downgraded datastore-preferences 1.1.0 -> 1.0.0 and have the apps in production for some time. All CorruptionException issues stopped.

Re #comment15

people need to "opt-in" this behavior, because their data could be sensitive and they would want to have some ad-hoc logic before the file just disappears and gets replaced automatically.

Yes, exactly. I do not understand this. You consider that it's important that developers add logic to handle the corruption case, but at the same time it's exposed via API as an optional opt-in.

There are three possibilities, how datastore-preferences initialization API behaves:

1. Have default behavior, to silently reset prefs on corrupted files (match SharedPreferences). On file corruption event: user silently loses data, the application continues to run with default values.
2. Have a mandatory corruptHandler parameter on preferences initialization. public fun preferencesDataStore(corruptionHandler:ReplaceFileCorruptionHandler<Preferences>,<..>). On file corruption event: Developers are forced to decide what should happen.
3. Have an optional/opt-in corruptHandler parameter [current default variant]. On file corruption event: Application is rendered as completely unusable. RuntimeExection is thrown. The only way end-users can use the app again, is to go and do ClearAppData in the App Settings.

The default behavior is the one that will be used by most developers (not opt-in version). Official documentation doesn't even have a word about corruption handling. That means that majority of application have such logic per documentation:

// At the top level of your kotlin file:
val Context.dataStore: DataStore<Preferences> by preferencesDataStore(name = "settings")

Meaning: application will be crashing until the end-user goes to Android app settings and clear data.

So for me ether one of 1. or 2. options sounds better then the current 3. one

Re #comment16, thank you for the inputs.

For the possibilities you've mentioned, we cannot switch to 3 to 1 directly as it would surprise the existing users who build their app based on the contract that DataStore doesn't automatically replace the file as default behavior.

I think 2 is the right way to go as I discussed with some other folks (cc'ed). Unfortunately, as it is a non-backwards-compatible API change, we can only introduce it in DataStore 2.0 release, for which we don't have a concrete timeline yet. I created a feature request to track this in https://issuetracker.google.com/368385681

Re #comment12,

Why these corruptions started appearing only after 1.1.0 version?

We restructured the library to be KMP (Kotlin multi-platform) which introduced the dependency of okio as the IO entrypoint instead of java.io.File. It is not clear to me why does it trigger more CorruptionExceptions stemmed from com.google.protobuf.InvalidProtoBufferExceptions.

Hi community, please let me know if you only see this issue for DataStore<Preferences>, but not the DataStore with other types (especially a protobuf)? In other words, do you also see the same exceptions for DataStore instances of types that are not a Preferences? This would greatly help us narrow the scope to triage, thanks!

We only use DataStore<Preferences>, but that is what we are seeing this issue with if that helps.