IssueTracker

My dependencies:

    implementation 'androidx.appcompat:appcompat:1.4.0'
    implementation 'androidx.preference:preference:1.1.1'
    implementation 'com.google.android.gms:play-services-wearable:17.1.0'
    implementation 'com.google.android.material:material:1.4.0'

If I remove androidx.appcompat:appcompat:1.4.0, the warning vanishes, so I imagine there's some transitive dependency on an older version of androidx.appcompat:appcompat that doesn't have the warning.

https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:preference/preference/src/main/java/androidx/preference/PreferenceDialogFragmentCompat.java;l=53;drc=ca9feb3b73769089afbfd36b4d4a3d91239f9cd5;bpv=1;bpt=0

Looks like a duplicate of https://issuetracker.google.com/issues/181793702, ow...@google.com I proposed a "solution" to this in the other issue.

Hi, thanks for the feedback, noticed the number of +1s, I'll prioritize an investigation.

To facilitate the process, could you please share how you create the DataStore instance?

Here is how we are creating our DataStore<Preferences> instances:

PreferenceDataStoreFactory.create(
    produceFile = { context.preferencesDataStoreFile(filename) }, // filename is [a-zA-Z]
    scope = scope, // CoroutineScope(Dispatchers.IO + SupervisorJob())
    migrations = listOf(
        SharedPreferencesMigration(
            context = context,
            sharedPreferencesName = sharedPrefFile, // we are using our package name here
            keysToMigrate = keysToMigrate, // a Set<String>, sometimes empty if nothing to be migrated
        ),
    ),
    corruptionHandler = ReplaceFileCorruptionHandler {
        // logging step omitted
        emptyPreferences()
    },
)

And these methods are called in Hilt @Provides methods in a @InstallIn(SingletonComponent::class) @Module for each preferences we use in our app.

@Module
@InstallIn(SingletonComponent::class)
class PreferencesDataStoreModule {

    @Provides
    @Singleton
    fun provideDataStore(@ApplicationContext context: Context): DataStore<Preferences> =
        PreferenceDataStoreFactory.create(
            produceFile = {
                context.preferencesDataStoreFile(PREFERENCES_FILE_NAME)
            },
        )
}

Re #comment8 and #comment10, you'll need to configure a corruptionHandler to handle the corrupted data file, like in #comment7, otherwise the library doesn't proactively fix the data corruption.

Re #comment11

Question1:

Why these corruptions started appearing only after 1.1.0 version? We were using DataStore forever with such config and everything worked fine.

Question2:

the library doesn't proactively fix the data corruption.

Why is that? Our expectation was that datastore-preferences would work as drop-in replacement for shared-preferences. And users of shared-preferences never had to care about such issues...

Hi I'm looking into it again, I think b/365889758 could be from the same root cause.

Re #comment12,

Why is that? Our expectation was that datastore-preferences would work as drop-in replacement for shared-preferences. And users of shared-preferences never had to care about such issues...

You didn't have to care such issues of shared preferences because SP just silently fails in such cases, so you won't know the failures. And that's one of the reason why we encouraged people to use Jetpack DataStore than SharedPreferences.

I understand that you would prefer to have the default corruption handler here to replace the file automatically, and it's reasonable to some extent. Unfortunately, we designed the API following a slightly different principle, where people need to "opt-in" this behavior, because their data could be sensitive and they would want to have some ad-hoc logic before the file just disappears and gets replaced automatically. Also you'll need to provide the value for the replacement anyways, right? As far as I can tell, that's not too much different from providing the corruption handler specifications, because you need to pass a param anyways.

Small update. We downgraded datastore-preferences 1.1.0 -> 1.0.0 and have the apps in production for some time. All CorruptionException issues stopped.

Re #comment15

people need to "opt-in" this behavior, because their data could be sensitive and they would want to have some ad-hoc logic before the file just disappears and gets replaced automatically.

Yes, exactly. I do not understand this. You consider that it's important that developers add logic to handle the corruption case, but at the same time it's exposed via API as an optional opt-in.

There are three possibilities, how datastore-preferences initialization API behaves:

1. Have default behavior, to silently reset prefs on corrupted files (match SharedPreferences). On file corruption event: user silently loses data, the application continues to run with default values.
2. Have a mandatory corruptHandler parameter on preferences initialization. public fun preferencesDataStore(corruptionHandler:ReplaceFileCorruptionHandler<Preferences>,<..>). On file corruption event: Developers are forced to decide what should happen.
3. Have an optional/opt-in corruptHandler parameter [current default variant]. On file corruption event: Application is rendered as completely unusable. RuntimeExection is thrown. The only way end-users can use the app again, is to go and do ClearAppData in the App Settings.

The default behavior is the one that will be used by most developers (not opt-in version). Official documentation doesn't even have a word about corruption handling. That means that majority of application have such logic per documentation:

// At the top level of your kotlin file:
val Context.dataStore: DataStore<Preferences> by preferencesDataStore(name = "settings")

Meaning: application will be crashing until the end-user goes to Android app settings and clear data.

So for me ether one of 1. or 2. options sounds better then the current 3. one

Re #comment16, thank you for the inputs.

For the possibilities you've mentioned, we cannot switch to 3 to 1 directly as it would surprise the existing users who build their app based on the contract that DataStore doesn't automatically replace the file as default behavior.

I think 2 is the right way to go as I discussed with some other folks (cc'ed). Unfortunately, as it is a non-backwards-compatible API change, we can only introduce it in DataStore 2.0 release, for which we don't have a concrete timeline yet. I created a feature request to track this in https://issuetracker.google.com/368385681

Re #comment12,

Why these corruptions started appearing only after 1.1.0 version?

We restructured the library to be KMP (Kotlin multi-platform) which introduced the dependency of okio as the IO entrypoint instead of java.io.File. It is not clear to me why does it trigger more CorruptionExceptions stemmed from com.google.protobuf.InvalidProtoBufferExceptions.

Hi community, please let me know if you only see this issue for DataStore<Preferences>, but not the DataStore with other types (especially a protobuf)? In other words, do you also see the same exceptions for DataStore instances of types that are not a Preferences? This would greatly help us narrow the scope to triage, thanks!

We only use DataStore<Preferences>, but that is what we are seeing this issue with if that helps.

We only use DataStore<Preferences>, too

We're seeing this with DataStore<Preferences> as well. Haven't used other types. We'll likely revert back to v1.0.0.

We use DataStore in a KMP project with PreferencesSerializer and custom proto serializers but only get crash reports for PreferencesSerializer.

Hi, we are actively working on this but no update yet, as we couldn't reproduce the issue yet. And we do suspect it is happening more than before so we believe there is an issue.

One theory we have right now is that this might be related to OkioStorage instead of FileStorage.

If anyone hitting this issue can try using FileStorage instead of OkioStorage, that would be great feedback for us. (they compatible with each-other so it shouldn't affect your code, but you'll need a CorruptionHandler to get the datastore out of that state).

If anyone hitting this issue can try using FileStorage instead of OkioStorage,

How would I do that? This is what I have at the moment but it doesn't compile because PreferencesSerializer is not a Serializer<T> but a OkioSerializer<Preferences> instead:

PreferenceDataStoreFactory.create(
  storage = FileStorage<Preferences>(PreferencesSerializer) { cacheFile },
  corruptionHandler = ReplaceFileCorruptionHandler { emptyPreferences() },
  scope = coroutineScope,
)

Datastore 1.1.1

You are right, sorry we don't have that available.

I've previously provided a FileSerializer in another bug, you can copy that one: https://b.corp.google.com/issues/256784635#comment44

Please let us know if you can get a chance to try it, thanks!

Thanks I'll give that a go. BTW public issue tracker link for anyone else watching: https://issuetracker.google.com/issues/256784635#comment44

Thanks! Looking forward to hearing about the results

Hi again, can I double check something please:

I've updated our app's code to use the above PreferenceFileSerializer which uses the Java buffered input / output APIs as opposed to the serialiser in Datastore 1.1.1 that uses Okio's BufferedSource / BufferedSink. However as far as I can tell, that's the only difference in these two serialisers. And the Okio version simply calls a method to convert its versions into the Java versions. Do I understand correctly so far?

On talking further with our team we think actually adding a corruption handler to our Datastore setup would work for the situation we've got ourselves into. So what, in addition to that, will you want us to look out for when testing these changes? We're still going to give the Java File based version a go. I'm just not sure what difference I'm expecting to see.

Cheers.

Hi there! Thank you for your follow-up.

The reason why we wanted to get a bit of a smoke test/check with the swapping of FileStorage for OkioStorage is that we theorize this issue came to be after a large refactor datastore underwent going from 1.0.0 to 1.1.0, which included many changes, a major one of which was the introduction of the Storage interface, as well as using OkioStorage as the default storage solution. All the reports we have received so far has been involved with OkioStorage, but we have no data point indicating whether:

[1] FileStorage works but OkioStorage does not

or

[2] Because OkioStorage is the default, no one has encountered the issue while using FileStorage by chance.

We are trying to narrow down the scope of the problem so we can hopefully have more input into the root cause.

I see, thank you for the clarification. I'll get back to you when we get some results from the release.

#34 As for data, if you have field metrics and track how many times your corruptionHandler gets called, that would be great.

If you have ability to a/b test this such that a portion of users get the OkioStorage and the rest gets FileStorage. and then count:

• % of users who use OkioStorage and get a callback to corruption handler
• % of users who use FileStorage and get a callback to corruption handler

It would be amazing help in narrowing down the problem for us.

This is a bit too much to ask so I'm not expecting. But if you can provide that comparative data, it would be great for us.

I am developing an application on a development board, using Datastore Preferences to store data. If I power off immediately after calling the edit function, the data will be damaged if I power it back on. My development board does not have a battery like ordinary mobile phones, but is powered directly by a DC plug. Currently, this test method is 100% reproducible on several development boards. I hope it will be helpful for you to reproduce the problem, and I hope to solve this problem as soon as possible. I am looking forward to the new version of DataStore!

#38, interesting case. What filesystem does your developer board use?

The way datastore writes work is we first write a secondary file, and after flushing it, we do an atomicMove to replace the primary file.

This should not be causing a loss assuming the filesystem is consistent on the device. If the power outage happens while writing the tmp file, nothing happens to the original file. If the power outage happens during atomic move, it is up to the filesystem to execute that operation atomically.

https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:datastore/datastore-core-okio/src/commonMain/kotlin/androidx/datastore/core/okio/OkioStorage.kt;l=125?q=OkioStorage&ss=androidx%2Fplatform%2Fframeworks%2Fsupport

#40 My development board system uses Android7.1 and Android9.0 respectively

I just tried to reproduce the problem using version v1.1.1. I cut off the power immediately after modification according to the above method. The results are as follows:

First time: Cut off the power immediately after modification. The file was not damaged after restarting. Instead, the data before modification was restored

Second time: Cut off the power immediately after modification. After restarting, the data could not be read, triggering the corruptionHandler

This step was tried twice. The first time it was not saved successfully. The second time it was still not able to read the data, triggering the corruptionHandler

I updated the Data Store to version v1.1.3 and used the same test method. The result was the same as v1.1.1. Next, I will try to use Proto DataStore instead of Preferences DataStore to test

I'm back again, testing using Proto DataStore 1.1.3, the results are consistent with Preferences DataStore

After modifying the stored data with Proto DataStore 1.1.3 The data still exists after the first power off and restart, but the modification was unsuccessful The second data was cleared

I can be sure that this will only happen after the power off and restart after the modification. The amazing thing is that I found that when I modify the stored data and switch the APP to the background, the storage is 100% successful. At this time, no matter how many times I power off and restart, its data exists normally

Thank you for your detailed analysis!

This is quite interesting indeed - have you tried the experiment using FileStorage instead of OkioStorage as well? If so, are the results the same?

#43 Haha, I just tried the FilePreferences.kt and JvmSerializer.kt provided by #31, that is, using FileStorage, and the results are as follows:

I tested 20 times, all of which were power off and restarted after changing the storage data. 15 of them were saved successfully and 5 failed to save. However, it was just a failure to save, the data was not damaged or lost, and the corruptionHandler was not triggered. This result has met my development needs 😍. I hope to find the problem as soon as possible or use FileStorage and provide a new version of DataStore.

Thank you again!

This definitely gives us some good insight, as we have been trying to pinpoint the issue by narrowing the scope down, and your experiment results do really help with this.

We'll keep investigating on our end with these new insights in mind, but please do feel free to update us on here if you encounter other findings.