IssueTracker

Information redacted by Android Beta Feedback.

Thank you for reporting this issue. We have shared this with our product and engineering team and will update this issue with more information as it becomes available.

not fixed on 6a QPR beta1.1

If you continue to have this issue, please file a new issue with the bug report here https://goo.gl/TbMiIO and reference this bug for context.

For other issues related to battery drain, we request you to file a separate issue with all the relevant details like bug report

There seems to be a theory that Google play service does not judge network conditions at all, and even requests a large number of invalid connection requests when the network is limited, resulting in power consumption. It seems that the Google app (com.google.quicksearchbox) will be the same. Because the bug interception mechanism of Android lacks some functions, it cannot be recorded in the bug report.

Hi, we are actively working on this but no update yet, as we couldn't reproduce the issue yet. And we do suspect it is happening more than before so we believe there is an issue.

One theory we have right now is that this might be related to OkioStorage instead of FileStorage.

If anyone hitting this issue can try using FileStorage instead of OkioStorage, that would be great feedback for us. (they compatible with each-other so it shouldn't affect your code, but you'll need a CorruptionHandler to get the datastore out of that state).

If anyone hitting this issue can try using FileStorage instead of OkioStorage,

How would I do that? This is what I have at the moment but it doesn't compile because PreferencesSerializer is not a Serializer<T> but a OkioSerializer<Preferences> instead:

PreferenceDataStoreFactory.create(
  storage = FileStorage<Preferences>(PreferencesSerializer) { cacheFile },
  corruptionHandler = ReplaceFileCorruptionHandler { emptyPreferences() },
  scope = coroutineScope,
)

Datastore 1.1.1

You are right, sorry we don't have that available.

I've previously provided a FileSerializer in another bug, you can copy that one: https://b.corp.google.com/issues/256784635#comment44

Please let us know if you can get a chance to try it, thanks!

Thanks I'll give that a go. BTW public issue tracker link for anyone else watching: https://issuetracker.google.com/issues/256784635#comment44

Thanks! Looking forward to hearing about the results

Hi again, can I double check something please:

I've updated our app's code to use the above PreferenceFileSerializer which uses the Java buffered input / output APIs as opposed to the serialiser in Datastore 1.1.1 that uses Okio's BufferedSource / BufferedSink. However as far as I can tell, that's the only difference in these two serialisers. And the Okio version simply calls a method to convert its versions into the Java versions. Do I understand correctly so far?

On talking further with our team we think actually adding a corruption handler to our Datastore setup would work for the situation we've got ourselves into. So what, in addition to that, will you want us to look out for when testing these changes? We're still going to give the Java File based version a go. I'm just not sure what difference I'm expecting to see.

Cheers.

Hi there! Thank you for your follow-up.

The reason why we wanted to get a bit of a smoke test/check with the swapping of FileStorage for OkioStorage is that we theorize this issue came to be after a large refactor datastore underwent going from 1.0.0 to 1.1.0, which included many changes, a major one of which was the introduction of the Storage interface, as well as using OkioStorage as the default storage solution. All the reports we have received so far has been involved with OkioStorage, but we have no data point indicating whether:

[1] FileStorage works but OkioStorage does not

or

[2] Because OkioStorage is the default, no one has encountered the issue while using FileStorage by chance.

We are trying to narrow down the scope of the problem so we can hopefully have more input into the root cause.

I see, thank you for the clarification. I'll get back to you when we get some results from the release.

#34 As for data, if you have field metrics and track how many times your corruptionHandler gets called, that would be great.

If you have ability to a/b test this such that a portion of users get the OkioStorage and the rest gets FileStorage. and then count:

• % of users who use OkioStorage and get a callback to corruption handler
• % of users who use FileStorage and get a callback to corruption handler

It would be amazing help in narrowing down the problem for us.

This is a bit too much to ask so I'm not expecting. But if you can provide that comparative data, it would be great for us.

I am developing an application on a development board, using Datastore Preferences to store data. If I power off immediately after calling the edit function, the data will be damaged if I power it back on. My development board does not have a battery like ordinary mobile phones, but is powered directly by a DC plug. Currently, this test method is 100% reproducible on several development boards. I hope it will be helpful for you to reproduce the problem, and I hope to solve this problem as soon as possible. I am looking forward to the new version of DataStore!

#38, interesting case. What filesystem does your developer board use?

The way datastore writes work is we first write a secondary file, and after flushing it, we do an atomicMove to replace the primary file.

This should not be causing a loss assuming the filesystem is consistent on the device. If the power outage happens while writing the tmp file, nothing happens to the original file. If the power outage happens during atomic move, it is up to the filesystem to execute that operation atomically.

https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:datastore/datastore-core-okio/src/commonMain/kotlin/androidx/datastore/core/okio/OkioStorage.kt;l=125?q=OkioStorage&ss=androidx%2Fplatform%2Fframeworks%2Fsupport

#40 My development board system uses Android7.1 and Android9.0 respectively

I just tried to reproduce the problem using version v1.1.1. I cut off the power immediately after modification according to the above method. The results are as follows:

First time: Cut off the power immediately after modification. The file was not damaged after restarting. Instead, the data before modification was restored

Second time: Cut off the power immediately after modification. After restarting, the data could not be read, triggering the corruptionHandler

This step was tried twice. The first time it was not saved successfully. The second time it was still not able to read the data, triggering the corruptionHandler

I updated the Data Store to version v1.1.3 and used the same test method. The result was the same as v1.1.1. Next, I will try to use Proto DataStore instead of Preferences DataStore to test

I'm back again, testing using Proto DataStore 1.1.3, the results are consistent with Preferences DataStore

After modifying the stored data with Proto DataStore 1.1.3 The data still exists after the first power off and restart, but the modification was unsuccessful The second data was cleared

I can be sure that this will only happen after the power off and restart after the modification. The amazing thing is that I found that when I modify the stored data and switch the APP to the background, the storage is 100% successful. At this time, no matter how many times I power off and restart, its data exists normally

Thank you for your detailed analysis!

This is quite interesting indeed - have you tried the experiment using FileStorage instead of OkioStorage as well? If so, are the results the same?

#43 Haha, I just tried the FilePreferences.kt and JvmSerializer.kt provided by #31, that is, using FileStorage, and the results are as follows:

I tested 20 times, all of which were power off and restarted after changing the storage data. 15 of them were saved successfully and 5 failed to save. However, it was just a failure to save, the data was not damaged or lost, and the corruptionHandler was not triggered. This result has met my development needs 😍. I hope to find the problem as soon as possible or use FileStorage and provide a new version of DataStore.

Thank you again!

This definitely gives us some good insight, as we have been trying to pinpoint the issue by narrowing the scope down, and your experiment results do really help with this.

We'll keep investigating on our end with these new insights in mind, but please do feel free to update us on here if you encounter other findings.

Project: platform/frameworks/support
Branch: androidx-main
Author: elifbilgin <elifbilgin@google.com>
Link:      https://android-review.googlesource.com/3532229

Replace the default storage from OkioStorage to FileStorage.

Replace the default storage from OkioStorage to FileStorage. 
 
Users have reported that a corruption exception is encountered in OkioStorage. Until we can debug the behavior and provide a fix, we are switching the default storage solution. 
 
Bug: 346197747 
Test: Existing 
Change-Id: I71181374929543c542b953daa553bec0b9aff0d9

Hi there - yes it will be out on March 26, 2025.

Hi, which version of datastore are you using? It should be resolved in the latest release that came out today.

Thanks for letting us know!

Closing this issue as Fixed, but if you encounter the issue again please let us know on here.

Hello can I check that I am using the updated version correctly please. The changes for 1.1.4 say Replace the default storage from OkioStorage to FileStorage. However when I try and use 1.1.4 I don't see FileStorage being used. I see OkioStorage still being used. Updating again to 1.2.0-alpha01 has the changed files. Is this what I should be expecting or am I looking / thinking about this in the wrong place?

For reference I am trying to use the Preferences Core version of the Datastore library because my own SDK is a JVM only SDK. Not sure that makes a difference but worth noting.

Thanks for letting us know, let me check on what is going on, as both 1.1.4 and the 1.2.0-alpha01 should have the same change in regards to FileStorage.

Thanks for the info - looks like we have a redundant PreferencesDataStoreFactory.jvm.kt (which still relies on Okio) where we should have only a PreferencesDataStoreFactory.jvmAndroid.kt in jvmAndroidMain (which does have the fix). Resolving this and will do another release 1.1.5 to have this removed.

Hello, we are also encountering this issue with androidx.datastore:datastore:1.1.4 when we are using a Proto DataStore in our app. I wanted to double check, since comment #58 mentions PreferencesDataStoreFactory.jvm.kt, do you expect the release 1.1.5 to fix the issue for Proto DataStores as well? I have attached a stack trace below for your reference, the other detail that may be of interest is that this crasher occurs much less on 1.1.4 than it did on 1.1.3. Thank you!

          Fatal Exception: java.lang.RuntimeException: Unable to get provider com.google.firebase.provider.FirebaseInitProvider: androidx.datastore.core.CorruptionException: Unable to parse preferences proto.
       at android.app.ActivityThread.installProvider(ActivityThread.java:8489)
       at android.app.ActivityThread.installContentProviders(ActivityThread.java:7989)
       at android.app.ActivityThread.handleBindApplication(ActivityThread.java:7675)
       at android.app.ActivityThread.-$$Nest$mhandleBindApplication()
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2478)
       at android.os.Handler.dispatchMessage(Handler.java:106)
       at android.os.Looper.loopOnce(Looper.java:230)
       at android.os.Looper.loop(Looper.java:319)
       at android.app.ActivityThread.main(ActivityThread.java:8919)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:578)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1103)

Caused by androidx.datastore.core.CorruptionException: Unable to parse preferences proto.
       at androidx.datastore.preferences.PreferencesMapCompat$Companion.readFrom(PreferencesMapCompat.kt:34)
       at androidx.datastore.preferences.core.PreferencesSerializer.readFrom(PreferencesSerializer.jvm.kt:46)
       at androidx.datastore.core.okio.OkioReadScope.readData$suspendImpl(OkioStorage.kt:180)
       at androidx.datastore.core.okio.OkioReadScope.readData()
       at androidx.datastore.core.StorageConnectionKt$readData$2.invokeSuspend(StorageConnection.kt:74)
       at androidx.datastore.core.StorageConnectionKt$readData$2.invoke(:9)
       at androidx.datastore.core.StorageConnectionKt$readData$2.invoke(:10)
       at androidx.datastore.core.okio.OkioStorageConnection.readScope(OkioStorage.kt:113)
       at androidx.datastore.core.StorageConnectionKt.readData(StorageConnection.kt:74)
       at androidx.datastore.core.DataStoreImpl.readDataFromFileOrDefault(DataStoreImpl.kt:323)
       at androidx.datastore.core.DataStoreImpl.readDataOrHandleCorruption(DataStoreImpl.kt:365)
       at androidx.datastore.core.DataStoreImpl.access$readDataOrHandleCorruption(DataStoreImpl.kt:48)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invokeSuspend(DataStoreImpl.kt:437)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invoke(:8)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invoke(:2)
       at androidx.datastore.core.SingleProcessCoordinator.lock(SingleProcessCoordinator.kt:41)
       at androidx.datastore.core.DataStoreImpl$InitDataStore.doRun(DataStoreImpl.kt:434)
       at androidx.datastore.core.RunOnce.runIfNeeded(DataStoreImpl.kt:497)
       at androidx.datastore.core.DataStoreImpl.readAndInitOrPropagateAndThrowFailure(DataStoreImpl.kt:266)
       at androidx.datastore.core.DataStoreImpl.access$readAndInitOrPropagateAndThrowFailure(DataStoreImpl.kt:48)
       at androidx.datastore.core.DataStoreImpl$readState$2.invokeSuspend(DataStoreImpl.kt:218)
       at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
       at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
       at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
       at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103)
       at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

Caused by androidx.datastore.preferences.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
       at androidx.datastore.preferences.protobuf.CodedInputStream$StreamDecoder.readTag(CodedInputStream.java:2074)
       at androidx.datastore.preferences.protobuf.CodedInputStreamReader.getFieldNumber(CodedInputStreamReader.java:59)
       at androidx.datastore.preferences.protobuf.MessageSchema.mergeFromHelper(MessageSchema.java:2973)
       at androidx.datastore.preferences.protobuf.MessageSchema.mergeFrom(MessageSchema.java:2955)
       at androidx.datastore.preferences.protobuf.GeneratedMessageLite.parsePartialFrom(GeneratedMessageLite.java:1599)
       at androidx.datastore.preferences.protobuf.GeneratedMessageLite.parseFrom(GeneratedMessageLite.java:1742)
       at androidx.datastore.preferences.PreferencesProto$PreferenceMap.parseFrom(PreferencesProto.java:198)
       at androidx.datastore.preferences.PreferencesMapCompat$Companion.readFrom(PreferencesMapCompat.kt:32)
       at androidx.datastore.preferences.core.PreferencesSerializer.readFrom(PreferencesSerializer.jvm.kt:46)
       at androidx.datastore.core.okio.OkioReadScope.readData$suspendImpl(OkioStorage.kt:180)
       at androidx.datastore.core.okio.OkioReadScope.readData()
       at androidx.datastore.core.StorageConnectionKt$readData$2.invokeSuspend(StorageConnection.kt:74)
       at androidx.datastore.core.StorageConnectionKt$readData$2.invoke(:9)
       at androidx.datastore.core.StorageConnectionKt$readData$2.invoke(:10)
       at androidx.datastore.core.okio.OkioStorageConnection.readScope(OkioStorage.kt:113)
       at androidx.datastore.core.StorageConnectionKt.readData(StorageConnection.kt:74)
       at androidx.datastore.core.DataStoreImpl.readDataFromFileOrDefault(DataStoreImpl.kt:323)
       at androidx.datastore.core.DataStoreImpl.readDataOrHandleCorruption(DataStoreImpl.kt:365)
       at androidx.datastore.core.DataStoreImpl.access$readDataOrHandleCorruption(DataStoreImpl.kt:48)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invokeSuspend(DataStoreImpl.kt:437)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invoke(:8)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invoke(:2)
       at androidx.datastore.core.SingleProcessCoordinator.lock(SingleProcessCoordinator.kt:41)
       at androidx.datastore.core.DataStoreImpl$InitDataStore.doRun(DataStoreImpl.kt:434)
       at androidx.datastore.core.RunOnce.runIfNeeded(DataStoreImpl.kt:497)
       at androidx.datastore.core.DataStoreImpl.readAndInitOrPropagateAndThrowFailure(DataStoreImpl.kt:266)
       at androidx.datastore.core.DataStoreImpl.access$readAndInitOrPropagateAndThrowFailure(DataStoreImpl.kt:48)
       at androidx.datastore.core.DataStoreImpl$readState$2.invokeSuspend(DataStoreImpl.kt:218)
       at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
       at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
       at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
       at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103)
       at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

The root cause should be the same, getting the fix in for 1.1.5 should resolve the issue for both.

Great, thank you!

Should be fixed in 1.1.5 going out next week.

I had another question about recovering from the crash: once we update to 1.1.5, assuming that the cause of the crash is fixed, will users who have file corruption issues be able to recover from the crash automatically once their app is updated, or is there any additional action for us to take for them to recover? For example, would these users have to clear app data, or is there a migration we could run in our datastore code to help them recover, or is it expected that just updating the dependency to 1.1.5 will prevent crashes from continuing to happen?

I haven't been able to reproduce this issue on my end, so am curious for your thoughts! I am also curious if anyone in the community has been able to find a method to reproduce the crash on mobile phones, aside from the method described to cause the crash on a development board in #38

Hi everyone, 1.1.5 will be going out this wednesday, the 23rd of April.

Regarding comment #65 the "fix" we have here is essentially to use FileStorage instead of OkioStorage by default. We have not introduced any additional changes that would handle any crash/corruption scenarios. If you want to add some form of handling in case of a crash adding a CorruptionHandler is probably the best choice.

As mentioned in the previous comments on this thread, since this is a a difficult to produce bug that is encountered in production, we are not 100% sure this fix resolves every crash mentioned on this thread, but have some evidence to feel confident that this is the root cause. If the crashes persist with 1.1.5 or 1.2.0-alpha01 please let us know.

Thank you, that makes sense! We do have a CorruptionHandler declared, so hopefully that will add an extra layer of protection. We will keep you posted on how 1.1.5 performs once released, thank you again for your time and effort to make the fixes and keep us updated here, it is much appreciated!

It looks like 1.1.5 removes emptyPreferences() extension function, breaking our existing ReplaceFileCorruptionHandler:

PreferenceDataStoreFactory.create(
   corruptionHandler = ReplaceFileCorruptionHandler(
      produceNewData = { emptyPreferences() }
   ),
   migrations = emptyList(),
   scope = CoroutineScope(Dispatchers.IO + SupervisorJob()),
   produceFile = { appContext.preferencesDataStoreFile("preferences") 
}

The issue mentioned in comment #71 has been reported on this tracker. Closing this issue now as it is not related.