IssueTracker

Hi,

Thank you for your feedback. Could you help provide logs when the issue occurs and if possible a code snippet of the part that uses CameraX, it will help us find the root cause.

A quick question: does this issue occur on other devices, or only on Samsung Galaxy M55.

Thank you for your kind reply! I think we will receive the phone in a while and we will update if we find anything.

Hi,

We got the device and did some investigating. The root cause seems to be that Samsung M55 has issues when using ImageAnalysis with another stream that uses TEMPLATE_RECORD, which is typically used by VideoCapture or by a stream shared between VideoCapture and Preview.

The issue also occurs when only ImageAnalysis and VideoCapture are bound. The video can only obtain the first few frames in this case.

We will fix this issues in a future release. One workaround you can try is to bind an additional ImageCaputre usecase (binds all 4 usecases), it seems the stream is normal in this situation.

Thanks for finding this bug!

Hi

Since in #comment1 the error was reported for the 1.4.1 release, maybe you meant 1.4.* release?

We will fix on both the 1.4 stable release, and the current 1.5 release. For 1.3, it is recommended to migrate to the 1.4 stable version. Or use one of the following workarounds:

1. bind an addiotional ImageCapture use case
2. use Camera2Interop to change VideoCapture builder's CONTROL_CAPTURE_INTENT to CONTROL_CAPTURE_INTENT_PREVIEW. As in the following code snippet:

VideoCapture.Builder<Recorder> videoCaptureBuilder = new VideoCapture.Builder<>(recorder);
new Camera2Interop.Extender<>(videoCaptureBuilder).setCaptureRequestOption(CaptureRequest.CONTROL_CAPTURE_INTENT, CaptureRequest.CONTROL_CAPTURE_INTENT_PREVIEW);

Project: platform/frameworks/support
Branch: androidx-main
Author: mingdatsai <mingdatsai@google.com>
Link:      https://android-review.googlesource.com/3529969

Fix preview freeze when using ImageAnalysis with a TEMPLATE_RECORD stream

Fix preview freeze when using ImageAnalysis with a TEMPLATE_RECORD stream 
 
Fixed preview freeze issue when using ImageAnalysis with another stream 
using TEMPLATE_RECORD. 
 
Relnote: "Fixed preview freeze issue when using ImageAnalysis with another stream that uses TEMPLATE_RECORD on Samsung SM-M556B device." 
 
Bug: 395822788 
Test: manual test and ./gradlew bOS 
Change-Id: Ic1a6a019c1525f30d7b863f732d421e36e5c978c

Project: platform/frameworks/support
Branch: androidx-main
Author: mingdatsai <mingdatsai@google.com>
Link:      https://android-review.googlesource.com/3538342

Apply AbnormalStreamWhenImageAnalysisBindWithTemplateRecordQuirk for other M55 models

Apply AbnormalStreamWhenImageAnalysisBindWithTemplateRecordQuirk for other M55 models 
 
Bug: 395822788 
Test: manual test and ./gradlew bOS 
Change-Id: I1768c199f0a4a62e82c7d75f89af574c3eda8604

Hi,

The fix has been merged. It will be released in the subsequest CameraX version, probably 1.4.2 (also on the current 1.5 branch).

I will close the bug, please feel free to reopen it if there are any issues.

Re #comment16, thank you for the inputs.

For the possibilities you've mentioned, we cannot switch to 3 to 1 directly as it would surprise the existing users who build their app based on the contract that DataStore doesn't automatically replace the file as default behavior.

I think 2 is the right way to go as I discussed with some other folks (cc'ed). Unfortunately, as it is a non-backwards-compatible API change, we can only introduce it in DataStore 2.0 release, for which we don't have a concrete timeline yet. I created a feature request to track this in https://issuetracker.google.com/368385681

Re #comment12,

Why these corruptions started appearing only after 1.1.0 version?

We restructured the library to be KMP (Kotlin multi-platform) which introduced the dependency of okio as the IO entrypoint instead of java.io.File. It is not clear to me why does it trigger more CorruptionExceptions stemmed from com.google.protobuf.InvalidProtoBufferExceptions.

Hi community, please let me know if you only see this issue for DataStore<Preferences>, but not the DataStore with other types (especially a protobuf)? In other words, do you also see the same exceptions for DataStore instances of types that are not a Preferences? This would greatly help us narrow the scope to triage, thanks!

We only use DataStore<Preferences>, but that is what we are seeing this issue with if that helps.

We only use DataStore<Preferences>, too

We're seeing this with DataStore<Preferences> as well. Haven't used other types. We'll likely revert back to v1.0.0.

We use DataStore in a KMP project with PreferencesSerializer and custom proto serializers but only get crash reports for PreferencesSerializer.

Hi, we are actively working on this but no update yet, as we couldn't reproduce the issue yet. And we do suspect it is happening more than before so we believe there is an issue.

One theory we have right now is that this might be related to OkioStorage instead of FileStorage.

If anyone hitting this issue can try using FileStorage instead of OkioStorage, that would be great feedback for us. (they compatible with each-other so it shouldn't affect your code, but you'll need a CorruptionHandler to get the datastore out of that state).

If anyone hitting this issue can try using FileStorage instead of OkioStorage,

How would I do that? This is what I have at the moment but it doesn't compile because PreferencesSerializer is not a Serializer<T> but a OkioSerializer<Preferences> instead:

PreferenceDataStoreFactory.create(
  storage = FileStorage<Preferences>(PreferencesSerializer) { cacheFile },
  corruptionHandler = ReplaceFileCorruptionHandler { emptyPreferences() },
  scope = coroutineScope,
)

Datastore 1.1.1

You are right, sorry we don't have that available.

I've previously provided a FileSerializer in another bug, you can copy that one: https://b.corp.google.com/issues/256784635#comment44

Please let us know if you can get a chance to try it, thanks!

Thanks I'll give that a go. BTW public issue tracker link for anyone else watching: https://issuetracker.google.com/issues/256784635#comment44

Thanks! Looking forward to hearing about the results

Hi again, can I double check something please:

I've updated our app's code to use the above PreferenceFileSerializer which uses the Java buffered input / output APIs as opposed to the serialiser in Datastore 1.1.1 that uses Okio's BufferedSource / BufferedSink. However as far as I can tell, that's the only difference in these two serialisers. And the Okio version simply calls a method to convert its versions into the Java versions. Do I understand correctly so far?

On talking further with our team we think actually adding a corruption handler to our Datastore setup would work for the situation we've got ourselves into. So what, in addition to that, will you want us to look out for when testing these changes? We're still going to give the Java File based version a go. I'm just not sure what difference I'm expecting to see.

Cheers.

Hi there! Thank you for your follow-up.

The reason why we wanted to get a bit of a smoke test/check with the swapping of FileStorage for OkioStorage is that we theorize this issue came to be after a large refactor datastore underwent going from 1.0.0 to 1.1.0, which included many changes, a major one of which was the introduction of the Storage interface, as well as using OkioStorage as the default storage solution. All the reports we have received so far has been involved with OkioStorage, but we have no data point indicating whether:

[1] FileStorage works but OkioStorage does not

or

[2] Because OkioStorage is the default, no one has encountered the issue while using FileStorage by chance.

We are trying to narrow down the scope of the problem so we can hopefully have more input into the root cause.

I see, thank you for the clarification. I'll get back to you when we get some results from the release.

#34 As for data, if you have field metrics and track how many times your corruptionHandler gets called, that would be great.

If you have ability to a/b test this such that a portion of users get the OkioStorage and the rest gets FileStorage. and then count:

• % of users who use OkioStorage and get a callback to corruption handler
• % of users who use FileStorage and get a callback to corruption handler

It would be amazing help in narrowing down the problem for us.

This is a bit too much to ask so I'm not expecting. But if you can provide that comparative data, it would be great for us.

I am developing an application on a development board, using Datastore Preferences to store data. If I power off immediately after calling the edit function, the data will be damaged if I power it back on. My development board does not have a battery like ordinary mobile phones, but is powered directly by a DC plug. Currently, this test method is 100% reproducible on several development boards. I hope it will be helpful for you to reproduce the problem, and I hope to solve this problem as soon as possible. I am looking forward to the new version of DataStore!

#38, interesting case. What filesystem does your developer board use?

The way datastore writes work is we first write a secondary file, and after flushing it, we do an atomicMove to replace the primary file.

This should not be causing a loss assuming the filesystem is consistent on the device. If the power outage happens while writing the tmp file, nothing happens to the original file. If the power outage happens during atomic move, it is up to the filesystem to execute that operation atomically.

https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:datastore/datastore-core-okio/src/commonMain/kotlin/androidx/datastore/core/okio/OkioStorage.kt;l=125?q=OkioStorage&ss=androidx%2Fplatform%2Fframeworks%2Fsupport

#40 My development board system uses Android7.1 and Android9.0 respectively

I just tried to reproduce the problem using version v1.1.1. I cut off the power immediately after modification according to the above method. The results are as follows:

First time: Cut off the power immediately after modification. The file was not damaged after restarting. Instead, the data before modification was restored

Second time: Cut off the power immediately after modification. After restarting, the data could not be read, triggering the corruptionHandler

This step was tried twice. The first time it was not saved successfully. The second time it was still not able to read the data, triggering the corruptionHandler

I updated the Data Store to version v1.1.3 and used the same test method. The result was the same as v1.1.1. Next, I will try to use Proto DataStore instead of Preferences DataStore to test

I'm back again, testing using Proto DataStore 1.1.3, the results are consistent with Preferences DataStore

After modifying the stored data with Proto DataStore 1.1.3 The data still exists after the first power off and restart, but the modification was unsuccessful The second data was cleared

I can be sure that this will only happen after the power off and restart after the modification. The amazing thing is that I found that when I modify the stored data and switch the APP to the background, the storage is 100% successful. At this time, no matter how many times I power off and restart, its data exists normally

Thank you for your detailed analysis!

This is quite interesting indeed - have you tried the experiment using FileStorage instead of OkioStorage as well? If so, are the results the same?

#43 Haha, I just tried the FilePreferences.kt and JvmSerializer.kt provided by #31, that is, using FileStorage, and the results are as follows:

I tested 20 times, all of which were power off and restarted after changing the storage data. 15 of them were saved successfully and 5 failed to save. However, it was just a failure to save, the data was not damaged or lost, and the corruptionHandler was not triggered. This result has met my development needs 😍. I hope to find the problem as soon as possible or use FileStorage and provide a new version of DataStore.

Thank you again!

This definitely gives us some good insight, as we have been trying to pinpoint the issue by narrowing the scope down, and your experiment results do really help with this.

We'll keep investigating on our end with these new insights in mind, but please do feel free to update us on here if you encounter other findings.

Project: platform/frameworks/support
Branch: androidx-main
Author: elifbilgin <elifbilgin@google.com>
Link:      https://android-review.googlesource.com/3532229

Replace the default storage from OkioStorage to FileStorage.

Replace the default storage from OkioStorage to FileStorage. 
 
Users have reported that a corruption exception is encountered in OkioStorage. Until we can debug the behavior and provide a fix, we are switching the default storage solution. 
 
Bug: 346197747 
Test: Existing 
Change-Id: I71181374929543c542b953daa553bec0b9aff0d9

Hi there - yes it will be out on March 26, 2025.

Hi, which version of datastore are you using? It should be resolved in the latest release that came out today.

Thanks for letting us know!

Closing this issue as Fixed, but if you encounter the issue again please let us know on here.

Hello can I check that I am using the updated version correctly please. The changes for 1.1.4 say Replace the default storage from OkioStorage to FileStorage. However when I try and use 1.1.4 I don't see FileStorage being used. I see OkioStorage still being used. Updating again to 1.2.0-alpha01 has the changed files. Is this what I should be expecting or am I looking / thinking about this in the wrong place?

For reference I am trying to use the Preferences Core version of the Datastore library because my own SDK is a JVM only SDK. Not sure that makes a difference but worth noting.

Thanks for letting us know, let me check on what is going on, as both 1.1.4 and the 1.2.0-alpha01 should have the same change in regards to FileStorage.

Thanks for the info - looks like we have a redundant PreferencesDataStoreFactory.jvm.kt (which still relies on Okio) where we should have only a PreferencesDataStoreFactory.jvmAndroid.kt in jvmAndroidMain (which does have the fix). Resolving this and will do another release 1.1.5 to have this removed.

Hello, we are also encountering this issue with androidx.datastore:datastore:1.1.4 when we are using a Proto DataStore in our app. I wanted to double check, since comment #58 mentions PreferencesDataStoreFactory.jvm.kt, do you expect the release 1.1.5 to fix the issue for Proto DataStores as well? I have attached a stack trace below for your reference, the other detail that may be of interest is that this crasher occurs much less on 1.1.4 than it did on 1.1.3. Thank you!

          Fatal Exception: java.lang.RuntimeException: Unable to get provider com.google.firebase.provider.FirebaseInitProvider: androidx.datastore.core.CorruptionException: Unable to parse preferences proto.
       at android.app.ActivityThread.installProvider(ActivityThread.java:8489)
       at android.app.ActivityThread.installContentProviders(ActivityThread.java:7989)
       at android.app.ActivityThread.handleBindApplication(ActivityThread.java:7675)
       at android.app.ActivityThread.-$$Nest$mhandleBindApplication()
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2478)
       at android.os.Handler.dispatchMessage(Handler.java:106)
       at android.os.Looper.loopOnce(Looper.java:230)
       at android.os.Looper.loop(Looper.java:319)
       at android.app.ActivityThread.main(ActivityThread.java:8919)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:578)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1103)

Caused by androidx.datastore.core.CorruptionException: Unable to parse preferences proto.
       at androidx.datastore.preferences.PreferencesMapCompat$Companion.readFrom(PreferencesMapCompat.kt:34)
       at androidx.datastore.preferences.core.PreferencesSerializer.readFrom(PreferencesSerializer.jvm.kt:46)
       at androidx.datastore.core.okio.OkioReadScope.readData$suspendImpl(OkioStorage.kt:180)
       at androidx.datastore.core.okio.OkioReadScope.readData()
       at androidx.datastore.core.StorageConnectionKt$readData$2.invokeSuspend(StorageConnection.kt:74)
       at androidx.datastore.core.StorageConnectionKt$readData$2.invoke(:9)
       at androidx.datastore.core.StorageConnectionKt$readData$2.invoke(:10)
       at androidx.datastore.core.okio.OkioStorageConnection.readScope(OkioStorage.kt:113)
       at androidx.datastore.core.StorageConnectionKt.readData(StorageConnection.kt:74)
       at androidx.datastore.core.DataStoreImpl.readDataFromFileOrDefault(DataStoreImpl.kt:323)
       at androidx.datastore.core.DataStoreImpl.readDataOrHandleCorruption(DataStoreImpl.kt:365)
       at androidx.datastore.core.DataStoreImpl.access$readDataOrHandleCorruption(DataStoreImpl.kt:48)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invokeSuspend(DataStoreImpl.kt:437)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invoke(:8)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invoke(:2)
       at androidx.datastore.core.SingleProcessCoordinator.lock(SingleProcessCoordinator.kt:41)
       at androidx.datastore.core.DataStoreImpl$InitDataStore.doRun(DataStoreImpl.kt:434)
       at androidx.datastore.core.RunOnce.runIfNeeded(DataStoreImpl.kt:497)
       at androidx.datastore.core.DataStoreImpl.readAndInitOrPropagateAndThrowFailure(DataStoreImpl.kt:266)
       at androidx.datastore.core.DataStoreImpl.access$readAndInitOrPropagateAndThrowFailure(DataStoreImpl.kt:48)
       at androidx.datastore.core.DataStoreImpl$readState$2.invokeSuspend(DataStoreImpl.kt:218)
       at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
       at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
       at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
       at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103)
       at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

Caused by androidx.datastore.preferences.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
       at androidx.datastore.preferences.protobuf.CodedInputStream$StreamDecoder.readTag(CodedInputStream.java:2074)
       at androidx.datastore.preferences.protobuf.CodedInputStreamReader.getFieldNumber(CodedInputStreamReader.java:59)
       at androidx.datastore.preferences.protobuf.MessageSchema.mergeFromHelper(MessageSchema.java:2973)
       at androidx.datastore.preferences.protobuf.MessageSchema.mergeFrom(MessageSchema.java:2955)
       at androidx.datastore.preferences.protobuf.GeneratedMessageLite.parsePartialFrom(GeneratedMessageLite.java:1599)
       at androidx.datastore.preferences.protobuf.GeneratedMessageLite.parseFrom(GeneratedMessageLite.java:1742)
       at androidx.datastore.preferences.PreferencesProto$PreferenceMap.parseFrom(PreferencesProto.java:198)
       at androidx.datastore.preferences.PreferencesMapCompat$Companion.readFrom(PreferencesMapCompat.kt:32)
       at androidx.datastore.preferences.core.PreferencesSerializer.readFrom(PreferencesSerializer.jvm.kt:46)
       at androidx.datastore.core.okio.OkioReadScope.readData$suspendImpl(OkioStorage.kt:180)
       at androidx.datastore.core.okio.OkioReadScope.readData()
       at androidx.datastore.core.StorageConnectionKt$readData$2.invokeSuspend(StorageConnection.kt:74)
       at androidx.datastore.core.StorageConnectionKt$readData$2.invoke(:9)
       at androidx.datastore.core.StorageConnectionKt$readData$2.invoke(:10)
       at androidx.datastore.core.okio.OkioStorageConnection.readScope(OkioStorage.kt:113)
       at androidx.datastore.core.StorageConnectionKt.readData(StorageConnection.kt:74)
       at androidx.datastore.core.DataStoreImpl.readDataFromFileOrDefault(DataStoreImpl.kt:323)
       at androidx.datastore.core.DataStoreImpl.readDataOrHandleCorruption(DataStoreImpl.kt:365)
       at androidx.datastore.core.DataStoreImpl.access$readDataOrHandleCorruption(DataStoreImpl.kt:48)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invokeSuspend(DataStoreImpl.kt:437)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invoke(:8)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invoke(:2)
       at androidx.datastore.core.SingleProcessCoordinator.lock(SingleProcessCoordinator.kt:41)
       at androidx.datastore.core.DataStoreImpl$InitDataStore.doRun(DataStoreImpl.kt:434)
       at androidx.datastore.core.RunOnce.runIfNeeded(DataStoreImpl.kt:497)
       at androidx.datastore.core.DataStoreImpl.readAndInitOrPropagateAndThrowFailure(DataStoreImpl.kt:266)
       at androidx.datastore.core.DataStoreImpl.access$readAndInitOrPropagateAndThrowFailure(DataStoreImpl.kt:48)
       at androidx.datastore.core.DataStoreImpl$readState$2.invokeSuspend(DataStoreImpl.kt:218)
       at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
       at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
       at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
       at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103)
       at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

The root cause should be the same, getting the fix in for 1.1.5 should resolve the issue for both.

Great, thank you!