IssueTracker

I found the way to reproduce it!

First you launch workaroundWithTransaction method, then getHolderById method. The result: getHolderById stops returning result.

This error is ever worse than before because it is reproduceable in almost 100% cases!

I have to return to BundledSQLiteDriver

Can you provide more details on the repro case? Is it possible to create a sample repro? Or maybe a test? Can you please show us some code?

Are you starting a transaction and in the transaction you are also collecting the Flow?

db.workaroundWithTransaction {
  db.dao.getHolderById(...).collect {
    
  }
}

This is not a valid usage of Flow because you are essentially starting a transaction that will never end and that might explain the database locked exception you are seeing.

Here another test I have created based on your description but it works fine for me on all drivers:

@Test
fun check() = runTest {
    async(Dispatchers.IO) {
        // Insert 100 items in a deferred transaction and notifying invalidation tracker
        repeat(100) { id ->
            db.useWriterConnection {
                it.deferredTransaction {
                    db.dao().insert(SampleEntity(id.toLong()))
                }
            }
            db.invalidationTracker.refreshAsync()
        }
    }

    async(Dispatchers.IO) {
        // Collect list of inserted items until list is of size 100, meaning
        // all 100 items where inserted and a result with a list of them was received
        db.dao().getItemListFlow().collect {
            if (it.size == 100) {
                cancel()
            }
        }
    }
}

Hi. The order of reproducing:

1. call db.dao().getHolderById("some_id").first() - will return a result
2. call db.useWriterConnection {...} - to make any change in db
3. call db.dao().getHolderById("some_id").first() - no longer returns a result

Only restarting the app will make db.dao().getHolderById("some_id").first() work again.

For now I can't create a simple repo, maybe later.

I don't have a minimal repro case, but I am facing this issue in my project (https://github.com/tasks/tasks)

I have been using AndroidSQLiteDriver in production and never had a database locked crash. Many months ago I briefly deployed BundledSQLiteDriver but reverted it because of this crash

This weekend I wanted to use window functions on Android so I switched to BundledSQLiteDriver again. It did not crash on my Pixel 9 Pro during development so I thought the issue was fixed. Then I deployed my changes and am getting tons of crashes again

I then tested on a CAT S22. This thing is a potato and I get database locked crashes easily.

I commented out my transactor.withTransaction calls so the only transactions are generated by Room and it still crashes

When I use PRAGMA busy_timeout to increase the timeout then the database just deadlocks instead of crashing

Just to add more details:

I can comment out my @RawQuery, useWriterConnection, and useReaderConnection calls, and I still get database locked crashes using only generated @Query, @Insert, @Update, and @Delete functions. The app runs in a single process

Switching from BundledSQLiteDriver back to AndroidSQLiteDriver fixes the issue completely

On my CAT S22 the app crashes on first launch after a clean install 100% of the time. The app will start after relaunching but it will then crash frequently during synchronization

Hi. Is there any updates? I believe it is should be P1 issue.

My guess, without looking at any of the library code, is that transactions are not being queued with the BundledSQLiteDriver. I have firebase crashes from 20+ different database functions at this point. My pixel 9 pro is probably servicing requests fast enough to not crash, the CAT S22 not so much.

I'll grab an old Nexus 5 I have around that is pretty potato to see if I can repro this.

Room has a connection pool manager and it has 4 reader connections and one writer, it uses coroutines to suspend callers when the connections are being used. This is indeed different than Android's driver that 'blocks' callers when a connection is not available.

I became aware of room's connection strategy when I learned that reading data with Room flows acquired a transaction, which was a surprising source of performance problems at my day job 🙃

I thought using AndroidSQLiteDriver was analogous to not specifying a driver at all, and would use the connection/transaction strategy you mentioned. Based on my observations I assumed this was related to the problem with the BundledSQLiteDriver - transactions are happening on multiple connections rather than queuing them on one connection? I'm just speculating and could also be mistaken about how this works under the hood.

If you're trying to repro by running my app but aren't having any luck let me know, I could create a branch that seeds the app with some test data or something.

I was able to successful run the Tasks app but indeed haven't had luck reproducing the issue, so if you do end up creating that branch with seed data I will gladly use it.

Reading data via a Flow returned from a @Dao has never required a transaction, unless specified in the DAO via @Transaction. Would love to hear more about the surprising performance issues.

However, I agree that transactions tend to be a source of performance issues. If no driver is used Room uses SupportSQLite and Android's SQLite bindings, transactions there always use the single primary connection and they block each other. When using the AndroidSQLiteDriver, similarly that uses Android's SQLite binding which will also uses the primary connection. There is a difference in code paths when no driver is specified and when AndroidSQLiteDriver is used, but it is in terms of backwards compatibility of other SupportSQLite APIs (see warning in https://developer.android.com/training/data-storage/room/room-kmp-migration#set_a_sqlite_driver).

Now BundledSQLiteDriver internally does not have a connection pool like the Android driver does and instead relies on Room's which we try to test as much as we can but of course it is not as mature's as Android. So we really appreciate you trying it out and reporting these issues. As for how different it is, the main difference is that the connection pool is suspending, other coroutines can continue while a connection is being used, this is very different from the other drivers that will 'block' the caller which for DAO functions Room alleviates by using a FIFO single threaded pool (the transaction executor) but can't control when other blocking APIs are used (beginTransaction). So yes, it is possible to start multiple transactions with Room's connection pool, but in essence they are also queued via suspending coroutines as the connection is being acquired and returned to the pool.

Reading data via a Flow returned from a @Dao has never required a transaction, unless specified in the DAO via @Transaction. Would love to hear more about the surprising performance issues.

The issue we were having was that on startup our sync loop would sometimes start a transaction before our UI created a Room flow, and the UI flow would be blocked until the sync transaction finished. I think we jumped to the conclusion that Room flows acquire a transaction when they're started, but I just looked at the library code again and I think its just acquiring a transaction the first time you observe a table 🤦 edit: OK I'll stop derailing this ticket, but when a flow is started and its the first observer on a table there is a transaction, and when a flow finishes and its the last observer on a table there is a transaction

I was able to successful run the Tasks app but indeed haven't had luck reproducing the issue, so if you do end up creating that branch with seed data I will gladly use it.

I will get this to you ASAP

Yes, if a Flow needs to observe tables that have no observer 'installed' then Room has to create some TRIGGERs to start observing the tables and it does it in a transaction. Unfortunately this has to block the Flow's initial query to guarantee consistency. I think we would need a different invalidation system, not TRIGGER based, to possibly avoid that start and end transaction but I don't think we have enough strong compelling reasons to change the system.

I have attached a backup file that you can import into Tasks by tapping on 'Settings > Backups > Import' and selecting the file.

On my CAT S22 with this commit that just adds BundledSQLiteDriver it crashes with a database locked error, and the previous commit does not.

The next-worst phone I have is a Galaxy S9 and I cannot get the database locked error on there, even with 10x as much data in the database

When I look at Firebase, these are the types of phones experiencing this crash: TP-Link Neffos A5 Tinno Y52 ZTE Blade A34 Moto E13 & E20 Xiamo Redmi A1 & A2 Alcatel 1C Transsion Tecno Spark Go 1 Mobiwire Smart C11

This list is not exhaustive, I just grabbed it from the first 5 distinct crashes. I've never even heard of most of these phones until just now

Thank you for participating. I am really glad to see some progress and discussion on this issue.

Project: platform/frameworks/support
Branch: androidx-main
Author: Daniel Santiago Rivera <danysantiago@google.com>
Link:      https://android-review.googlesource.com/3462864

Reduce no-op transaction while syncing triggers

Reduce no-op transaction while syncing triggers 
 
The ObservedTableStates instance will return a list of observe operations (add or remove triggers) to perform if it 'needs sync'. A sync is required if at least a trigger needs to be added or removed from a table since the last sync. However, it might be that multiple observers are added and removed such that the end result of operations is actually to do nothing. In this case Room still tries to start a database transaction to not add or remove any triggers. Since transactions have lock implications it is best to avoid the transaction that if all observer operations are no-op. 
 
Example case this change is optimizing: 
1.  Flow A in UI begins collecting, trigger A is installed 
2.  UI is rotated, not using collectAsStateWithLifecycle() or equivalent 
2.a Flow A stops collecting, a sync is required 
2.b Flow A starts collecting, a sync is still required 
3.  Triggers are synced, operation on table A is no-op 
 
Bug: 380088809 
Test: Existing 
Change-Id: Iae612a1c4f8b8f1e8e15d2f60eb9bf68241570ef

Are you were able to reproduce the issue and fix it or it is an other issue got fixed?

I've spent an hour trying to repro the deadlock I mentioned in https://issuetracker.google.com/u/1/issues/380088809#comment30 when using PRAGMA busy_timeout = <time in ms> with the CAT S22. I have not been successful, so maybe that does fix my problem. I'm going to try releasing with BundledSQLiteDriver again 🤞

Hi. Are there any updates on this?

Hey - 2.7.0-alpha13 has a change that should reduce the issue, but not guaranteed.

But I was not able to reproduce the issue with the project and data from comment#39. 🙁

Things got busy so I have to dedicate more time to this issue, but its sort of a wild goose chase without a reliable repro.

Starting with the latest version, 2.7.0-alpha13, we switched to BundledSQLiteDriver, which produces hundreds of database-locked exceptions.

   Non-fatal Exception: android.database.SQLException: Error code: 5, message: database is locked
       at androidx.sqlite.driver.bundled.BundledSQLiteStatementKt.nativeStep(BundledSQLiteStatement.jvmAndroid.kt)
       at androidx.sqlite.driver.bundled.BundledSQLiteStatementKt.access$nativeStep(BundledSQLiteStatement.jvmAndroid.kt)
       at androidx.sqlite.driver.bundled.BundledSQLiteStatement.step(BundledSQLiteStatement.jvmAndroid.kt:100)
       at app.resubs.core.database.dao.PendingNotificationDao_Impl.getAll$lambda$1(PendingNotificationDao_Impl.kt:94)
       at androidx.room.util.DBUtil__DBUtil_androidKt$performSuspending$lambda$1$$inlined$internalPerform$1.invokeSuspend(DBUtil.kt:68)
       at androidx.room.util.DBUtil__DBUtil_androidKt$performSuspending$lambda$1$$inlined$internalPerform$1.invoke(DBUtil.kt:8)
       at androidx.room.util.DBUtil__DBUtil_androidKt$performSuspending$lambda$1$$inlined$internalPerform$1.invoke(DBUtil.kt:4)
       at androidx.room.coroutines.ConnectionPoolImpl$useConnection$4.invokeSuspend(ConnectionPoolImpl.kt:144)
       at androidx.room.coroutines.ConnectionPoolImpl$useConnection$4.invoke(ConnectionPoolImpl.kt:8)
       at androidx.room.coroutines.ConnectionPoolImpl$useConnection$4.invoke(ConnectionPoolImpl.kt:4)
       at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:43)
       at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:166)
       at kotlinx.coroutines.BuildersKt.withContext(Builders.kt)
       at androidx.room.coroutines.ConnectionPoolImpl.useConnection(ConnectionPoolImpl.kt:144)
       at androidx.room.coroutines.ConnectionPoolImpl$useConnection$1.invokeSuspend(ConnectionPoolImpl.kt:13)
       at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
       at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:35)

Hi ck...@gmail.com. If you have any ideas on how to reproduce the error, please help with this.

I did found that we were not configuring the busy_timeout in the initial connection which can lead to SQLITE_BUSY (error code 5). I've sent a cl to fix that.

I have another alternative / workaround for those with this issue which will also hint me to the root of the cause. In comment#10 I mentioned how increasing the timeout can help, but I realize my suggestion there was not great because it should be done on all connections and not just on the initial connection, to do this you will need to wrap your driver, here is an example code:

val actualDriver = BundledSQLiteDriver()
val driverWrapper =
  object : SQLiteDriver by actualDriver {
    override fun open(fileName: String): SQLiteConnection {
      return actualDriver.open(fileName).also { newConnection ->
        newConnection.execSQL("PRAGMA busy_timeout = $customBusyTimeout")
      }
    }
  }
roomDatabaseBuilder
  .setDriver(driverWrapper)

Let me know if this helps or not.

Hi ck...@gmail.com, ba...@gmail.com. Could you please test this solution on your side?

Project: platform/frameworks/support
Branch: androidx-main
Author: Daniel Santiago Rivera <danysantiago@google.com>
Link:      https://android-review.googlesource.com/3486790

Apply busy_timeout config for initial connection

Apply busy_timeout config for initial connection 
 
Fix an issue where busy_timeout was not being configured in the initial connection where schema validation is also done. The busy_timeout must be set on all connections to avoid SQLITE_BUSY. 
 
Bug: 380088809 
Test: BaseBuilderTest#setCustomBusyTimeout 
Change-Id: I9320856f64363b05bcf6407eed0efe36ef3312a3

Hi. I have applied this fix to my app and release for 10% of users. I got 4000 installs and for now I don't see this error. I am going to try with version 2.7.0-rc01.

I'm still seeing this or a similar issue on 2.7.0-rc01. I haven't tried using AndroidSQLiteDriver or any of the other workarounds listed here. Previously, I was using setQueryCoroutineContext(Dispatchers.IO.limitedParallelism(1)) which did solve the problem. I removed that after upgrading to 2.7.0-rc01.

I got this crash yesterday:

android.database.SQLException: Error code: 5, message: database is locked
    at androidx.sqlite.driver.bundled.BundledSQLiteStatementKt.nativeStep(SourceFile)
    at androidx.sqlite.driver.bundled.BundledSQLiteStatementKt.access$nativeStep(BundledSQLiteStatement.jvmAndroid.kt:0)
    at androidx.sqlite.driver.bundled.BundledSQLiteStatement.step(BundledSQLiteStatement.jvmAndroid.kt:0)
    at androidx.sqlite.driver.bundled.BundledSQLiteStatement.step(BundledSQLiteStatement.jvmAndroid.kt:100)
    at androidx.sqlite.SQLite.execSQL(com.google.android.gms:play-services-mlkit-barcode-scanning@@18.3.1:56)
    at androidx.room.coroutines.PooledConnectionImpl.endTransaction(ConnectionPoolImpl.kt:420)
    at androidx.room.coroutines.PooledConnectionImpl.transaction(ConnectionPoolImpl.kt:388)
    at androidx.room.coroutines.PooledConnectionImpl.isRecycled(ConnectionPoolImpl.kt:0)
    at androidx.room.coroutines.PooledConnectionImpl.access$isRecycled(ConnectionPoolImpl.kt:0)
    at androidx.room.coroutines.PooledConnectionImpl.withTransaction(ConnectionPoolImpl.kt:0)
    at androidx.room.coroutines.PooledConnectionImpl.withTransaction(ConnectionPoolImpl.kt:344)
    at kotlin.coroutines.intrinsics.IntrinsicsKt__IntrinsicsKt.getCOROUTINE_SUSPENDED
    at androidx.room.util.DBUtil__DBUtil_androidKt$performInTransactionSuspending$3$invokeSuspend$$inlined$internalPerform$1.invokeSuspend(DBUtil.kt:0)
    at androidx.room.util.DBUtil__DBUtil_androidKt$performInTransactionSuspending$3$invokeSuspend$$inlined$internalPerform$1.invokeSuspend(DBUtil.kt:59)
    at androidx.room.util.DBUtil__DBUtil_androidKt$performInTransactionSuspending$3$invokeSuspend$$inlined$internalPerform$1.invoke(DBUtil.kt:0)
    at androidx.room.util.DBUtil__DBUtil_androidKt$performInTransactionSuspending$3$invokeSuspend$$inlined$internalPerform$1.invoke(DBUtil.kt:0)
    at kotlin.coroutines.intrinsics.IntrinsicsKt__IntrinsicsKt.getCOROUTINE_SUSPENDED
    at androidx.room.coroutines.ConnectionPoolImpl$useConnection$4.invokeSuspend(ConnectionPoolImpl.kt:0)
    at androidx.room.coroutines.ConnectionPoolImpl$useConnection$4.invokeSuspend(ConnectionPoolImpl.kt:144)
    at androidx.room.coroutines.ConnectionPoolImpl$useConnection$4.invoke(ConnectionPoolImpl.kt:0)
    at androidx.room.coroutines.ConnectionPoolImpl$useConnection$4.invoke(ConnectionPoolImpl.kt:0)
    at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(ChildStackFactory.kt:43)
    at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext
    at kotlinx.coroutines.BuildersKt.withContext(DebugStrings.kt:0)
    at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext
    at kotlinx.coroutines.BuildersKt.withContext(DebugStrings.kt:1)
    at androidx.room.coroutines.ConnectionPoolImpl.useConnection(ConnectionPoolImpl.kt:144)
    at androidx.room.RoomDatabase.useConnection$room_runtime_release(RoomDatabase.android.kt:0)
    at androidx.room.RoomConnectionManager.useConnection
    at androidx.room.RoomDatabase.useConnection$room_runtime_release(RoomDatabase.android.kt:587)
    at kotlin.coroutines.intrinsics.IntrinsicsKt__IntrinsicsKt.getCOROUTINE_SUSPENDED
    at androidx.room.util.DBUtil__DBUtil_androidKt$performInTransactionSuspending$3.invokeSuspend(DBUtil.android.kt:0)
    at androidx.room.util.DBUtil__DBUtil_androidKt$performInTransactionSuspending$3.invokeSuspend(DBUtil.android.kt:243)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:0)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
    at java.lang.Thread.run(Thread.java:1012)

Hi. I got a new error: Fatal Exception: android.database.SQLException: Error code: 5, message: Timed out attempting to acquire a reader connection.

Writer pool:
	Xu1@d088033 (capacity=1)
		[1] - ow@e204ff0
		Status: Free connection
Reader pool:
	Xu1@6c0ec8f (capacity=4)
		[1] - ow@dd27b1c
		Status: Acquired connection
		Coroutine: [go2@7c92325, eo2{Active}@cf76cfa, Dispatchers.IO]
		Acquired:
			at lR.C(Unknown Source:310)
			at iR.invokeSuspend(Unknown Source:13)
			at dq.resumeWith(Unknown Source:8)
			at TR1.o(Unknown Source:6)
			at W.resumeWith(Unknown Source:22)
			at dq.resumeWith(Unknown Source:31)
			at xc0.run(Unknown Source:109)
			at K2.run(Unknown Source:1024)
			at qc2.run(Unknown Source:2)
			at SV.run(Unknown Source:93)
		
		[2] - ow@71a2d08
		Status: Free connection
		[3] - ow@c0dcca1
		Status: Free connection
		[4] - ow@bbf5ec6
		Status: Free connection

"Timed out attempting to acquire a reader connection" is a better error than "database is locked", hehe

It seems you have 3 zombie connections in your pool. I suspect this is because currently don't refill the pool when a connection dies. I'll work on a change to make sure the pool is always of the correct size. Thanks again for your patience on helping us make the library better.

Project: platform/frameworks/support
Branch: androidx-main
Author: Daniel Santiago Rivera <danysantiago@google.com>
Link:      https://android-review.googlesource.com/3546974

Reimplement suspending pool with Semaphore

Reimplement suspending pool with Semaphore 
 
This changes the implementation of Room's connection pool to use a Semaphore instead of a Channel. It makes the implementation simpler and removes the possibility of connections getting 'lost' due the the various branches needed to handle when a channel send / receive fails. 
 
The Channel was also used as a 'queue' instead a circular array is used to push and pop connections acquired and released into the pool. The semaphore still controls the amount of connections being locked and unlocked and is the suspending API that waits for a connection to be free. 
 
Bug: 322386871 
Bug: 380088809 
Test: BaseConnectionPoolTest 
Change-Id: I9b7f2d44b511eb3e3d8cdaf8cb8905bb6203f8c5

The next release (2.7.0-rc03) will have a change that should alleviate the 'Timed out attempting to acquire a reader connection' issue, I'll continue to keep this issue open so others can report if they are still experiencing it or not.

Hi. Thank you for your fixes. I have released 2.7.0-rc01 for 100% of users. For now it have about 22000 installs and I don't see the error SQLException: Error code: 5, message: database is locked anymore.