IssueTracker

Yigit, do you have time to fix it?

reemission of the same liveData is racy

yea i'll take it.

Thanks for the detailed analysis. This may not be an issue anymore since we've started using Main.immediate there but I' not sure; I'll try to create a test case.

just emitting same live data reproduces the issue.

    @Test
    fun raceTest() {
        val subLiveData = MutableLiveData(1)
        val subject = liveData(testScope.coroutineContext) {
            emitSource(subLiveData)
            emitSource(subLiveData) //crashes
        }
        subject.addObserver().apply {
            testScope.advanceUntilIdle()
        }
    }

With 2.2.0-alpha04 (that use Main.immediate), the issue seems to be still there (I tested it by calling emitSource() twice, like your test case)

yea sorry immediate does not fix it.
I actually have a WIP fix for it:
https://android-review.googlesource.com/c/platform/frameworks/support/+/1112186

if your case is the one i found (emitting same LiveData multiple times, as shown in #5) you can work around it by adding a dummy transformation.


val subLiveData = MutableLiveData(1)
        val subject = liveData(testScope.coroutineContext) {
            emitSource(subLiveData.map {it })
            emitSource(subLiveData.map {it} )
        }

Project: platform/frameworks/support
Branch: androidx-master-dev

commit af12e75e6b4110f48e44ca121466943909de8f06
Author: Yigit Boyar <yboyar@google.com>
Date:   Tue Sep 03 12:58:11 2019

    Fix coroutine livedata race condition
   
    This CL fixes a bug in liveData builder where emitting same
    LiveData source twice would make it crash because the second
    emission registry could possibly happen before first one is
    removed as source.
   
    We fix it by using a suspending dispose function. It does feel
    a bit hacky but we cannot make DisposableHandle.dispose async
    and we do not want to block there. This does not mean that there
    is a problem if developer disposes it manually since our emit
    functions take care of making sure it disposes (and there is
    no other way to add source to the underlying MediatorLiveData)
   
    Bug: 140249349
    Test: BuildLiveDataTest#raceTest_*
   
    Change-Id: I0b464c242a583da4669af195cf2504e2adc4de40

M       lifecycle/lifecycle-livedata-ktx/api/2.2.0-alpha05.txt
M       lifecycle/lifecycle-livedata-ktx/api/current.txt
M       lifecycle/lifecycle-livedata-ktx/api/public_plus_experimental_2.2.0-alpha05.txt
M       lifecycle/lifecycle-livedata-ktx/api/public_plus_experimental_current.txt
M       lifecycle/lifecycle-livedata-ktx/api/restricted_2.2.0-alpha05.txt
M       lifecycle/lifecycle-livedata-ktx/api/restricted_current.txt
M       lifecycle/lifecycle-livedata-ktx/src/main/java/androidx/lifecycle/CoroutineLiveData.kt
M       lifecycle/lifecycle-livedata-ktx/src/test/java/androidx/lifecycle/BuildLiveDataTest.kt

https://android-review.googlesource.com/1112186
https://goto.google.com/android-sha1/af12e75e6b4110f48e44ca121466943909de8f06

#7 do we have to check if it is a main-process or not before calling JobScheduler.cancelAll() or WorkManager.cancelAllWork()?

You can be on any thread when calling cancelAll() / cancelAllWork().

Questions and Answers:

#3 * Can you reproduce this issue at all on your end ?
We can not. We have done tests with several devices that we have here and it does not happen.

#3 * Can you send us a breakdown of devices and total # of users affected ?

Here you can see the devices and the total number of errors in the last 7 days for each one.

Moto C Plus (panell_s)  6,624   33.9%
Moto C (namath) 4,909   25.1%
Moto C Plus (panell_dl) 1,227   6.3%
Aquaris A4.5 (Aquaris_A45_sprout)       1,197   6.1%
Moto C Plus (panell_dt) 1,030   5.3%
LAIQ_Monaco     838     4.3%
Moto C Plus (panell_d)  686     3.5%
Alcatel 1C (U5A_PLUS_3G)        332     1.7%
4047G (U5_3G)   257     1.3%
BLADE A612 (P635F50)    236     1.2%
Xperia XA (F3111)       230     1.2%
HUAWEI Y5 2018 (HWDRA-MG)       201     1.0%
Aquaris M5 (Aquaris_M5) 179     0.9%
S550 (S550)     153     0.8%
Nokia 2 (E1M)   144     0.7%
F23 (HS6737MT)  134     0.7%
LG K20 Plus (lv517)     120     0.6%
iris80 (iris80) 96      0.5%
KIICAA MIX (KIICAA_MIX) 92      0.5%
Wooze L (Wooze_L)       78      0.4%
Others  788     4.0%

96% of these devices use Android 7.0
The total number of users affected by this error is 325 currently, which is a very very small percentage of active users with the latest update.


#4 * Also from the first screenshot, it looks like you are also seeing IllegalStateException's in SystemJobService#onCreate().
* The only reason that happens is if WorkManager is not been initialized for your process. Do you have multiple processes at all ?

We call "WorkManager.getInstance()...." whenever it is used, and no special initialization is made. It is called in 3 different places:

a) By using Broadcast,
Specifically when restarting the phone with: "android.intent.action.BOOT_COMPLETED"
and on the other hand when making an update with: "android.intent.action.MY_PACKAGE_REPLACED" or "android.intent.action.PACKAGE_REPLACED"

b) When starting the app (main thread).

c) Within a "Worker",
Some workers can start others.


#5 * Are you initializing WorkManager on a background thread or inside a coroutine ?

The first time it is called it could be inside a "BroadcastReceiver" if the user has just updated or restarted the phone. In any other case, it is called when you start the app.


#6 * ...The best hypothesis I could come up is that the OS Scheduler is keeping this bad job running even after the update.

We think that is what is currently happening.
In this update several "works" have been migrated to workmanager, which still continued using other old systems, specifically AlarmManager and FirebaseJobDispatcher in the previous version.

It seems that the errors are stabilizing, they are even receding. I will keep you informed, especially in future updates, when there is no change of works between versions.

I attached a crash report for this issue. Notice that even though we removed WorkManager from our build, this is one of the threads trace:

--------- thread trace ------------
Thread : 9955/WorkManager-WorkManagerTaskExecutor-thread-1
        isAlive : true
        isInterrupted : false
        isDaemon : false
                at android.database.CursorWindow.nativeDispose(Native Method)
                at android.database.CursorWindow.dispose(CursorWindow.java:175)
                at android.database.CursorWindow.onAllReferencesReleased(CursorWindow.java:730)
                at android.database.sqlite.SQLiteClosable.releaseReference(SQLiteClosable.java:74)
                at android.database.sqlite.SQLiteClosable.close(SQLiteClosable.java:106)
                at android.database.AbstractWindowedCursor.closeWindow(AbstractWindowedCursor.java:184)
                at android.database.AbstractWindowedCursor.onDeactivateOrClose(AbstractWindowedCursor.java:208)
                at android.database.AbstractCursor.close(AbstractCursor.java:170)
                at android.database.sqlite.SQLiteCursor.close(SQLiteCursor.java:214)
                at androidx.work.impl.b.j.a(WorkSpecDao_Impl.java:1211)
                at androidx.work.impl.i.b(WorkerWrapper.java:427)
                at androidx.work.impl.i.e(WorkerWrapper.java:393)
                at androidx.work.impl.i.d(WorkerWrapper.java:150)
                at androidx.work.impl.i.run(WorkerWrapper.java:128)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
                at java.lang.Thread.run(Thread.java:764)
-----------------------------------

This thread is the only thread on our report that is using workmanager somehow.


- # of users we rolled out the build with workmanager 1.0.1: 347
- # of users affected by this crash: 157
- Android versions affected by this crash: 7.0, 8.0 and 9.0

- # of users we rolled out the patch where we removed workmanager: 92
- # of users still seeing this crash: 47
- Android versions still affected by this crash: 7.0, 8.0 and 9.0

Another piece of information, these numbers are gather from Crashlitics (Fabric), and we notice a discrepancy between Crashlitics numbers and our internal tools.

Hi, this thread shouldn't exist if WorkManager isn't in your app.  So I'm not sure what's happening here, but you're still pulling in WorkManager somehow.  You should check your dependencies.

The dependency is still there so we could call `cancelAll` from the library. But none of our code is using any workmanager code. The next patch we have is to change that to a JobScheduler directly.

That won't work at all.

WorkManager does a lot of things including:
* Check for force-stops of your app and reschedule things automatically - without you having to do nothing.
* Handle existing onStartJob requests for persisted jobs in JobScheduler.

Hi Adrian, in #11 you mentioned:

#4 * Also from the first screenshot, it looks like you are also seeing IllegalStateException's in SystemJobService#onCreate().
* The only reason that happens is if WorkManager is not been initialized for your process. Do you have multiple processes at all ?

We call "WorkManager.getInstance()...." whenever it is used, and no special initialization is made. It is called in 3 different places:

This will not work. Because WorkManager is used in many places without you explicitly doing anything.
You should always ensure WorkManager is initialized in Application#onCreate() unless you happen to using on-demand initialization which is something we introduced in 2.1.0-alpha01.

Project: platform/frameworks/support
Branch: androidx-master-dev

commit d025815ea5cc023c0d33b810f99f5afb7cc40b85
Author: Sumir Kataria <sumir@google.com>
Date:   Fri May 31 14:15:54 2019

    Cancel invalid jobs attributed to WorkManager.
   
    Sometimes, there are invalid jobs returned in getAllPendingJobs() with no
    extras attributed to WorkManager.  These should be cleaned up proactively
    on startup.
   
    Fixes: 134058261
    Test: Added new test
    Change-Id: I9bdf821c589bc4ba99542307ea76ca4c0cb9ac2e

M       work/workmanager/src/androidTest/java/androidx/work/impl/background/systemjob/SystemJobSchedulerTest.java
M       work/workmanager/src/main/java/androidx/work/impl/WorkManagerImpl.java
M       work/workmanager/src/main/java/androidx/work/impl/background/systemjob/SystemJobScheduler.java
M       work/workmanager/src/main/java/androidx/work/impl/utils/ForceStopRunnable.java

https://android-review.googlesource.com/974683
https://goto.google.com/android-sha1/d025815ea5cc023c0d33b810f99f5afb7cc40b85

HI. Just wanted to add to this. We are also seeing this same thing with v 2.0.1. This is the first version we have used in production, so I cannot say if it occurred with previous versions. We had 4 unique, periodic jobs running in production, and had to do a hotfix removing all of them, the problem has subsided as we cancelled all of the pending jobs.

The reason I think it is the same bug is that we have a very similar breakdown of affected devices

100% people on Android 7
83% people on Motorola phones

Trace:

Fatal Exception: java.lang.IllegalStateException: JobScheduler 100 job limit exceeded.  We count 101 WorkManager jobs in JobScheduler; we have 4 tracked jobs in our DB; our Configuration limit is 20.
       at androidx.work.impl.background.systemjob.SystemJobScheduler.scheduleInternal(SystemJobScheduler.java:189)
       at androidx.work.impl.background.systemjob.SystemJobScheduler.schedule(SystemJobScheduler.java:129)
       at androidx.work.impl.Schedulers.schedule(Schedulers.java:93)
       at androidx.work.impl.utils.ForceStopRunnable.run(ForceStopRunnable.java:100)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
       at java.lang.Thread.run(Thread.java:761)

We have a very similar situation in our end, but we were using v1.0.1 with a crash rate of 40% across all devices and Android versions from API 21 and up.

In our case we rolled out an emergency fix to remove WorkManager entirely from our project, including the dependency on it, and the problem was not seen again. We didn't cancel the remaining jobs, but they're gone.

Hi, crash reports and device models are very useful.  There are some upcoming fixes in 2.1.0 that will address this; please make sure you try them out.

Re: #19, this crash should only occur on API 23 and up, so I'm not sure if there's something else going on there for you.  Unfortunately, we are chasing down some device differences that don't conform to what stock Android seems to do, and without more data.  Without identifying the models, it's hard to see test these.

Motorola devices in particular seem to exhibit this class of problems much more.

#19, also in general crash rate as a metric is not a useful one here. We know once these devices get into this state, its hard to recover.
Can you take a look at your crashes to see the total number of users affected ? It will be useful to know what devices they are on as well.

Sorry for spam this issue, but ther eis just 6 starred people, so I think it's ok.

Could you please look at: https://issuetracker.google.com/issues/63622293 (200 starred people, for a 2 years).
And some next info in https://issuetracker.google.com/issues/115560697 60 starred people for a year).

We use now 2.1.0-alpha03 and this crash (100 jobs) looks fixed for us. But https://issuetracker.google.com/issues/63622293 still happen on almost all platforms and it's not possible to workaround it, intead of this one which was possible workaround by cancelling all jobs after start app.

Thanks a lot

You posted screenshots of your build.gradle file and it's using WorkManager 2.1.0-alpha03. (https://issuetracker.google.com/issues//63622293)

I am not sure i understand why you are conflating WorkManager and JobIntentService here.

The crash in JobIntentService and your usage of 2.1.0-alpha03 are not related at all.

I'm sorry, I and some next pelople think, that JobIntentService is related to Workmanager, because before WorkManager we used JobScheduler and JobIntentService... etc, still have mess around jobs and notifications and background services...

We believe we're seeing 1M crashes a day from this problem. Would be great to get this fixed quickly.

#26: It would be great to get some more details as we believe the problem has been addressed (this bug is marked as fixed).  What version of WorkManager are you using?  Can you give more details?

Hi aarmstro@

Which version of WorkManager are you using. This bug has been marked fixed for around ~3 weeks. Are you using the new beta ?

I believe we are not yet using the beta (since we generally try to run only stable libraries for our production app). We'll investigate the fix there though. Do you feel confident that the beta is stable enough for large apps to use?

Google apps usually take drops from HEAD all the time. So I would say yes.

 If you are uncomfortable then you can always try using it in a beta
rollout of the app.

On Thu, Jun 20, 2019 at 5:45 PM <buganizer-system@google.com> wrote: