IssueTracker

Hello,

We recently launched an update that includes "androidx.work:work-runtime:2.0.1", and we used it to schedule all our tasks now, in a satisfactory way. I just want to inform, (as you can see in the images attached below), that this problem keeps happening sporadically.

Keep in mind that the number of users affected in our case is small, since the app has several million active users, but it also starts to be worrisome, since the number of errors generated by each user is high.

The errors mainly occur in Android 6 and 7, and sometimes in 8, but especially in Android 7.

I do not expose implementation details, since the documentation of this SDK has been thoroughly studied, and in case there was a bad implementation, the number of errors would be infinitely greater.

Greetings.

PD: If you need any information that I can provide you, you just have to ask me.

Thanks for re-uploading the screenshots.

A couple of questions:

* Can you reproduce this issue at all on your end ? If so, can you send us all WorkManager logs after turning on logging. For this you will have to use custom initialization and use Log.DEBUG. For more information look at https://developer.android.com/topic/libraries/architecture/workmanager/advanced/custom-configuration
* Can you send us a breakdown of devices and total # of users affected ? The number of crashes will be dis proportionally high because once the app gets into this state - its hard to get out of it.

Also from the first screenshot, it looks like you are also seeing IllegalStateException's in SystemJobService#onCreate().
The only reason that happens is if WorkManager is not been initialized for your process. Do you have multiple processes at all ?

Are you initializing WorkManager on a background thread or inside a coroutine ?

We're seeing this crash with WorkManager 1.0.1 on Android 7, 8 and 9. The interesting bit for us is that we released an urgent hot fix where we removed the use of WorkManager entirely but the crashe still happens. The best hypothesis I could come up is that the OS Scheduler is keeping this bad job running even after the update.

We didn't schedule the job in a coroutine nor in the background. We did in the Application.onCreate() and we always try to clean up the jobs for the predefined tag.

Seeing that this problem is still happening with WorkManager 2.0.1 is worrisome. Is there any advice and/or workaround for anyone in this situation?

Please send us logs for your crashes. They are super useful especially given you are seeing them on Android 8, 9.

To work around this issue for now you should probably do a https://developer.android.com/reference/android/app/job/JobScheduler.html#cancelAll() for now (inside Application.onCreate()).
Jobs don't really go away after an application update. So this step is important.

Also information about no of users affected will be useful (and %s). The number of crashes are going to be disproportionally high - so that is not a very useful number. That's because once a device gets into this state - it's hard to recover.

#7 do we have to check if it is a main-process or not before calling JobScheduler.cancelAll() or WorkManager.cancelAllWork()?

You can be on any thread when calling cancelAll() / cancelAllWork().

Questions and Answers:

#3 * Can you reproduce this issue at all on your end ?
We can not. We have done tests with several devices that we have here and it does not happen.

#3 * Can you send us a breakdown of devices and total # of users affected ?

Here you can see the devices and the total number of errors in the last 7 days for each one.

Moto C Plus (panell_s)  6,624   33.9%
Moto C (namath) 4,909   25.1%
Moto C Plus (panell_dl) 1,227   6.3%
Aquaris A4.5 (Aquaris_A45_sprout)       1,197   6.1%
Moto C Plus (panell_dt) 1,030   5.3%
LAIQ_Monaco     838     4.3%
Moto C Plus (panell_d)  686     3.5%
Alcatel 1C (U5A_PLUS_3G)        332     1.7%
4047G (U5_3G)   257     1.3%
BLADE A612 (P635F50)    236     1.2%
Xperia XA (F3111)       230     1.2%
HUAWEI Y5 2018 (HWDRA-MG)       201     1.0%
Aquaris M5 (Aquaris_M5) 179     0.9%
S550 (S550)     153     0.8%
Nokia 2 (E1M)   144     0.7%
F23 (HS6737MT)  134     0.7%
LG K20 Plus (lv517)     120     0.6%
iris80 (iris80) 96      0.5%
KIICAA MIX (KIICAA_MIX) 92      0.5%
Wooze L (Wooze_L)       78      0.4%
Others  788     4.0%

96% of these devices use Android 7.0
The total number of users affected by this error is 325 currently, which is a very very small percentage of active users with the latest update.


#4 * Also from the first screenshot, it looks like you are also seeing IllegalStateException's in SystemJobService#onCreate().
* The only reason that happens is if WorkManager is not been initialized for your process. Do you have multiple processes at all ?

We call "WorkManager.getInstance()...." whenever it is used, and no special initialization is made. It is called in 3 different places:

a) By using Broadcast,
Specifically when restarting the phone with: "android.intent.action.BOOT_COMPLETED"
and on the other hand when making an update with: "android.intent.action.MY_PACKAGE_REPLACED" or "android.intent.action.PACKAGE_REPLACED"

b) When starting the app (main thread).

c) Within a "Worker",
Some workers can start others.


#5 * Are you initializing WorkManager on a background thread or inside a coroutine ?

The first time it is called it could be inside a "BroadcastReceiver" if the user has just updated or restarted the phone. In any other case, it is called when you start the app.


#6 * ...The best hypothesis I could come up is that the OS Scheduler is keeping this bad job running even after the update.

We think that is what is currently happening.
In this update several "works" have been migrated to workmanager, which still continued using other old systems, specifically AlarmManager and FirebaseJobDispatcher in the previous version.

It seems that the errors are stabilizing, they are even receding. I will keep you informed, especially in future updates, when there is no change of works between versions.

I attached a crash report for this issue. Notice that even though we removed WorkManager from our build, this is one of the threads trace:

--------- thread trace ------------
Thread : 9955/WorkManager-WorkManagerTaskExecutor-thread-1
        isAlive : true
        isInterrupted : false
        isDaemon : false
                at android.database.CursorWindow.nativeDispose(Native Method)
                at android.database.CursorWindow.dispose(CursorWindow.java:175)
                at android.database.CursorWindow.onAllReferencesReleased(CursorWindow.java:730)
                at android.database.sqlite.SQLiteClosable.releaseReference(SQLiteClosable.java:74)
                at android.database.sqlite.SQLiteClosable.close(SQLiteClosable.java:106)
                at android.database.AbstractWindowedCursor.closeWindow(AbstractWindowedCursor.java:184)
                at android.database.AbstractWindowedCursor.onDeactivateOrClose(AbstractWindowedCursor.java:208)
                at android.database.AbstractCursor.close(AbstractCursor.java:170)
                at android.database.sqlite.SQLiteCursor.close(SQLiteCursor.java:214)
                at androidx.work.impl.b.j.a(WorkSpecDao_Impl.java:1211)
                at androidx.work.impl.i.b(WorkerWrapper.java:427)
                at androidx.work.impl.i.e(WorkerWrapper.java:393)
                at androidx.work.impl.i.d(WorkerWrapper.java:150)
                at androidx.work.impl.i.run(WorkerWrapper.java:128)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
                at java.lang.Thread.run(Thread.java:764)
-----------------------------------

This thread is the only thread on our report that is using workmanager somehow.


- # of users we rolled out the build with workmanager 1.0.1: 347
- # of users affected by this crash: 157
- Android versions affected by this crash: 7.0, 8.0 and 9.0

- # of users we rolled out the patch where we removed workmanager: 92
- # of users still seeing this crash: 47
- Android versions still affected by this crash: 7.0, 8.0 and 9.0

Another piece of information, these numbers are gather from Crashlitics (Fabric), and we notice a discrepancy between Crashlitics numbers and our internal tools.

Hi, this thread shouldn't exist if WorkManager isn't in your app.  So I'm not sure what's happening here, but you're still pulling in WorkManager somehow.  You should check your dependencies.

The dependency is still there so we could call `cancelAll` from the library. But none of our code is using any workmanager code. The next patch we have is to change that to a JobScheduler directly.

That won't work at all.

WorkManager does a lot of things including:
* Check for force-stops of your app and reschedule things automatically - without you having to do nothing.
* Handle existing onStartJob requests for persisted jobs in JobScheduler.

Hi Adrian, in #11 you mentioned:

#4 * Also from the first screenshot, it looks like you are also seeing IllegalStateException's in SystemJobService#onCreate().
* The only reason that happens is if WorkManager is not been initialized for your process. Do you have multiple processes at all ?

We call "WorkManager.getInstance()...." whenever it is used, and no special initialization is made. It is called in 3 different places:

This will not work. Because WorkManager is used in many places without you explicitly doing anything.
You should always ensure WorkManager is initialized in Application#onCreate() unless you happen to using on-demand initialization which is something we introduced in 2.1.0-alpha01.

Project: platform/frameworks/support
Branch: androidx-master-dev

commit d025815ea5cc023c0d33b810f99f5afb7cc40b85
Author: Sumir Kataria <sumir@google.com>
Date:   Fri May 31 14:15:54 2019

    Cancel invalid jobs attributed to WorkManager.
   
    Sometimes, there are invalid jobs returned in getAllPendingJobs() with no
    extras attributed to WorkManager.  These should be cleaned up proactively
    on startup.
   
    Fixes: 134058261
    Test: Added new test
    Change-Id: I9bdf821c589bc4ba99542307ea76ca4c0cb9ac2e

M       work/workmanager/src/androidTest/java/androidx/work/impl/background/systemjob/SystemJobSchedulerTest.java
M       work/workmanager/src/main/java/androidx/work/impl/WorkManagerImpl.java
M       work/workmanager/src/main/java/androidx/work/impl/background/systemjob/SystemJobScheduler.java
M       work/workmanager/src/main/java/androidx/work/impl/utils/ForceStopRunnable.java

https://android-review.googlesource.com/974683
https://goto.google.com/android-sha1/d025815ea5cc023c0d33b810f99f5afb7cc40b85

HI. Just wanted to add to this. We are also seeing this same thing with v 2.0.1. This is the first version we have used in production, so I cannot say if it occurred with previous versions. We had 4 unique, periodic jobs running in production, and had to do a hotfix removing all of them, the problem has subsided as we cancelled all of the pending jobs.

The reason I think it is the same bug is that we have a very similar breakdown of affected devices

100% people on Android 7
83% people on Motorola phones

Trace:

Fatal Exception: java.lang.IllegalStateException: JobScheduler 100 job limit exceeded.  We count 101 WorkManager jobs in JobScheduler; we have 4 tracked jobs in our DB; our Configuration limit is 20.
       at androidx.work.impl.background.systemjob.SystemJobScheduler.scheduleInternal(SystemJobScheduler.java:189)
       at androidx.work.impl.background.systemjob.SystemJobScheduler.schedule(SystemJobScheduler.java:129)
       at androidx.work.impl.Schedulers.schedule(Schedulers.java:93)
       at androidx.work.impl.utils.ForceStopRunnable.run(ForceStopRunnable.java:100)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
       at java.lang.Thread.run(Thread.java:761)

We have a very similar situation in our end, but we were using v1.0.1 with a crash rate of 40% across all devices and Android versions from API 21 and up.

In our case we rolled out an emergency fix to remove WorkManager entirely from our project, including the dependency on it, and the problem was not seen again. We didn't cancel the remaining jobs, but they're gone.

Hi, crash reports and device models are very useful.  There are some upcoming fixes in 2.1.0 that will address this; please make sure you try them out.

Re: #19, this crash should only occur on API 23 and up, so I'm not sure if there's something else going on there for you.  Unfortunately, we are chasing down some device differences that don't conform to what stock Android seems to do, and without more data.  Without identifying the models, it's hard to see test these.

Motorola devices in particular seem to exhibit this class of problems much more.