IssueTracker

Hello,

We recently launched an update that includes "androidx.work:work-runtime:2.0.1", and we used it to schedule all our tasks now, in a satisfactory way. I just want to inform, (as you can see in the images attached below), that this problem keeps happening sporadically.

Keep in mind that the number of users affected in our case is small, since the app has several million active users, but it also starts to be worrisome, since the number of errors generated by each user is high.

The errors mainly occur in Android 6 and 7, and sometimes in 8, but especially in Android 7.

I do not expose implementation details, since the documentation of this SDK has been thoroughly studied, and in case there was a bad implementation, the number of errors would be infinitely greater.

Greetings.

PD: If you need any information that I can provide you, you just have to ask me.

Thanks for re-uploading the screenshots.

A couple of questions:

* Can you reproduce this issue at all on your end ? If so, can you send us all WorkManager logs after turning on logging. For this you will have to use custom initialization and use Log.DEBUG. For more information look at https://developer.android.com/topic/libraries/architecture/workmanager/advanced/custom-configuration
* Can you send us a breakdown of devices and total # of users affected ? The number of crashes will be dis proportionally high because once the app gets into this state - its hard to get out of it.

Also from the first screenshot, it looks like you are also seeing IllegalStateException's in SystemJobService#onCreate().
The only reason that happens is if WorkManager is not been initialized for your process. Do you have multiple processes at all ?

Are you initializing WorkManager on a background thread or inside a coroutine ?

We're seeing this crash with WorkManager 1.0.1 on Android 7, 8 and 9. The interesting bit for us is that we released an urgent hot fix where we removed the use of WorkManager entirely but the crashe still happens. The best hypothesis I could come up is that the OS Scheduler is keeping this bad job running even after the update.

We didn't schedule the job in a coroutine nor in the background. We did in the Application.onCreate() and we always try to clean up the jobs for the predefined tag.

Seeing that this problem is still happening with WorkManager 2.0.1 is worrisome. Is there any advice and/or workaround for anyone in this situation?

Please send us logs for your crashes. They are super useful especially given you are seeing them on Android 8, 9.

To work around this issue for now you should probably do a https://developer.android.com/reference/android/app/job/JobScheduler.html#cancelAll() for now (inside Application.onCreate()).
Jobs don't really go away after an application update. So this step is important.

Also information about no of users affected will be useful (and %s). The number of crashes are going to be disproportionally high - so that is not a very useful number. That's because once a device gets into this state - it's hard to recover.

#7 do we have to check if it is a main-process or not before calling JobScheduler.cancelAll() or WorkManager.cancelAllWork()?

You can be on any thread when calling cancelAll() / cancelAllWork().

Questions and Answers:

#3 * Can you reproduce this issue at all on your end ?
We can not. We have done tests with several devices that we have here and it does not happen.

#3 * Can you send us a breakdown of devices and total # of users affected ?

Here you can see the devices and the total number of errors in the last 7 days for each one.

Moto C Plus (panell_s)  6,624   33.9%
Moto C (namath) 4,909   25.1%
Moto C Plus (panell_dl) 1,227   6.3%
Aquaris A4.5 (Aquaris_A45_sprout)       1,197   6.1%
Moto C Plus (panell_dt) 1,030   5.3%
LAIQ_Monaco     838     4.3%
Moto C Plus (panell_d)  686     3.5%
Alcatel 1C (U5A_PLUS_3G)        332     1.7%
4047G (U5_3G)   257     1.3%
BLADE A612 (P635F50)    236     1.2%
Xperia XA (F3111)       230     1.2%
HUAWEI Y5 2018 (HWDRA-MG)       201     1.0%
Aquaris M5 (Aquaris_M5) 179     0.9%
S550 (S550)     153     0.8%
Nokia 2 (E1M)   144     0.7%
F23 (HS6737MT)  134     0.7%
LG K20 Plus (lv517)     120     0.6%
iris80 (iris80) 96      0.5%
KIICAA MIX (KIICAA_MIX) 92      0.5%
Wooze L (Wooze_L)       78      0.4%
Others  788     4.0%

96% of these devices use Android 7.0
The total number of users affected by this error is 325 currently, which is a very very small percentage of active users with the latest update.


#4 * Also from the first screenshot, it looks like you are also seeing IllegalStateException's in SystemJobService#onCreate().
* The only reason that happens is if WorkManager is not been initialized for your process. Do you have multiple processes at all ?

We call "WorkManager.getInstance()...." whenever it is used, and no special initialization is made. It is called in 3 different places:

a) By using Broadcast,
Specifically when restarting the phone with: "android.intent.action.BOOT_COMPLETED"
and on the other hand when making an update with: "android.intent.action.MY_PACKAGE_REPLACED" or "android.intent.action.PACKAGE_REPLACED"

b) When starting the app (main thread).

c) Within a "Worker",
Some workers can start others.


#5 * Are you initializing WorkManager on a background thread or inside a coroutine ?

The first time it is called it could be inside a "BroadcastReceiver" if the user has just updated or restarted the phone. In any other case, it is called when you start the app.


#6 * ...The best hypothesis I could come up is that the OS Scheduler is keeping this bad job running even after the update.

We think that is what is currently happening.
In this update several "works" have been migrated to workmanager, which still continued using other old systems, specifically AlarmManager and FirebaseJobDispatcher in the previous version.

It seems that the errors are stabilizing, they are even receding. I will keep you informed, especially in future updates, when there is no change of works between versions.

I attached a crash report for this issue. Notice that even though we removed WorkManager from our build, this is one of the threads trace:

--------- thread trace ------------
Thread : 9955/WorkManager-WorkManagerTaskExecutor-thread-1
        isAlive : true
        isInterrupted : false
        isDaemon : false
                at android.database.CursorWindow.nativeDispose(Native Method)
                at android.database.CursorWindow.dispose(CursorWindow.java:175)
                at android.database.CursorWindow.onAllReferencesReleased(CursorWindow.java:730)
                at android.database.sqlite.SQLiteClosable.releaseReference(SQLiteClosable.java:74)
                at android.database.sqlite.SQLiteClosable.close(SQLiteClosable.java:106)
                at android.database.AbstractWindowedCursor.closeWindow(AbstractWindowedCursor.java:184)
                at android.database.AbstractWindowedCursor.onDeactivateOrClose(AbstractWindowedCursor.java:208)
                at android.database.AbstractCursor.close(AbstractCursor.java:170)
                at android.database.sqlite.SQLiteCursor.close(SQLiteCursor.java:214)
                at androidx.work.impl.b.j.a(WorkSpecDao_Impl.java:1211)
                at androidx.work.impl.i.b(WorkerWrapper.java:427)
                at androidx.work.impl.i.e(WorkerWrapper.java:393)
                at androidx.work.impl.i.d(WorkerWrapper.java:150)
                at androidx.work.impl.i.run(WorkerWrapper.java:128)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
                at java.lang.Thread.run(Thread.java:764)
-----------------------------------

This thread is the only thread on our report that is using workmanager somehow.


- # of users we rolled out the build with workmanager 1.0.1: 347
- # of users affected by this crash: 157
- Android versions affected by this crash: 7.0, 8.0 and 9.0

- # of users we rolled out the patch where we removed workmanager: 92
- # of users still seeing this crash: 47
- Android versions still affected by this crash: 7.0, 8.0 and 9.0

Another piece of information, these numbers are gather from Crashlitics (Fabric), and we notice a discrepancy between Crashlitics numbers and our internal tools.

Hi, this thread shouldn't exist if WorkManager isn't in your app.  So I'm not sure what's happening here, but you're still pulling in WorkManager somehow.  You should check your dependencies.