ANR/Deadlock [134361006]

Fixed

Bug

Status Update

No update yet.

Description

d....@googlemail.com

created issue #1

Jun 3, 2019 10:37AM

Component used: WorkManager
Version used: 1.0.1
Devices/Android versions reproduced on: different Samsung Galaxy devices running Android 8 and 9

I have received deadlock reports like this:

"main" prio=5 tid=1 Blocked
| group="main" sCount=1 dsCount=0 flags=1 obj=0x76790a58 self=0x7d81014c00
| sysTid=27874 nice=0 cgrp=default sched=0/0 handle=0x7e07c1a560
| state=S schedstat=( 1835900780 7285545518 5958 ) utm=82 stm=100 core=1 HZ=100
| stack=0x7febcdd000-0x7febcdf000 stackSize=8MB
| held mutexes=
at androidx.work.impl.constraints.trackers.ConstraintTracker.removeListener (ConstraintTracker.java:77)
- waiting to lock <0x08a4f10d> (a java.lang.Object) held by thread 13 // ConstraintTracker.mLock
at androidx.work.impl.constraints.controllers.ConstraintController.replace (ConstraintController.java:98)
at androidx.work.impl.constraints.WorkConstraintsTracker.replace (WorkConstraintsTracker.java:99)
- locked <0x0cbac8c2> (a java.lang.Object) // WorkConstrainsTracker.mLock
at androidx.work.impl.background.greedy.GreedyScheduler.removeConstraintTrackingFor (GreedyScheduler.java:153)
- locked <0x0cbec6d3> (a java.lang.Object) // GreedyScheduler.mLock
at androidx.work.impl.background.greedy.GreedyScheduler.onExecuted (GreedyScheduler.java:141)
at androidx.work.impl.Processor.onExecuted (Processor.java:230)
- locked <0x08012210> (a java.lang.Object)
at androidx.work.impl.Processor$FutureListener.run (Processor.java:263)
at android.os.Handler.handleCallback (Handler.java:873)
at android.os.Handler.dispatchMessage (Handler.java:99)
at android.os.Looper.loop (Looper.java:214)
at android.app.ActivityThread.main (ActivityThread.java:7032)
at java.lang.reflect.Method.invoke (Method.java)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run (RuntimeInit.java:494)
at com.android.internal.os.ZygoteInit.main (ZygoteInit.java:965)
"ConnectivityThread" tid=13 Blocked
"ConnectivityThread" prio=5 tid=13 Blocked
| group="main" sCount=1 dsCount=0 flags=1 obj=0x12fc0770 self=0x7d7ab86800
| sysTid=27897 nice=0 cgrp=default sched=0/0 handle=0x7d684114f0
| state=S schedstat=( 247567475 11539483672 1368 ) utm=16 stm=7 core=2 HZ=100
| stack=0x7d6830e000-0x7d68310000 stackSize=1041KB
| held mutexes=
at androidx.work.impl.a.d.c (SourceFile:157) // synchronized (mLock)
- waiting to lock <0x0cbac8c2> (a java.lang.Object) held by thread 1 // trying to lock WorkConstraintsTracker.mLock
at androidx.work.impl.a.a.c.b (SourceFile:133) ConstraintController: mCallback.onConstraintNotMet(mMatchingWorkSpecIds);
at androidx.work.impl.a.a.c.a (SourceFile:142) ConstraintController: updateCallback();
at androidx.work.impl.a.b.d.a (SourceFile:103) listener.onConstraintChanged(mCurrentState);
- locked <0x08a4f10d> (a java.lang.Object) (ConstraintTracker.mLock)
at androidx.work.impl.a.b.e$b.onCapabilitiesChanged (SourceFile:149) (NetworkStateTracker)
at android.net.ConnectivityManager$CallbackHandler.handleMessage (ConnectivityManager.java:3331)
at android.os.Handler.dispatchMessage (Handler.java:106)
at android.os.Looper.loop (Looper.java:214)
at android.os.HandlerThread.run (HandlerThread.java:65)

I have written after the lines what is happening there.

The GreedyScheduler has finished executing and onExecuted is called. This tries to remove the workSpecId from the mConstrainedWorkSpecs list. After removing it it calls mWorkConstraintsTracker.replace to send the new list to the WorkContraintsTracker. This causes a lock of WorkConstraintsTracker.mLock and a call to ConstraintController.replace. This calls ConstraintController.removeListener which tries to lock ConstraintController.mLock but fails/deadlocks. So as you can see it has first locked WorkConstraintsTracker.mLock and then it tries to lock ConstraintController.mLock.

As you can see now the other thread does the opposite, which leads to the deadlock: The WorkManager receives a "onCapabilitiesChanges" message and locks ConstraintTracker.mLock. Then it tries to call onConstraintNotMet but it cannot lock WorkConstraintsTracker.mLock, because it is already locked by the upper thread.

So there is a deadlock because both functions are running at the same time and are locking the locks in opposite order. Can you please fix it? Or did I make a mistake somewhere?

(BTW I am only using WorkManager.getInstance, I am not calling "initialize").

Thank you very much!

Comments

ra...@google.com <ra...@google.com> #2Jun 3, 2019 04:57PM

Assigned to ra...@google.com.

The stack trace doesnt seem to be complete.

Assigned to UI since the issue happened on UI update but compiler stayed the same.

It is possible it is "Can't represent a size of $size in Constraints". Assigned to see if it rings a bell.

ra...@google.com <ra...@google.com> #3Jun 3, 2019 05:11PM

The value represented here is Int.MAX_VALUE - 1 (with Constraints.Infinity being Int.MAX_VALUE). It may be possible we subtract 1 from that value, leading to this crash.

d....@googlemail.com <d....@googlemail.com> #4Jun 3, 2019 05:26PM

This looks like it may have been caused by changes to Row & Column

ra...@google.com <ra...@google.com> #5Jun 3, 2019 05:56PM

It looks like it's crashing on this line. The -1 is likely from remainderUnit, and the if statement indicates childMainAxisSize is expected to be sometimes Infinity.

This code was touched in the 1.4 timeframe in http://aosp/2260390, but that patch mostly moving around longstanding code so I'm not sure why the crash would be new. Is it possible the 1.4 upgrade merely changed the stacktrace for a preexisting crash? Did your total amount of crashes for all crashstacks increase with the upgrade?

Anyway, one idea for how to fix this would be to rewrite the codeblock right above from:

                val childMainAxisSize = max(
                    0,
                    (weightUnitSpace * weight).roundToInt() + remainderUnit
                )

to:

                val resolvedWeight = (weightUnitSpace * weight).roundToInt()
                val childMainAxisSize =
                    if (resolvedWeight == Constraints.Infinity) { Constraints.Infinity } else { 
                        max(0, resolvedWeight + remainderUnit)
                    }

d....@googlemail.com <d....@googlemail.com> #6Jun 3, 2019 09:38PM

The behavior has existed before the change in 1.4. The changes from 1.4 simply moves things around.

The change offered by Alex is a good solution to the issue. I'll see if we can have a Cl up shortly.

ra...@google.com <ra...@google.com> #7Jun 5, 2019 05:31PM

Is it possible the 1.4 upgrade merely changed the stacktrace for a preexisting crash? Did your total amount of crashes for all crashstacks increase with the upgrade?

Our total amount of crashes did increase with the upgrade. It would have caused us to miss stability targets so we rolled back to 1.3 for that release. After the 1.3 rollback the crashes stopped so it was definitely related to the upgrade to 1.4.3. The stacktrace in the initial report seemed to be coming from the measure of a material TopAppBar so we focused efforts on trying to figure out if there was something we were doing wrong there. However we got another trace from a composable that didn't have a TopAppBar in it so that sort of invalidated that theory. Here's that trace if it helps at all:

java.lang.IllegalArgumentException: Can't represent a size of 2147483646 in Constraints
        at androidx.compose.ui.unit.Constraints$Companion.bitsNeedForSize(Constraints.kt:403)
        at androidx.compose.ui.unit.Constraints$Companion.createConstraints-Zbe2FdA$ui_unit_release(Constraints.kt:366)
        at androidx.compose.ui.unit.ConstraintsKt.Constraints(Constraints.kt:433)
        at androidx.compose.foundation.layout.OrientationIndependentConstraints.toBoxConstraints-OenEA2s(RowColumnImpl.kt:339)
        at androidx.compose.foundation.layout.RowColumnMeasurementHelper.measureWithoutPlacing-_EkL_-Y(RowColumnMeasurementHelper.kt:180)
        at androidx.compose.foundation.layout.RowColumnImplKt$rowColumnMeasurePolicy$1.measure-3p2s80s(RowColumnImpl.kt:70)
        at androidx.compose.ui.node.InnerNodeCoordinator.measure-BRTryo0(InnerNodeCoordinator.kt:103)
        at androidx.compose.foundation.layout.FillModifier.measure-3p2s80s(Size.kt:658)
        at androidx.compose.ui.node.BackwardsCompatNode.measure-3p2s80s(BackwardsCompatNode.kt:323)
        at androidx.compose.ui.node.LayoutModifierNodeCoordinator.measure-BRTryo0(LayoutModifierNodeCoordinator.kt:155)
        at androidx.compose.foundation.layout.PaddingModifier.measure-3p2s80s(Padding.kt:364)
        at androidx.compose.ui.node.BackwardsCompatNode.measure-3p2s80s(BackwardsCompatNode.kt:323)
        at androidx.compose.ui.node.LayoutModifierNodeCoordinator.measure-BRTryo0(LayoutModifierNodeCoordinator.kt:155)
        at androidx.compose.ui.graphics.SimpleGraphicsLayerModifier.measure-3p2s80s(GraphicsLayerModifier.kt:635)
        at androidx.compose.ui.node.LayoutModifierNodeCoordinator.measure-BRTryo0(LayoutModifierNodeCoordinator.kt:155)
        at androidx.compose.ui.graphics.BlockGraphicsLayerModifier.measure-3p2s80s(GraphicsLayerModifier.kt:572)
        at androidx.compose.ui.node.LayoutModifierNodeCoordinator.measure-BRTryo0(LayoutModifierNodeCoordinator.kt:155)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate$performMeasure$2.invoke(LayoutNodeLayoutDelegate:1090)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate$performMeasure$2.invoke(LayoutNodeLayoutDelegate.kt:1086)
        at androidx.compose.runtime.snapshots.Snapshot$Companion.observe(Snapshot.kt:2200)
        at androidx.compose.runtime.snapshots.SnapshotStateObserver$observeReads$1$1.invoke(SnapshotStateObserver:234)
        at androidx.compose.runtime.snapshots.SnapshotStateObserver$observeReads$1$1.invoke(SnapshotStateObserver.kt:230)
        at androidx.compose.runtime.SnapshotStateKt__DerivedStateKt.observeDerivedStateRecalculations(DerivedState.kt:341)
        at androidx.compose.runtime.SnapshotStateKt.observeDerivedStateRecalculations(SnapshotStateKt:1)
        at androidx.compose.runtime.snapshots.SnapshotStateObserver.observeReads(SnapshotStateObserver.kt:230)
        at androidx.compose.ui.node.OwnerSnapshotObserver.observeReads$ui_release(OwnerSnapshotObserver.kt:120)
        at androidx.compose.ui.node.OwnerSnapshotObserver.observeMeasureSnapshotReads$ui_release(OwnerSnapshotObserver.kt:107)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate.performMeasure-BRTryo0(LayoutNodeLayoutDelegate:1086)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate.access$performMeasure-BRTryo0(LayoutNodeLayoutDelegate.kt:36)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate$MeasurePassDelegate.remeasure-BRTryo0(LayoutNodeLayoutDelegate.kt:342)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate$MeasurePassDelegate.measure-BRTryo0(LayoutNodeLayoutDelegate.kt:321)
        at androidx.compose.foundation.layout.BoxKt$boxMeasurePolicy$1.measure-3p2s80s(Box.kt:115)
        at androidx.compose.ui.node.InnerNodeCoordinator.measure-BRTryo0(InnerNodeCoordinator.kt:103)
        at androidx.compose.foundation.layout.PaddingModifier.measure-3p2s80s(Padding.kt:364)
        at androidx.compose.ui.node.BackwardsCompatNode.measure-3p2s80s(BackwardsCompatNode.kt:323)
        at androidx.compose.ui.node.LayoutModifierNodeCoordinator.measure-BRTryo0(LayoutModifierNodeCoordinator.kt:155)
        at androidx.compose.foundation.layout.FillModifier.measure-3p2s80s(Size.kt:658)
        at androidx.compose.ui.node.BackwardsCompatNode.measure-3p2s80s(BackwardsCompatNode.kt:323)
        at androidx.compose.ui.node.LayoutModifierNodeCoordinator.measure-BRTryo0(LayoutModifierNodeCoordinator.kt:155)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate$performMeasure$2.invoke(LayoutNodeLayoutDelegate:1090)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate$performMeasure$2.invoke(LayoutNodeLayoutDelegate.kt:1086)
        at androidx.compose.runtime.snapshots.Snapshot$Companion.observe(Snapshot.kt:2200)
        at androidx.compose.runtime.snapshots.SnapshotStateObserver$observeReads$1$1.invoke(SnapshotStateObserver:234)
        at androidx.compose.runtime.snapshots.SnapshotStateObserver$observeReads$1$1.invoke(SnapshotStateObserver.kt:230)
        at androidx.compose.runtime.SnapshotStateKt__DerivedStateKt.observeDerivedStateRecalculations(DerivedState.kt:341)
        at androidx.compose.runtime.SnapshotStateKt.observeDerivedStateRecalculations(SnapshotStateKt:1)
        at androidx.compose.runtime.snapshots.SnapshotStateObserver.observeReads(SnapshotStateObserver.kt:230)
        at androidx.compose.ui.node.OwnerSnapshotObserver.observeReads$ui_release(OwnerSnapshotObserver.kt:120)
        at androidx.compose.ui.node.OwnerSnapshotObserver.observeMeasureSnapshotReads$ui_release(OwnerSnapshotObserver.kt:107)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate.performMeasure-BRTryo0(LayoutNodeLayoutDelegate:1086)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate.access$performMeasure-BRTryo0(LayoutNodeLayoutDelegate.kt:36)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate$MeasurePassDelegate.remeasure-BRTryo0(LayoutNodeLayoutDelegate.kt:342)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate$MeasurePassDelegate.measure-BRTryo0(LayoutNodeLayoutDelegate.kt:321)
        at androidx.compose.ui.layout.RootMeasurePolicy.measure-3p2s80s(RootMeasurePolicy.kt:38)
        at androidx.compose.ui.node.InnerNodeCoordinator.measure-BRTryo0(InnerNodeCoordinator.kt:103)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate$performMeasure$2.invoke(LayoutNodeLayoutDelegate:1090)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate$performMeasure$2.invoke(LayoutNodeLayoutDelegate.kt:1086)
        at androidx.compose.runtime.snapshots.Snapshot$Companion.observe(Snapshot.kt:2200)
        at androidx.compose.runtime.snapshots.SnapshotStateObserver$observeReads$1$1.invoke(SnapshotStateObserver:234)
        at androidx.compose.runtime.snapshots.SnapshotStateObserver$observeReads$1$1.invoke(SnapshotStateObserver.kt:230)
        at androidx.compose.runtime.SnapshotStateKt__DerivedStateKt.observeDerivedStateRecalculations(DerivedState.kt:341)
        at androidx.compose.runtime.SnapshotStateKt.observeDerivedStateRecalculations(SnapshotStateKt:1)
        at androidx.compose.runtime.snapshots.SnapshotStateObserver.observeReads(SnapshotStateObserver.kt:230)
        at androidx.compose.ui.node.OwnerSnapshotObserver.observeReads$ui_release(OwnerSnapshotObserver.kt:120)
        at androidx.compose.ui.node.OwnerSnapshotObserver.observeMeasureSnapshotReads$ui_release(OwnerSnapshotObserver.kt:107)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate.performMeasure-BRTryo0(LayoutNodeLayoutDelegate:1086)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate.access$performMeasure-BRTryo0(LayoutNodeLayoutDelegate.kt:36)
        at androidx.compose.ui.node.LayoutNodeLayoutDelegate$MeasurePassDelegate.remeasure-BRTryo0(LayoutNodeLayoutDelegate.kt:342)
        at androidx.compose.ui.node.LayoutNode.remeasure-_Sx5XlM$ui_release(LayoutNode.kt:1188)
        at androidx.compose.ui.node.MeasureAndLayoutDelegate.doRemeasure-sdFAvZA(MeasureAndLayoutDelegate.kt:307)
        at androidx.compose.ui.node.MeasureAndLayoutDelegate.remeasureOnly(MeasureAndLayoutDelegate.kt:479)
        at androidx.compose.ui.node.MeasureAndLayoutDelegate.recurseRemeasure(MeasureAndLayoutDelegate.kt:357)
        at androidx.compose.ui.node.MeasureAndLayoutDelegate.measureOnly(MeasureAndLayoutDelegate.kt:348)
        at androidx.compose.ui.node.MeasureAndLayoutDelegate.measureOnly(MeasureAndLayoutDelegate.kt:348)
        at androidx.compose.ui.platform.AndroidComposeView.onMeasure(AndroidComposeView.android.kt:881)
        at android.view.View.measure(View.java:25161)
        at androidx.compose.ui.platform.AbstractComposeView.internalOnMeasure$ui_release(ComposeView.android.kt:302)
        at androidx.compose.ui.platform.AbstractComposeView.onMeasure(ComposeView.android.kt:289)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:7109)
        at android.widget.FrameLayout.onMeasure(FrameLayout.java:194)
        at com.google.android.material.snackbar.BaseTransientBottomBar$SnackbarBaseLayout.onMeasure(BaseTransientBottomBar.java:1257)
        at com.google.android.material.snackbar.Snackbar$SnackbarLayout.onMeasure(Snackbar.java:513)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:7109)
        at androidx.coordinatorlayout.widget.CoordinatorLayout.onMeasureChild(CoordinatorLayout.java:760)
        at androidx.coordinatorlayout.widget.CoordinatorLayout.onMeasure(CoordinatorLayout.java:833)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:7109)
        at android.widget.FrameLayout.onMeasure(FrameLayout.java:194)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:7109)
        at android.widget.FrameLayout.onMeasure(FrameLayout.java:194)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:7109)
        at android.widget.LinearLayout.measureChildBeforeLayout(LinearLayout.java:1580)
        at android.widget.LinearLayout.measureVertical(LinearLayout.java:849)
        at android.widget.LinearLayout.onMeasure(LinearLayout.java:728)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:7109)
        at android.widget.FrameLayout.onMeasure(FrameLayout.java:194)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:7109)
        at android.widget.FrameLayout.onMeasure(FrameLayout.java:194)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:7109)
        at android.widget.FrameLayout.onMeasure(FrameLayout.java:194)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:7109)
        at android.widget.FrameLayout.onMeasure(FrameLayout.java:194)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:7109)
        at android.widget.FrameLayout.onMeasure(FrameLayout.java:194)
        at android.view.View.measure(View.java:25161)
        at androidx.drawerlayout.widget.DrawerLayout.onMeasure(DrawerLayout.java:1156)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:7109)
        at android.widget.FrameLayout.onMeasure(FrameLayout.java:194)
        at androidx.appcompat.widget.ContentFrameLayout.onMeasure(ContentFrameLayout.java:145)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:7109)
        at android.widget.LinearLayout.measureChildBeforeLayout(LinearLayout.java:1580)
        at android.widget.LinearLayout.measureVertical(LinearLayout.java:849)
        at android.widget.LinearLayout.onMeasure(LinearLayout.java:728)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:7109)
        at android.widget.FrameLayout.onMeasure(FrameLayout.java:194)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:7109)
        at android.widget.LinearLayout.measureChildBeforeLayout(LinearLayout.java:1580)
        at android.widget.LinearLayout.measureVertical(LinearLayout.java:849)
        at android.widget.LinearLayout.onMeasure(LinearLayout.java:728)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:7109)
        at android.widget.FrameLayout.onMeasure(FrameLayout.java:194)
        at com.android.internal.policy.DecorView.onMeasure(DecorView.java:1121)
        at android.view.View.measure(View.java:25161)
        at android.view.ViewRootImpl.performMeasure(ViewRootImpl.java:3691)
        at android.view.ViewRootImpl.measureHierarchy(ViewRootImpl.java:2316)
        at android.view.ViewRootImpl.performTraversals(ViewRootImpl.java:2713)
        at android.view.ViewRootImpl.doTraversal(ViewRootImpl.java:2188)
        at android.view.ViewRootImpl$TraversalRunnable.run(ViewRootImpl.java:8918)
        at android.view.Choreographer$CallbackRecord.run(Choreographer.java:1280)
        at android.view.Choreographer.doCallbacks(Choreographer.java:1019)
        at android.view.Choreographer.doFrame(Choreographer.java:911)
        at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:1248)
        at android.os.Handler.handleCallback(Handler.java:900)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loop(Looper.java:219)
        at android.app.ActivityThread.main(ActivityThread.java:8676)
        at java.lang.reflect.Method.invoke(Method.java:-2)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:513)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1109)

ap...@google.com <ap...@google.com> #8Jun 5, 2019 06:15PM

Marked as fixed.

Not sure if it is relevant. The two instance I have seen of this crash, both originate from the same device model, but from 2 independent users: Brand:Samsung Model:Galaxy S7 Android 8.0.0