[API] Add ability to inject other forms of input than touch input (e.g. mouse, stylus) [190493367]

Assigned

Feature Request

Status Update

No update yet.

Description

je...@google.com

created issue #1

Jun 8, 2021 02:59PM

For touch input we currently have `SemanticsNodeInteraction.performGesture { /* GestureScope functions */ }`.

This pattern should be extended/generalized to include other forms of input injection, primarily mouse input.

Comments

je...@google.com <je...@google.com> #2Jul 29, 2021 12:34PM

First of all thanks for this detailed issue.

This issue had been investigated thoroughly when it was first reported internally. The surprising detail in this report is that the issue is not reproducible before 1.7. I will look into this.

The main problem with POBox is the fact that it is deprecated. Since 2021 Sony has been shipping new Xperia devices with Gboard pre-installed. Although we are aware that there is still a considerable amount of users still using POBox, the described behavior is caused by POBox's noncompliant behavior with InputConnection and InputMethodManager documentation. However, this is understandable since TextView implementation was also not respecting the behavior that is expected from Editors.

Ultimately we have decided to enforce the documented behavior with specifically regards to when editors should call InputMethodManager.updateSelection. Also, although unconfirmed, there were traces of possible custom code being included in Sony OEM images that changed how InputMethodManager was notified from TextView. If POBox also depended on something like this, it would be impossible for Compose code to replicate the same unknown behavior.

ap...@google.com <ap...@google.com> #3Aug 2, 2021 10:38AM

If the investigation takes time, would it be possible to revert the surrounding implementation to 1.6.x equivalent?

Or is that option not available?

Even if the root cause is POBox, from the perspective of the app's customers, it looks like an app bug, so this issue is a blocker against updating Jetpack Compose.

ap...@google.com <ap...@google.com> #4Aug 11, 2021 11:32AM

Just to be sure, it is dangerous to replace Compose TextField with Android View EditText as a workaround for this issue.

Compose 1.7 has a bug that causes ANR when the focus is on EditText. https://issuetracker.google.com/issues/369354336

Another View-related bug in Compose 1.7 is that an Android View is focused by calling FocusManager.clearFocus(). https://issuetracker.google.com/issues/374031296

Perhaps there is a lack of testing of Compose 1.7 in combination with Android View. There is also a possibility that there are other fatal bugs related to View.

In other words, the only options for apps targeting the Japanese market that require POBox support are to continue using Compose 1.6 or to use EditText in combination with various workarounds.

ap...@google.com <ap...@google.com> #5Aug 26, 2021 09:33AM

Project: platform/frameworks/support
Branch: androidx-main
Author: Halil Ozercan <halilibo@google.com>
Link: https://android-review.googlesource.com/3317256

Fix POBox keyboard issue

Expand for full commit details

Fix POBox keyboard issue 
 
Fix: 373743376 
Fix: 329209241 
Test: NullableInputConnectionWrapperTest 
Change-Id: I94e0e598274fb88b255f977f9fbd50dfbbb1ecb1

Files:

M compose/ui/ui/src/androidInstrumentedTest/kotlin/androidx/compose/ui/text/input/NullableInputConnectionWrapperTest.kt
M compose/ui/ui/src/androidMain/kotlin/androidx/compose/ui/text/input/NullableInputConnectionWrapper.android.kt

Hash: 57f58c4b80d5d8470b2aca325dfdcd55f235231e
Date: Thu Oct 24 01:25:20 2024

ap...@google.com <ap...@google.com> #6Sep 2, 2021 01:51AM

Many thanks again for this report. Especially for giving us a huge clue in terms of what could be going wrong. The fix is now merged and I will ask for a cherry-pick into a stable release.

je...@google.com <je...@google.com> #7Sep 16, 2021 10:59AM

Thank you for fixing the issue, I confirmed the fix was released in version 1.8.0-alpha05.
Do you have any concrete plan to cherry-pick the fix into current stable version (1.7.x)? We are currently waiting it.

Message last modified on Sep 16, 2021 11:03AM

screenshot-mouse-events.png

858 KB

View

Download

ap...@google.com <ap...@google.com> #8Sep 16, 2021 11:10AM

Yes, this fix is planned to be included in a future 1.7.x release.

ap...@google.com <ap...@google.com> #9Sep 16, 2021 05:18PM

Thanks for the fix. Sorry to follow up on this. is it possible for you to share specific release version/date for the stable version? We are waiting on this to decide on our direction.

je...@google.com <je...@google.com> Sep 23, 2021 08:43AM

Accepted by je...@google.com.

ap...@google.com <ap...@google.com> #10Sep 23, 2021 09:03AM

Project: platform/frameworks/support
Branch: androidx-main

commit c6884d4c8e23a6914db4eb1afd077df119db45a6
Author: Jelle Fresen <jellefresen@google.com>
Date: Fri Sep 17 18:24:03 2021

Make MultiModalInjectionScope an interface

Which is implemented by MultiModalInjectionScopeImpl. The interface is
sealed to allow future addition of properties, and the only
implementation is internal so exhaustive when over the type of an
instance of that interface is forced to include an else block.

It will also improve layering of the API by preventing implementation
details to be leaked into the public API of MultiModalInjectionScope,
which would happen for example when it decides to implement Density by
delegation.

Bug: 190493367
Test: ./gradlew compose:ui:ui-test:cC && \
./gradlew compose:ui:ui-test:test
Relnote: N/A
Change-Id: Ib76c2ccbfe6393dc3397dfc7c3a0ec08ff4cd842

M compose/ui/ui-test/api/current.txt
M compose/ui/ui-test/api/public_plus_experimental_current.txt
M compose/ui/ui-test/src/commonMain/kotlin/androidx/compose/ui/test/GestureScope.kt
M compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/inputdispatcher/TouchUpTest.kt
M compose/ui/ui-test/src/commonMain/kotlin/androidx/compose/ui/test/MultiModalInjectionScope.kt
M compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/partialgesturescope/SendCancelTest.kt
M compose/ui/ui-test/api/restricted_current.txt
M compose/ui/ui-test/src/commonMain/kotlin/androidx/compose/ui/test/Actions.kt
M compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/inputdispatcher/InputDispatcherTest.kt
M compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/injectionscope/touch/UpTest.kt
M compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/inputdispatcher/TouchCancelTest.kt
M compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/injectionscope/touch/CancelTest.kt
M compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/partialgesturescope/SendUpTest.kt

https://android-review.googlesource.com/1831652

ap...@google.com <ap...@google.com> #11Sep 24, 2021 05:40PM

Project: platform/frameworks/support
Branch: androidx-main

commit d3d5b9fc5b602ab9edc83c921e861e6dd9deacde
Author: Jelle Fresen <jellefresen@google.com>
Date: Wed Sep 08 17:53:29 2021

Move InputDispatcher tests to host side tests

Because some code is used from both host side tests and instrumented
tests, a shared folder has been added that is included in both the host
side test variant and the instrumented test variant.

Bug: 190493367
Test: ./gradlew compose:ui:ui-test:test
Change-Id: I114d9f0b6462caf3c3267b6d81d2fe3c90a44024

M compose/ui/ui-test/src/test/kotlin/androidx/compose/ui/test/inputdispatcher/MouseEventsTest.kt
D compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/inputdispatcher/TouchUpTest.kt
A compose/ui/ui-test/src/test/kotlin/androidx/compose/ui/test/Constants.kt
A compose/ui/ui-test/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker
M compose/ui/ui-test/src/test/kotlin/androidx/compose/ui/test/inputdispatcher/BatchingTest.kt
M compose/ui/ui-test/src/test/kotlin/androidx/compose/ui/test/inputdispatcher/IsGestureInProgressTest.kt
M compose/ui/ui-test/build.gradle
M compose/ui/ui-test/src/test/kotlin/androidx/compose/ui/test/util/SplitsDurationEquallyIntoTest.kt
D compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/util/Truth.kt
M compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/injectionscope/touch/UpTest.kt
M compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/injectionscope/touch/CancelTest.kt
M compose/ui/ui-test/src/test/kotlin/androidx/compose/ui/test/util/IsMonotonicBetweenTest.kt
A compose/ui/ui-test/src/test/kotlin/androidx/compose/ui/test/inputdispatcher/TouchEventsTest.kt
A compose/ui/ui-test/src/androidCommonTest/kotlin/androidx/compose/ui/test/util/MotionEventRecorder.kt
D compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/inputdispatcher/TouchDownTest.kt
M compose/ui/ui-test/src/androidCommonTest/kotlin/androidx/compose/ui/test/util/InputDispatcherTestRule.kt
M compose/ui/ui-test/src/test/kotlin/androidx/compose/ui/test/inputdispatcher/InputDispatcherTest.kt
M compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/partialgesturescope/SendCancelTest.kt
M compose/ui/ui-test/src/test/kotlin/androidx/compose/ui/test/inputdispatcher/AdvanceEventTimeTest.kt
M compose/ui/ui-test/src/androidCommonTest/kotlin/androidx/compose/ui/test/util/Verifications.kt
D compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/inputdispatcher/TouchCancelTest.kt
A compose/ui/ui-test/src/androidCommonTest/README.md
M compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/partialgesturescope/SendUpTest.kt
D compose/ui/ui-test/src/androidAndroidTest/kotlin/androidx/compose/ui/test/inputdispatcher/TouchMoveTest.kt

https://android-review.googlesource.com/1822243

ap...@google.com <ap...@google.com> #12Oct 6, 2021 09:54AM

Project: platform/frameworks/support
Branch: androidx-main

commit be6e11888b09d95b6064935562aeb51d05f01cf2
Author: Jelle Fresen <jellefresen@google.com>
Date: Wed Sep 08 18:44:15 2021

Consider root bounds when injecting mouse events

When going out of bounds, send an exit event and stop sending more
events. Only allow cancel events and events that are part of an ongoing
gesture.

Bug: 190493367
Test: ./gradlew compose:ui:ui-test:test
Change-Id: I5372348a9731ee339ca194c728a0ba80c60ab784

M compose/ui/ui-test/src/commonMain/kotlin/androidx/compose/ui/test/InputDispatcher.kt
M compose/ui/ui-test/src/test/kotlin/androidx/compose/ui/test/inputdispatcher/MouseEventsTest.kt
M compose/ui/ui-test/src/androidMain/kotlin/androidx/compose/ui/test/AndroidInputDispatcher.android.kt
M compose/ui/ui-test/src/test/kotlin/androidx/compose/ui/test/inputdispatcher/InputDispatcherTest.kt

https://android-review.googlesource.com/1822244

cl...@google.com <cl...@google.com> Sep 14, 2022 01:17PM

Assigned to je...@google.com.

ap...@google.com <ap...@google.com> #13Apr 15, 2024 06:08PM

Project: platform/frameworks/support
Branch: androidx-main

commit fa5e67454053827a65bde026b4d85133d7467a71
Author: Jelle Fresen <jellefresen@google.com>
Date: Wed Apr 10 15:45:18 2024

Revisit mouse click injection API

Added a button parameter to all click methods in MouseInjectionScope
(click(), doubleClick(), tripleClick() and longClick()), and deprecated
the incorrectly named rightClick() method.

Bug: 190493367
Bug: 261439695
Test: Added tests for each method
Relnote: "`MouseInjectionScope.click()`,
`MouseInjectionScope.doubleClick()`,
`MouseInjectionScope.tripleClick()`, `MouseInjectionScope.longClick()`
now accept a `button: MouseButton` parameter to make them more
universally applicable. The default value is `MouseButton.Primary` for
all methods."

Change-Id: I31a23e6819a82b5513db3731dce3bf8cd7587d69

M compose/ui/ui-test/api/current.txt
M compose/ui/ui-test/api/restricted_current.txt
M compose/ui/ui-test/samples/src/main/java/androidx/compose/ui/test/samples/MouseInjectionScopeSamples.kt
M compose/ui/ui-test/src/androidCommonTest/kotlin/androidx/compose/ui/test/util/Verifications.kt
M compose/ui/ui-test/src/androidInstrumentedTest/kotlin/androidx/compose/ui/test/injectionscope/mouse/ClickTest.kt
M compose/ui/ui-test/src/androidInstrumentedTest/kotlin/androidx/compose/ui/test/injectionscope/mouse/Common.kt
M compose/ui/ui-test/src/androidInstrumentedTest/kotlin/androidx/compose/ui/test/util/PointerInputs.kt
M compose/ui/ui-test/src/commonMain/kotlin/androidx/compose/ui/test/MouseInjectionScope.kt

https://android-review.googlesource.com/3037717

ap...@google.com <ap...@google.com> #14Jun 7, 2024 03:35PM

Project: platform/frameworks/support
Branch: androidx-main

commit 277edf07f9f40b6d551e29409fe61fde3d2c785f
Author: Jelle Fresen <jellefresen@google.com>
Date: Tue Mar 19 16:55:19 2024

Graduate mouse injection API to stable

Removed methods that were deprecated hidden and experimental

Bug: 190493367
Fix: 261439695
Test: bOS
Relnote: "All mouse injection API for tests are now stable"
Change-Id: I974c9e6c9114c0babf6cfeae278930c537c02847

M compose/foundation/foundation/integration-tests/lazy-tests/src/androidTest/kotlin/androidx/compose/foundation/lazy/list/LazyNestedScrollingTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/BasicMarqueeTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/BasicTooltipTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/ClickableTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/CombinedClickableTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/Draggable2DTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/DraggableTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/HoverableTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/OverscrollTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/ScrollableTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/contextmenu/ContextMenuAreaTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/contextmenu/ContextMenuGestureTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/selection/SelectableTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/selection/ToggleableTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/text/BasicTextHoverTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/text/BasicTextLinkTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/text/ClickableTextTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/text/CoreTextFieldHoverTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/text/input/internal/BasicTextFieldHoverTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/text/input/internal/selection/TextFieldTextToolbarTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/text/selection/gestures/AbstractSelectionGesturesTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/text/selection/gestures/MultiTextSelectionGesturesRtlTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/text/selection/gestures/MultiTextSelectionGesturesTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/text/selection/gestures/MultiTextWithSpaceSelectionGesturesRegressionTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/text/selection/gestures/TextFieldSelectionGesturesTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/text/selection/gestures/TextSelectionGesturesBidiTest.kt
M compose/foundation/foundation/src/androidInstrumentedTest/kotlin/androidx/compose/foundation/text/selection/gestures/TextSelectionGesturesTest.kt
M compose/material/material/src/androidInstrumentedTest/kotlin/androidx/compose/material/ButtonScreenshotTest.kt
M compose/material/material/src/androidInstrumentedTest/kotlin/androidx/compose/material/CheckboxScreenshotTest.kt
M compose/material/material/src/androidInstrumentedTest/kotlin/androidx/compose/material/FloatingActionButtonScreenshotTest.kt
M compose/material/material/src/androidInstrumentedTest/kotlin/androidx/compose/material/RadioButtonScreenshotTest.kt
M compose/material/material/src/androidInstrumentedTest/kotlin/androidx/compose/material/SwitchScreenshotTest.kt
M compose/material3/material3/src/androidInstrumentedTest/kotlin/androidx/compose/material3/CardScreenshotTest.kt
M compose/material3/material3/src/androidInstrumentedTest/kotlin/androidx/compose/material3/CheckboxScreenshotTest.kt
M compose/material3/material3/src/androidInstrumentedTest/kotlin/androidx/compose/material3/FloatingActionButtonScreenshotTest.kt
M compose/material3/material3/src/androidInstrumentedTest/kotlin/androidx/compose/material3/IconButtonScreenshotTest.kt
M compose/material3/material3/src/androidInstrumentedTest/kotlin/androidx/compose/material3/RadioButtonScreenshotTest.kt
M compose/material3/material3/src/androidInstrumentedTest/kotlin/androidx/compose/material3/SwitchScreenshotTest.kt
M compose/ui/ui-test/api/current.txt
M compose/ui/ui-test/api/restricted_current.txt
M compose/ui/ui-test/samples/src/main/java/androidx/compose/ui/test/samples/MouseInjectionScopeSamples.kt
M compose/ui/ui-test/samples/src/main/java/androidx/compose/ui/test/samples/MultiModalInjectionScopeSamples.kt
M compose/ui/ui-test/src/androidInstrumentedTest/kotlin/androidx/compose/ui/test/injectionscope/mouse/CancelTest.kt
M compose/ui/ui-test/src/androidInstrumentedTest/kotlin/androidx/compose/ui/test/injectionscope/mouse/ClickTest.kt
M compose/ui/ui-test/src/androidInstrumentedTest/kotlin/androidx/compose/ui/test/injectionscope/mouse/MoveTest.kt
M compose/ui/ui-test/src/androidMain/kotlin/androidx/compose/ui/test/AndroidInputDispatcher.android.kt
M compose/ui/ui-test/src/androidMain/kotlin/androidx/compose/ui/test/Mouse.android.kt
M compose/ui/ui-test/src/androidUnitTest/kotlin/androidx/compose/ui/test/inputdispatcher/KeyAndMouseEventsTest.kt
M compose/ui/ui-test/src/androidUnitTest/kotlin/androidx/compose/ui/test/inputdispatcher/MouseEventsTest.kt
M compose/ui/ui-test/src/commonMain/kotlin/androidx/compose/ui/test/Actions.kt
M compose/ui/ui-test/src/commonMain/kotlin/androidx/compose/ui/test/InputDispatcher.kt
M compose/ui/ui-test/src/commonMain/kotlin/androidx/compose/ui/test/Mouse.kt
M compose/ui/ui-test/src/commonMain/kotlin/androidx/compose/ui/test/MouseInjectionScope.kt
M compose/ui/ui-test/src/commonMain/kotlin/androidx/compose/ui/test/MultiModalInjectionScope.kt
M compose/ui/ui-test/src/desktopMain/kotlin/androidx/compose/ui/test/DesktopInputDispatcher.desktop.kt
M compose/ui/ui-test/src/desktopMain/kotlin/androidx/compose/ui/test/Mouse.desktop.kt
M compose/ui/ui/src/androidInstrumentedTest/kotlin/androidx/compose/ui/input/nestedscroll/NestedScrollModifierTest.kt
M compose/ui/ui/src/androidInstrumentedTest/kotlin/androidx/compose/ui/input/pointer/MouseEventTest.kt
M compose/ui/ui/src/androidInstrumentedTest/kotlin/androidx/compose/ui/input/pointer/PointerIconTest.kt
M compose/ui/ui/src/androidInstrumentedTest/kotlin/androidx/compose/ui/platform/AndroidViewCompatTest.kt
M tv/tv-material/src/androidTest/java/androidx/tv/material3/IconButtonScreenshotTest.kt
M tv/tv-material/src/androidTest/java/androidx/tv/material3/RadioButtonScreenshotTest.kt
M tv/tv-material/src/androidTest/java/androidx/tv/material3/SwitchScreenshotTest.kt

https://android-review.googlesource.com/3043092