[Compose] Hardware keyboard & Text/Field [139322105]

Assigned

Feature Request

Status Update

No update yet.

Description

si...@google.com

created issue #1

Aug 13, 2019 02:19AM

General keyboard event
i.e. by clicking on Tab key the focus moves.

Comments

sh...@google.com <sh...@google.com> #2Aug 13, 2019 04:10PM

I don't think this falls under pointer input so this is simply less tied to the area I've been focusing on. I think this should be considered open for anyone else to start working on. I think the two main areas of work for keyboard input are building out the equivalent of focusableInTouchMode and routing keyboard input events to the appropriate node/modifier.

Is this something that Semantics should handle? Seems that if it did, both keyboard events, auto fill, google assistant, could all use the same mechanism to send text data to the ui.

si...@google.com <si...@google.com> #3Aug 13, 2019 07:14PM

Hey Shep, thanks for the info.
For the ticket itself it is a reminder for us that is not prioritized yet.
For editable text (TextField) this should already work since IME handles it.
However I am not sure if and when we want to support what Android has (things like menu shortcuts).

Happy to hear from Ryan and Ralston in case this subject was discussed before.

ry...@google.com <ry...@google.com> #4Aug 14, 2019 11:26AM

I don't think this is/should be within Semantics's domain. Hardware keyboard input should, as far as I know, follow the same path as (and is, as far as I understand it, indistinguishable from) soft keyboard input (in fact, Gboard actually interacts with hardware keyboards).

I also don't think we want autofill/assistant to actually send keyboard events. That could lead to very strange results as well as maybe performance issues - consider it trying to send dozens of key events instantaneously. I'd think we'd want it to fill text fields directly via some other mechanism (probably an action callback).

sh...@google.com <sh...@google.com> #5Aug 14, 2019 07:54PM

Ah, I based on Siyamed's comment, this bug is intended to track the work necessary to handle keyboard events that aren't related to IME. (like menu shortcuts). I'm not sure what is needed here.

ry...@google.com <ry...@google.com> #6Aug 15, 2019 03:22AM

Oh I see. There's probably some overlap there. For instance, the accessibility focus traversal ordering is probably pretty similar to the input focus traversal ordering (though a11y would likely focus things that input would not).

I think one question is probably at what level these things live. There are a bunch of keys that probably don't need to go to substantially all widgets with keyboard input, like volume/menu/search/back. Then there are things like tab, which some input widgets will want, but others should ignore and pass through to another handler (to, say, move input focus).

I think Android's model for this is pretty decent - an intercept pass down the hierachy, then a pass back up for normal handling. It handles both cases well, as each component can decide which keys it cares about, rather than us trying to predict all use-cases in advance. We would, of course, handle these correctly in our controls, and we'd need to make it easy (no internals or copy-pasting of internals) to implement, say, correct tab behavior.

All of which is a long-winded way of saying: "I think we should handle these like any other keyboard input."

si...@google.com <si...@google.com> #7Feb 4, 2020 02:42AM

bulk comment, fyi: we have added icebox to all the tickets because it was suggested by blintz in order to prevent auto-close tickets. we want to keep our Compose tickets to remain opened.

see

b/123001228#comment6

si...@google.com <si...@google.com> #8Apr 27, 2020 10:36PM

Looks like there is currently a ticket assigned to Ralston. Will keep this ticket active for testing as well. Ralston feel free to dup this ticket, or we can use it as a reminder to test.

ra...@google.com <ra...@google.com> #9Apr 28, 2020 05:27PM

Let's keep this ticket open. I can see that it is already marked as blocked by the keyboard Input ticket that I am working on right now.

si...@google.com <si...@google.com> May 26, 2020 02:13AM

Assigned to an...@google.com.

si...@google.com <si...@google.com> Aug 21, 2020 07:15PM

Reassigned to qq...@google.com.

si...@google.com <si...@google.com> Nov 13, 2020 12:51AM

Reassigned to an...@google.com.

ap...@google.com <ap...@google.com> #10Mar 8, 2021 04:19PM

Project: platform/frameworks/support
Branch: androidx-main

commit 072a880c9ab1a245711fe9e67e9a56f2da04b790
Author: Andrey Rudenko <andrei.rudenko@jetbrains.com>
Date: Fri Feb 05 22:43:02 2021

Hardware Keyboard support in TextField

It adds onKeyEvent Modifier to CoreTextField and sets up the handling of KeyEvent in a uniform way across Android and Desktop. This includes support of all main keyboard shortcuts to edit, navigate and select text in TextField.

This CL doesn't touch any of the existing APIs and doesn't introduce the new ones. On Desktop in changes significantly how hardware keyboard events are handled. PlatformInput is responsible only for interaction with IM.

Most of the text operations in this CL were abstracted to TextFieldPreparedSelection and TextPreparedSelection, the latter one can be used in future CL to implement similar keyboard shortcuts for CoreText.

It's incomplete, we have to define better the Compose behavior on Android with a hardware keyboard, possibly it has to react to TouchMode changes and, for example, do not show software keyboard when hardware one is in action

Relnote: N/A
Bug: 139322105
Test: ./gradlew :compose:foundation:foundation:connectedAndroidTest
Change-Id: I2d5d462ec1f57de41ddfd9218411e695fe4bb2e2

M compose/desktop/desktop/samples/src/jvmMain/kotlin/androidx/compose/desktop/examples/example1/Main.jvm.kt
M compose/foundation/foundation/api/1.0.0-beta02.txt
M compose/foundation/foundation/api/current.txt
M compose/foundation/foundation/api/public_plus_experimental_1.0.0-beta02.txt
M compose/foundation/foundation/api/public_plus_experimental_current.txt
M compose/foundation/foundation/api/restricted_1.0.0-beta02.txt
M compose/foundation/foundation/api/restricted_current.txt
M compose/foundation/foundation/build.gradle
A compose/foundation/foundation/src/androidAndroidTest/kotlin/androidx/compose/foundation/textfield/HardwareKeyboardTest.kt
M compose/foundation/foundation/src/androidMain/kotlin/androidx/compose/foundation/text/KeyMapping.android.kt
M compose/foundation/foundation/src/androidMain/kotlin/androidx/compose/foundation/text/TextFieldKeyInput.android.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/CoreTextField.kt
A compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/KeyCommand.kt
A compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/KeyMapping.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/StringHelpers.kt
A compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/TextFieldKeyInput.kt
A compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/selection/TextPreparedSelection.kt
M compose/foundation/foundation/src/desktopMain/kotlin/androidx/compose/foundation/text/DesktopCoreTextField.desktop.kt
A compose/foundation/foundation/src/desktopMain/kotlin/androidx/compose/foundation/text/KeyMapping.desktop.kt
A compose/foundation/foundation/src/desktopMain/kotlin/androidx/compose/foundation/text/TextFieldKeyInput.desktop.kt
A compose/foundation/foundation/src/jvmMain/kotlin/androidx/compose/foundation/text/StringHelpers.jvm.kt
M compose/ui/ui-text/src/desktopMain/kotlin/androidx/compose/ui/text/platform/DesktopParagraph.desktop.kt
M compose/ui/ui-text/src/desktopTest/kotlin/androidx/compose/ui/text/DesktopParagraphTest.kt
M compose/ui/ui/src/desktopMain/kotlin/androidx/compose/desktop/ComposeLayer.desktop.kt
M compose/ui/ui/src/desktopMain/kotlin/androidx/compose/ui/input/key/Key.desktop.kt
M compose/ui/ui/src/desktopMain/kotlin/androidx/compose/ui/platform/DesktopOwner.desktop.kt
M compose/ui/ui/src/desktopMain/kotlin/androidx/compose/ui/platform/DesktopOwners.desktop.kt
M compose/ui/ui/src/desktopMain/kotlin/androidx/compose/ui/platform/DesktopPlatformInput.desktop.kt

https://android-review.googlesource.com/1578803

si...@google.com <si...@google.com> #11Mar 8, 2021 05:46PM

Reassigned to Deleted User.

Andrei, can you please share with us

overall requirements
what is implemented
what is to be implemented
what are we planning in the near team to support those requirements

[Deleted User] <[Deleted User]> #12Mar 9, 2021 12:29PM

Overall requirements

I can formulate some general goals:

it should be as much "native" for the user as possible. The definition of "native" is not very clear, for desktop, for example, it differs from platform to platform, and on Linux is anarchy. On Android keyboard was some sort of a second-class citizen for a long time, as I understand, so I'm not sure how it's "native" at all for most of the users. Should we use the original text field implementation as a reference? Or Chrome one?
full first-class rtl/bidi support
CoreText interaction with the keyboard and especially mouse is very similar to TextField interaction, so I guess they have to share both experience and implementation, where it's possible / needed.
exotic shortcuts like ctrl-t (emacs-like transpose) and kill-"ring" (in quotes because this "ring" is of size 1) ctrl-k/ctrl-y. Most users have no idea of them. But Chrome supports it
undo/redo functionality
uniformed implementation between Android and Compose for Desktop platforms

what is implemented

Adding support of key events was the easy part, thanks to the current state of focus and KeyEvent in compose. A small addition to KeyEvent interface (isTypedEvent) allowed uniform handling between Android and Desktop. Few dozens of basic keyboard commands are implemented and abstracted to the separate internal interface (TextPreparedSelection).

what is to be implemented

rtl/bidi support is broken currently (the separate issue is here https://partnerissuetracker.corp.google.com/u/1/issues/181900696)
better mouse support, at least double / triple clicks (the separate issue is here https://partnerissuetracker.corp.google.com/u/1/issues/180639271)
make CoreText supports the same selection capabilities, including keyboard shortcuts
undo/redo

what are we planning in the near term to support those requirements

Proper support of keyboard and mouse in textfield is essential for desktop experience, so we gonna continue to work on it. Everything described in "general goals" should be implemented before the beta release of CfD. With exception of "exotic shortcuts", maybe :) It would be great to hear thoughts/vision from Google, especially about Android platform.

Before the next steps I want to finish the current iteration, address some of the comments in merged CL, first of all, better test coverage, then RTL/bidi support.

I also going to write undo/redo doc, where we can discuss options that we have.

[Deleted User] <[Deleted User]> #13Mar 10, 2021 09:23AM

One more thing. Cursor animation should be disabled during cursor navigation. Currently, the cursor is blinking as usual.

ap...@google.com <ap...@google.com> #14Mar 23, 2021 12:17PM

Project: platform/frameworks/support
Branch: androidx-main

commit 8423691f9f730b9c6da33310183169ed94392101
Author: Andrey Rudenko <andrei.rudenko@jetbrains.com>
Date: Thu Mar 11 11:42:17 2021

Hardware Keyboard vertical navigation improvements and tests

To properly support navigation by lines, the logic should be stateful. It saves the x-position of the cursor when navigation was started and tries to preserve it during vertical navigation

Bug: 139322105
Test: ./gradlew :compose:foundation:foundation:connectedAndroidTest
Change-Id: Ic4daa8159146649dcc8b8982b590b700db7c8a90

A compose/foundation/foundation/src/androidAndroidTest/kotlin/androidx/compose/foundation/text/TextPreparedSelectionTest.kt
M compose/foundation/foundation/src/androidAndroidTest/kotlin/androidx/compose/foundation/textfield/HardwareKeyboardTest.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/TextFieldKeyInput.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/selection/TextPreparedSelection.kt

https://android-review.googlesource.com/1628580

ap...@google.com <ap...@google.com> #15Mar 30, 2021 10:24PM

Project: platform/frameworks/support
Branch: androidx-main

commit 91ea279ff99d2ade8998596eb34254c472df8baf
Author: Siyamed Sinir <siyamed@google.com>
Date: Mon Mar 29 21:05:25 2021

RecordingInputConnection.sendKeyEvent

Since hardware keyboard on view/composable level
is added, we do not need to handle sendKeyEvent.

Test: Manual test on Pixel 4, Android 11, GBoard
Test: Manual test on Samsung S8, Android 9, Samsung keyboard

Test: ./gradlew compose:ui:ui-text:test
Test: ./gradlew compose:ui:ui-text:connectedAndroidTest
Test: ./gradlew compose:ui:ui:test
Test: ./gradlew compose:ui:ui:connectedAndroidTest

Bug: 184003208
Bug: 139322105
Change-Id: I68c6369a046c8e66b6945afa7679d2166eb9e08a

M compose/ui/ui/src/androidAndroidTest/kotlin/androidx/compose/ui/input/RecordingInputConnectionTest.kt
A compose/ui/ui/src/androidMain/kotlin/androidx/compose/ui/text/input/InputEventCallback2.android.kt
M compose/ui/ui/src/androidMain/kotlin/androidx/compose/ui/text/input/RecordingInputConnection.android.kt
M compose/ui/ui/src/androidMain/kotlin/androidx/compose/ui/text/input/TextInputServiceAndroid.android.kt
M compose/ui/ui/src/test/kotlin/androidx/compose/ui/input/RecordingInputConnectionUpdateTextFieldValueTest.kt

https://android-review.googlesource.com/1657169

ap...@google.com <ap...@google.com> #16Apr 9, 2021 03:36PM

Project: platform/frameworks/support
Branch: androidx-main

commit d6c91be0eb229033bd6bab039e770275f358bb1d
Author: Andrey Rudenko <andrei.rudenko@jetbrains.com>
Date: Thu Mar 18 15:45:20 2021

Mouse Selection improvements

- uniform support of mouse selection in CoreText and CoreTextField
- support of selection expansion/shrinking with shift pressed clicks
- double- (select by word) and triple- (select by paragraph) support

Change-Id: I031fcd716c8e795efb5c7e355b123d2d92458f0e
Test: ./gradlew :compose:foundation:foundation:connectedAndroidTest
Bug: 180639271
Bug: 139322105
Relnote: N/A

M compose/foundation/foundation/api/1.0.0-beta05.txt
M compose/foundation/foundation/api/current.txt
M compose/foundation/foundation/api/public_plus_experimental_1.0.0-beta05.txt
M compose/foundation/foundation/api/public_plus_experimental_current.txt
M compose/foundation/foundation/api/restricted_1.0.0-beta05.txt
M compose/foundation/foundation/api/restricted_current.txt
M compose/foundation/foundation/src/androidAndroidTest/kotlin/androidx/compose/foundation/text/selection/MultiWidgetSelectionDelegateTest.kt
M compose/foundation/foundation/src/androidAndroidTest/kotlin/androidx/compose/foundation/text/selection/TextFieldSelectionDelegateTest.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/CoreText.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/CoreTextField.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/StringHelpers.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/TextFieldGestureModifiers.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/selection/MultiWidgetSelectionDelegate.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/selection/Selectable.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/selection/SelectionManager.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/selection/SelectionRegistrar.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/selection/SelectionRegistrarImpl.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/selection/TextFieldSelectionDelegate.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/selection/TextFieldSelectionManager.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/selection/TextPreparedSelection.kt
M compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/selection/TextSelectionDelegate.kt
A compose/foundation/foundation/src/commonMain/kotlin/androidx/compose/foundation/text/selection/TextSelectionMouseDetector.kt
M compose/foundation/foundation/src/desktopTest/kotlin/androidx/compose/foundation/text/selection/DesktopTextFieldSelectionManagerTest.kt
M compose/foundation/foundation/src/test/kotlin/androidx/compose/foundation/text/TextSelectionLongPressDragTest.kt
M compose/foundation/foundation/src/test/kotlin/androidx/compose/foundation/text/selection/MockSelectable.kt
M compose/foundation/foundation/src/test/kotlin/androidx/compose/foundation/text/selection/SelectionManagerDragTest.kt
M compose/foundation/foundation/src/test/kotlin/androidx/compose/foundation/text/selection/SelectionManagerTest.kt
A compose/foundation/foundation/src/test/kotlin/androidx/compose/foundation/text/selection/StringHelpersTest.kt

https://android-review.googlesource.com/1645491

si...@google.com <si...@google.com> Nov 11, 2021 12:35AM

Reassigned to ma...@google.com.

ma...@google.com <ma...@google.com> Nov 18, 2021 10:20AM

Status: New

ma...@google.com <ma...@google.com> #17Nov 18, 2021 06:31PM

Chatted with Siyamed. He is tracking hardware and keyboard as a part of text features.

ae...@google.com <ae...@google.com> Nov 20, 2021 01:46AM

Assigned to si...@google.com.

si...@google.com <si...@google.com> #18Nov 23, 2021 01:00AM

Reassigned to ae...@google.com.

Alex, is there a bug for keyboard input framework so that we can add as a blocker to this one?

si...@google.com <si...@google.com> #19Jan 13, 2022 06:35AM

as an FYI: this ticket requires investigation into KeyListener classes in https://cs.android.com/android/platform/superproject/+/master:frameworks/base/core/java/android/text/method/

There are bunch of them with lots of rules and they are mostly not implemented in compose.

si...@google.com <si...@google.com> Jan 25, 2022 05:12PM