AnnotatedString rendered with URLs do not appear in TalkBack 'Links' context menu [231495122]

Fixed

Bug

Status Update

No update yet.

Description

br...@monzo.com

created issue #1

May 5, 2022 10:32AM

I'm currently exploring rendering markdown-style links using ClickableText/AnnotatedStrings, similar to how JetChat works here: https://github.com/android/compose-samples/blob/ee198110d8a7575da281de9bd0f84e91970468ca/Jetchat/app/src/main/java/com/example/compose/jetchat/conversation/Conversation.kt#L481

I'm running into issues with matching the accessibility experience of the old View system. In the old system, URLSpans automatically provided a great accessibility UX:

When reading out any link, there would be a small ding sound to help indicate where the link is.
TalkBack tells the user that this text has links in it.
Upon opening the TalkBack context menu, there would be a "Links" submenu where the user can scroll through all of the links contained within the text.

This doesn't seem to be possible to replicate in Compose. All I can find in the semantics API is "custom actions", which is a similar but worse experience (no ding sound, no specific "links" submenu, no hint to the user that there's links to be clicked, etc).

I think this inconsistency will be confusing for accessibility users. Can we get a new API to help replicate the functionality of the View system? For example, could we add an annotation to indicate to TalkBack where the links are?

Thanks!

Comments

so...@google.com <so...@google.com> May 24, 2022 09:06AM

Reassigned to so...@google.com.

zs...@salesforce.com <zs...@salesforce.com> #2May 26, 2022 10:58PM

Project: platform/frameworks/support
Branch: androidx-master-dev

commit c60f33e229e31ab328ef6b59dab63b264954831c
Author: Alexandre Elias <aelias@google.com>
Date: Fri Jul 10 16:23:09 2020

Semantics no-op cleanups

Partly in response to lmr's broad code review, I did a pass of
superficial API/implementation cleanup. The main changes are:

- I changed each Boolean SemanticsProperty where false is equivalent to
not being present to take "Unit" instead. This is conceptually
clearer: it avoids questions like "can I cancel out a semantics from a
merged child by setting it to false?" Because "property = Unit" looks
weird, I also changed the style of these to "property()".

- I moved the Semantics id generator closer to where it's used, in
SemanticsModifierCore. I made it internal and an AtomicInt.
(Note that integer ids are heavily used in the Android
AccessibilityNodeInfo APIs so I can't simply remove them entirely.)

- I deleted dead code. Some examples include SemanticsHintOverrides,
a public API not connected to anything, and SemanticsPropertyKey
merge() open method which is never called. (In both cases I have
a different plan in mind for accessibility.)

Fixes: 145951226
Fixes: 145955412
Test: existing tests
Relnote: "Single-value semantics properties now use a calling style.
For example, 'semantics { hidden = true }' is now written as:
'semantics { hidden() }'."

Change-Id: Ic1afd12ea22c926babc9662f1804d80b33aa0cfc

M ui/integration-tests/benchmark/src/androidTest/java/androidx/ui/benchmark/test/LayoutNodeModifierBenchmark.kt
M ui/ui-core/api/0.1.0-dev15.txt
M ui/ui-core/api/current.txt
M ui/ui-core/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-core/api/public_plus_experimental_current.txt
M ui/ui-core/api/restricted_0.1.0-dev15.txt
M ui/ui-core/api/restricted_current.txt
M ui/ui-core/src/androidAndroidTest/kotlin/androidx/ui/graphics/vector/VectorTest.kt
M ui/ui-core/src/androidAndroidTest/kotlin/androidx/ui/semantics/SemanticsTests.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidActuals.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidComposeView.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidComposeViewAccessibilityDelegateCompat.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidPopup.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/Expect.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsConfiguration.kt
D ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsHintOverrides.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsModifier.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsNode.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsOwner.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsWrapper.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/semantics/SemanticsProperties.kt
M ui/ui-foundation/api/0.1.0-dev15.txt
M ui/ui-foundation/api/current.txt
M ui/ui-foundation/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-foundation/api/public_plus_experimental_current.txt
M ui/ui-foundation/api/restricted_0.1.0-dev15.txt
M ui/ui-foundation/api/restricted_current.txt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Clickable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Dialog.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Scroller.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/selection/Selectable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/selection/Toggleable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/semantics/FoundationSemanticsProperties.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ButtonTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/CardTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/CheckboxScreenshotTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/RadioButtonScreenshotTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ScaffoldTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/SnackbarTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/SurfaceTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ripple/RippleIndicationTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/textfield/TextFieldScreenshotTest.kt
M ui/ui-material/src/main/java/androidx/ui/material/AppBar.kt
M ui/ui-material/src/main/java/androidx/ui/material/TextFieldImpl.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/AssertsTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/CallSemanticsActionTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/ErrorMessagesTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/FindersTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/PrintToStringTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/ScrollToTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/TextActionsTest.kt
M ui/ui-test/src/main/java/androidx/ui/test/Actions.kt
M ui/ui-test/src/main/java/androidx/ui/test/Filters.kt
M ui/ui-text/api/0.1.0-dev15.txt
M ui/ui-text/api/current.txt
M ui/ui-text/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-text/api/public_plus_experimental_current.txt
M ui/ui-text/api/restricted_0.1.0-dev15.txt
M ui/ui-text/api/restricted_current.txt
M ui/ui-text/src/commonMain/kotlin/androidx/ui/text/CoreTextField.kt
M ui/ui-text/src/commonMain/kotlin/androidx/ui/text/TextSemanticsProperties.kt

https://android-review.googlesource.com/1360099

kl...@google.com <kl...@google.com> #3May 26, 2022 11:17PM

I think we need to add some sort of UrlAnnotation to AnnotatedString to support this, like we did for TtsAnnotation. We could do this before implementing support for actual clickable links, to at least support a11y.

so...@google.com <so...@google.com> #4May 27, 2022 09:13AM

Reassigned to kl...@google.com.

Re-assigning to Zach as he has a CL already aosp/2108696

so...@google.com <so...@google.com> #5May 31, 2022 05:46PM

Reassigned to so...@google.com.

Re-assigning back for now

cl...@google.com <cl...@google.com> #6Jun 7, 2022 06:19PM

is this a P4?

so...@google.com <so...@google.com> #7Jun 7, 2022 06:36PM

Reassigned to kl...@google.com.

To comment#6, generally speaking, we haven't been using priorities due to its specific meaning/ SLO status.

Re-assigning back to Zach. As discussed, we'll proceed with aosp/2108696

kl...@google.com <kl...@google.com> Jun 8, 2022 11:02PM

Accepted by kl...@google.com.

ap...@google.com <ap...@google.com> #8Jun 9, 2022 04:55PM

Marked as fixed.

Project: platform/frameworks/support
Branch: androidx-main

commit 0c11fb9996e4baf9bf46dc57de9df714b7f61214
Author: Zach Klippenstein <klippenstein@google.com>
Date: Thu May 26 17:01:28 2022

Introduce UrlAnnotation to support link a11y.

This change is basically just a copy of `VerbatimTtsAnnotation`.
Includes the new annotation type, methods on `AnnotatedString` and
`Builder`, a new `Saver`, support for converting to `URLSpan`s when creating
`SpannableString`s, a demo, and tests including a missing test for
`VerbatimTtsAnnotation`. All new APIs are marked experimental.

Fixes:

b/231495122

Test: ./gradlew :compose:ui:ui-text:test
Test: Tested TalkBack integration manually
Relnote: "Introduced the `UrlAnnotation` annotation type and associated
methods to support TalkBack link support in `AnnotatedString`s."
Change-Id: I1c754dfa0ce88a36989888b43816333dfc94b0aa

A compose/ui/ui-text/src/commonMain/kotlin/androidx/compose/ui/text/UrlAnnotation.kt
M compose/ui/ui-text/api/restricted_current.txt
M compose/ui/ui-text/src/commonMain/kotlin/androidx/compose/ui/text/AnnotatedString.kt
M compose/ui/ui-text/src/androidAndroidTest/kotlin/androidx/compose/ui/text/platform/AndroidAccessibilitySpannableStringTest.kt
M compose/ui/ui-text/src/test/java/androidx/compose/ui/text/AnnotatedStringBuilderTest.kt
M compose/ui/ui-text/src/test/java/androidx/compose/ui/text/SaversTest.kt
M compose/ui/ui-text/src/test/java/androidx/compose/ui/text/input/TextFieldValueTest.kt
M compose/foundation/foundation/src/androidAndroidTest/kotlin/androidx/compose/foundation/textfield/TextFieldTest.kt
M compose/ui/ui-text/src/commonMain/kotlin/androidx/compose/ui/text/Savers.kt
M compose/foundation/foundation/integration-tests/foundation-demos/src/main/java/androidx/compose/foundation/demos/text/ComposeTextAccessibility.kt
A compose/ui/ui-text/src/androidMain/kotlin/androidx/compose/ui/text/platform/extensions/UrlAnnotationExtensions.android.kt
M compose/ui/ui-text/api/public_plus_experimental_current.txt
M compose/ui/ui-text/src/androidMain/kotlin/androidx/compose/ui/text/platform/AndroidAccessibilitySpannableString.android.kt
M compose/ui/ui-text/api/current.txt

https://android-review.googlesource.com/2108696

[Deleted User] <[Deleted User]> #9Oct 7, 2022 12:24AM

This has a bug in it - it does not call through to the onClick for a ClickableText. See repro example app here:

https://github.com/chanakin/ClickableTextWithUrlExample

Should I open a new issue to track this?

br...@monzo.com <br...@monzo.com> #10Oct 7, 2022 10:46AM

I'm also just trying this for the first time and:

a) can confirm the above issue. When double clicking on a link in the accessibility menu, nothing happens.

b) even if the click handler ran, it would just launch an ACTION_VIEW intent, but this default behaviour isn't good enough for our use case. We want to be able to intercept the link in our Compose code and handle it appropriately.

zl...@google.com <zl...@google.com> #11Oct 7, 2022 10:56PM

Also experiencing the issue in

https://github.com/chanakin/ClickableTextWithUrlExample, in which double-clicking a UrlAnnotation link in the TalkBack menu does not fire the onClick handler set in a ClickableText

[Deleted User] <[Deleted User]> #12Oct 12, 2022 11:20PM

I created a new issue, hopefully that gets some attention. Those of you who are following along and reporting the same issue, please star it!

https://issuetracker.google.com/issues/253292081

Message last modified on Oct 12, 2022 11:27PM

sr...@gmail.com <sr...@gmail.com> #13Jan 22, 2024 01:15AM

Any Update on this issue?