ClickableText with AnnotatedString using UrlAnnotation with Talkback enabled does not call onClick() [253292081]

Fixed

Bug

Status Update

No update yet.

Description

[Deleted User]

created issue #1

Oct 12, 2022 11:25PM

- Jetpack Compose version: 1.3.0-rc01
- Jetpack Compose component used: ClickableText with AnnotatedString including UrlAnnotation
- Android Studio Build: Android Studio Chipmunk | 2021.2.1 Patch 2
Build #AI-212.5712.43.2112.8815526, built on July 10, 2022

- Kotlin version: 1.7.20
- Devices/Android versions reproduced on: Android 30, 33 on Pixel 4As

- Keyboard (i.e. Gboard, Samsung, etc): Stock Google keyboard

If this is a bug in the library, we would appreciate if you could attach:
- Sample project to trigger the issue:

https://github.com/chanakin/ClickableTextWithUrlExample

- A screenrecord or screenshots showing the issue (if UI related). This is included with the sample project but I attached it again here.

- Stack trace (if applicable) - N/A

There is a bug where the recently introduced UrlAnnotation is read correctly by Talkback, but the onClick is not called, so the link is not opened.

Steps to Reproduce:

1. Turn on Talkback in Accessibility settings
2. Run app
3. App will display a single line of text
4. Part of this text will be a URLAnnotation containing a link
5. Open the Talkback action menu per the prompt
6. Select "Links" in Talkback menu
7. Select the link to trigger the onClick method call, which opens the browser.

Expected:
1.Talkback correctly recognizes the link in the text
2. Talkback shows the link in the Talkback menu
3. When the link is selected in the Talkback menu, the onClick is called in ClickableText and the user is navigated to the browser.

Actual:

1. The onClick for the ClickableText is never called. Note that it is not the browser navigation that is broken. This can be confirmed using the logs in the sample app provided.

screen-20221012-161435.mp4

5.1 MB

Download

Comments

kl...@google.com <kl...@google.com> Oct 17, 2022 11:47PM

Status: Duplicate of

139312671

pa...@hinge.co <pa...@hinge.co> #2Dec 28, 2022 05:51PM

Project: platform/frameworks/support
Branch: androidx-master-dev

commit c60f33e229e31ab328ef6b59dab63b264954831c
Author: Alexandre Elias <aelias@google.com>
Date: Fri Jul 10 16:23:09 2020

Semantics no-op cleanups

Partly in response to lmr's broad code review, I did a pass of
superficial API/implementation cleanup. The main changes are:

- I changed each Boolean SemanticsProperty where false is equivalent to
not being present to take "Unit" instead. This is conceptually
clearer: it avoids questions like "can I cancel out a semantics from a
merged child by setting it to false?" Because "property = Unit" looks
weird, I also changed the style of these to "property()".

- I moved the Semantics id generator closer to where it's used, in
SemanticsModifierCore. I made it internal and an AtomicInt.
(Note that integer ids are heavily used in the Android
AccessibilityNodeInfo APIs so I can't simply remove them entirely.)

- I deleted dead code. Some examples include SemanticsHintOverrides,
a public API not connected to anything, and SemanticsPropertyKey
merge() open method which is never called. (In both cases I have
a different plan in mind for accessibility.)

Fixes: 145951226
Fixes: 145955412
Test: existing tests
Relnote: "Single-value semantics properties now use a calling style.
For example, 'semantics { hidden = true }' is now written as:
'semantics { hidden() }'."

Change-Id: Ic1afd12ea22c926babc9662f1804d80b33aa0cfc

M ui/integration-tests/benchmark/src/androidTest/java/androidx/ui/benchmark/test/LayoutNodeModifierBenchmark.kt
M ui/ui-core/api/0.1.0-dev15.txt
M ui/ui-core/api/current.txt
M ui/ui-core/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-core/api/public_plus_experimental_current.txt
M ui/ui-core/api/restricted_0.1.0-dev15.txt
M ui/ui-core/api/restricted_current.txt
M ui/ui-core/src/androidAndroidTest/kotlin/androidx/ui/graphics/vector/VectorTest.kt
M ui/ui-core/src/androidAndroidTest/kotlin/androidx/ui/semantics/SemanticsTests.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidActuals.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidComposeView.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidComposeViewAccessibilityDelegateCompat.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidPopup.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/Expect.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsConfiguration.kt
D ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsHintOverrides.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsModifier.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsNode.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsOwner.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsWrapper.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/semantics/SemanticsProperties.kt
M ui/ui-foundation/api/0.1.0-dev15.txt
M ui/ui-foundation/api/current.txt
M ui/ui-foundation/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-foundation/api/public_plus_experimental_current.txt
M ui/ui-foundation/api/restricted_0.1.0-dev15.txt
M ui/ui-foundation/api/restricted_current.txt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Clickable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Dialog.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Scroller.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/selection/Selectable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/selection/Toggleable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/semantics/FoundationSemanticsProperties.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ButtonTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/CardTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/CheckboxScreenshotTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/RadioButtonScreenshotTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ScaffoldTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/SnackbarTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/SurfaceTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ripple/RippleIndicationTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/textfield/TextFieldScreenshotTest.kt
M ui/ui-material/src/main/java/androidx/ui/material/AppBar.kt
M ui/ui-material/src/main/java/androidx/ui/material/TextFieldImpl.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/AssertsTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/CallSemanticsActionTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/ErrorMessagesTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/FindersTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/PrintToStringTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/ScrollToTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/TextActionsTest.kt
M ui/ui-test/src/main/java/androidx/ui/test/Actions.kt
M ui/ui-test/src/main/java/androidx/ui/test/Filters.kt
M ui/ui-text/api/0.1.0-dev15.txt
M ui/ui-text/api/current.txt
M ui/ui-text/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-text/api/public_plus_experimental_current.txt
M ui/ui-text/api/restricted_0.1.0-dev15.txt
M ui/ui-text/api/restricted_current.txt
M ui/ui-text/src/commonMain/kotlin/androidx/ui/text/CoreTextField.kt
M ui/ui-text/src/commonMain/kotlin/androidx/ui/text/TextSemanticsProperties.kt

https://android-review.googlesource.com/1360099

pa...@hinge.co <pa...@hinge.co> #3Dec 28, 2022 05:53PM

This issue prevents us from using Compose throughout several parts of our app because links are not accessible.

kl...@google.com <kl...@google.com> Jan 3, 2023 06:49PM

Status: Assigned (reopened)

kl...@google.com <kl...@google.com> #4Mar 16, 2023 11:15PM

This is broken because when the AccessibilityInteractionController asks the AccessibilityClickableSpan to get its clickable span from the original text (code), the span IDs don't match (code), so it doesn't return the span.

ID is a property of ClickableSpan, which is automatically generated for every new instance from a monotonically increasing global counter. It looks like the getId() method is overridable, so we should be able to customize it, but that means Compose needs a way to associate a stable ID with annotations somehow, which we don't currently have a way of doing.

si...@google.com <si...@google.com> Mar 17, 2023 12:26AM

Reassigned to si...@google.com.

kl...@google.com <kl...@google.com> #5Mar 17, 2023 12:30AM

We may also need to use enableAccessibleClickableSpanSupport as documented in https://support.google.com/accessibility/android/answer/6378148?hl=en to support < API 26.

Unrelated to this bug, we may also want to implement this using a custom span that overrides onClick to invoke the UrlHandler from the composition instead of letting the default implementation directly fire an intent.

Message last modified on Mar 17, 2023 12:31AM

kl...@google.com <kl...@google.com> #6Mar 17, 2023 10:33PM

It does not seem to be possible to override ClickableSpan's getId() method, which means that the only way to make this work is to actually use the same instance across a11y string conversions.

kl...@google.com <kl...@google.com> #7Mar 18, 2023 12:03AM

Accepted by kl...@google.com.

We don't need to call enableAccessibleClickableSpanSupport because AndroidComposeView is already using the a11y delegate compat stuff. This will be fixed in

ap...@google.com <ap...@google.com> #8Mar 20, 2023 05:40PM

Marked as fixed.

Project: platform/frameworks/support
Branch: androidx-main

commit 41183c41f686979fb051549196f7b06cc172022b
Author: Zach Klippenstein <klippenstein@google.com>
Date: Fri Mar 17 16:33:38 2023

Fix UrlAnnotation links not being opened by a11y services.

Relnote: "`UrlAnnotation`s in `AnnotatedString`s can now be opened via
accessibility services like TalkBack."
Test: AndroidAccessibilitySpannableStringTest.kt
Test: Manually with TalkBack
Fixes:

b/253292081

Change-Id: If4d82999d8c62aa3718e1e681eeeaac12a9b0f55

M compose/foundation/foundation/integration-tests/foundation-demos/src/main/java/androidx/compose/foundation/demos/text/ComposeTextAccessibility.kt
M compose/ui/ui-text/api/public_plus_experimental_current.txt
M compose/ui/ui-text/src/androidAndroidTest/kotlin/androidx/compose/ui/text/platform/AndroidAccessibilitySpannableStringTest.kt
M compose/ui/ui-text/src/androidMain/kotlin/androidx/compose/ui/text/platform/AndroidAccessibilitySpannableString.android.kt
A compose/ui/ui-text/src/androidMain/kotlin/androidx/compose/ui/text/platform/URLSpanCache.kt
D compose/ui/ui-text/src/androidMain/kotlin/androidx/compose/ui/text/platform/extensions/UrlAnnotationExtensions.android.kt
M compose/ui/ui/src/androidMain/kotlin/androidx/compose/ui/platform/AndroidComposeViewAccessibilityDelegateCompat.android.kt

https://android-review.googlesource.com/2496352

na...@google.com <na...@google.com> #9Apr 5, 2023 06:27PM

The following release(s) address this bug.It is possible this bug has only been partially addressed:

androidx.compose.foundation:foundation:1.5.0-alpha02
androidx.compose.ui:ui:1.5.0-alpha02
androidx.compose.ui:ui-text:1.5.0-alpha02

ju...@google.com <ju...@google.com> #10May 24, 2023 07:37PM

The following release(s) address this bug.It is possible this bug has only been partially addressed:

androidx.compose.ui:ui-text-android:1.5.0-beta01

sv...@gmail.com <sv...@gmail.com> #11Jan 19, 2024 12:35PM

Still reproducible.
Versions:
androidx.compose.ui:ui-text-android:1.5.1
androidx.compose.foundation:foundation:1.5.1
androidx.compose.ui:ui:1.5.1
androidx.compose.ui:ui-text:1.5.1

fu...@gmail.com <fu...@gmail.com> #12Apr 15, 2024 10:31AM

Comment has been deleted.

Message last modified on Apr 15, 2024 03:01PM

fu...@gmail.com <fu...@gmail.com> #13Apr 15, 2024 03:33PM

The issue is only fixed in URL redirection case but not custom action case, looks there will be a LinkAnnotation added in 1.7.0 for it.

In case anyone need solution now or unable to use newest compose version, might try these approaches:

Use `customActions` instead of link

@OptIn(ExperimentalTextApi::class)
@Composable
fun TestInlineLink1() = Column(
        modifier = Modifier.fillMaxWidth(),
        verticalArrangement = Arrangement.Center,
        horizontalAlignment = Alignment.CenterHorizontally,
) {
    var result by remember { mutableStateOf("") }
    fun action(tag: String) {
        result = "Inline link [ $tag ] get clicked"
    }

    val text = buildAnnotatedString {
        append("This is an example for ")
        withAnnotation(tag = "link1", annotation = "link1") {
            append("link1")
        }
        append(" and ")
        withAnnotation(tag = "link2", annotation = "link2") {
            append("link2")
        }
    }
    ClickableText(
        modifier = Modifier.semantics {
            customActions = listOf(
                CustomAccessibilityAction("link1") {
                    action("link1")
                    true
                },
                CustomAccessibilityAction("link2") {
                    action("link2")
                    true
                },
            )
        },
        text = text,
        onClick = { offset ->
            val tag = text.getStringAnnotations(offset, offset)
                .singleOrNull()?.tag
            tag?.let(::action)
        }
    )
    Text(text = result)
}

Modify `AccessibilityNodeInfo`

@OptIn(ExperimentalTextApi::class)
@Composable
fun TestInlineLink2() = Column(
        modifier = Modifier.fillMaxWidth(),
        verticalArrangement = Arrangement.Center,
        horizontalAlignment = Alignment.CenterHorizontally,
) {
    var result by remember { mutableStateOf("") }
    fun action(tag: String) {
        result = "Inline link [ $tag ] get clicked"
    }

    val text = buildAnnotatedString {
        append("This is an example for ")
        withAnnotation(urlAnnotation = UrlAnnotation(url = "link1")) {
            append("link1")
        }
        append(" and ")
        withAnnotation(urlAnnotation = UrlAnnotation(url = "link2")) {
            append("link2")
        }
    }
    WrappingView {
        ReplaceAccessibilityNodeInfoText(
            text = text.toSpannableString(::action)
        )
        ClickableText(
            text = text,
            onClick = { offset ->
                val tag = text.getUrlAnnotations(offset, offset)
                    .singleOrNull()?.item?.url
                tag?.let(::action)
            },
        )
    }
    Text(text = result)
}

@Composable
private fun ReplaceAccessibilityNodeInfoText(
    text: CharSequence,
) {
    val view = LocalView.current
    val defaultDelegate = ViewCompat.getAccessibilityDelegate(view)
    val newDelegate = object : AccessibilityDelegateCompat() {
        override fun onInitializeAccessibilityNodeInfo(
            host: View,
            info: AccessibilityNodeInfoCompat,
        ) {
            defaultDelegate?.onInitializeAccessibilityNodeInfo(host, info)
            info.text = text
        }
    }
    ViewCompat.setAccessibilityDelegate(view, newDelegate)
}

/**
 *  use AndroidView to create an AndroidComposeView,
 *  aim to prevent modify accessibilityDelegate of outer view.
 */
@Composable
private fun WrappingView(
    modifier: Modifier = Modifier,
    content: @Composable () -> Unit,
) {
    AndroidView(
        modifier = modifier,
        factory = { context ->
            ComposeView(context).apply {
                setContent(content)
            }
        }
    )
}

@OptIn(ExperimentalTextApi::class)
private fun AnnotatedString.toSpannableString(
    action: (String) -> Unit
) = buildSpannedString {
    append(this@toSpannableString)
    getUrlAnnotations(0, length).forEach { (item, start, end, _) ->
        setSpan(
            object : ClickableSpan() {
                override fun onClick(widget: View) {
                    action(item.url)
                }
            },
            start, end, SPAN_INCLUSIVE_EXCLUSIVE
        )
    }
}