vkCmdClearAttachments in the middle of render pass creates visual artifacts on Pixel devices [166809097]

Assigned

Bug

Status Update

No update yet.

Description

sy...@google.com

created issue #1

Aug 28, 2020 07:59PM

This change exposes a bug with Qualcomm drivers where vkCmdClearAttachments is used in the middle of the render pass to clear one of the attachments. The corruption (also seen with renderdoc) is that in the following sequence:

Begin RP (loadOp=LOAD, storeOp=STORE)
A draw call is used to perform a masked color clear of the color attachment
vkCmdClearAttachment is used to reclear the same attachment! (test originating from dEQP). We use vkCmdClearAttachment in this case to avoid having to break the render pass (just to use loadOp=CLEAR)
Draw
End RP

at time of the second draw, some tiles show the value of the first draw, instead of the clear color. It's as if the first draw and the vkCmdClearAttachments call are racing.

Debugging with renderdoc, I see the masked-clear draw call work fine. I see the vkCmdClearAttachment also work fine. However, running up to the first draw call after that, I see parts of the color attachment have the masked-clear draw call output, as if it raced with vkCmdClearAttachment. See the following screenshots from renderdoc:

Masked clear (not clearing G): 1.png
Overriden by vkCmdClearAttachment: 2.png
Tiled corruption: 3.png

The description of vkCmdClearAttachment says:

Unlike other clear commands, vkCmdClearAttachments executes as a drawing command, rather than a transfer command, with writes performed by it executing in rasterization order.

So no barrier should be necessary for draw->clearAttachments->draw, which leads me to believe this is a Qualcomm driver bug.

With the above change, the following tests fail on dEQP due to this. Note that the change affects these tests by changing the first draw call (which does a masked clear) to issue one large triangle covering the render area instead of two triangles.

dEQP.GLES3/functional_fragment_ops_interaction_basic_shader_11 dEQP.GLES3/functional_fragment_ops_interaction_basic_shader_40 dEQP.GLES3/functional_fragment_ops_random_1 dEQP.GLES3/functional_fragment_ops_random_12 dEQP.GLES3/functional_fragment_ops_random_17 dEQP.GLES3/functional_fragment_ops_random_18 dEQP.GLES3/functional_fragment_ops_random_2 dEQP.GLES3/functional_fragment_ops_random_32 dEQP.GLES3/functional_fragment_ops_random_40 dEQP.GLES3/functional_fragment_ops_random_41 dEQP.GLES3/functional_fragment_ops_random_52 dEQP.GLES3/functional_fragment_ops_random_81 dEQP.GLES3/functional_fragment_ops_random_93 dEQP.GLES3/functional_fragment_ops_random_96

1.png

23 KB

View

Download

2.png

16 KB

View

Download

3.png

36 KB

View

Download

Comments

sy...@google.com <sy...@google.com> #2Sep 8, 2020 05:48PM

Reassigned to ph...@qualcomm.corp-partner.google.com.

Here's a renderdoc capture of the issue taken on a Pixel2XL with a very recent Android build.

android_bug_with_clear_attachment.rdc

298 KB

Download

ap...@google.com <ap...@google.com> #3Sep 25, 2020 11:49PM

Project: angle/angle
Branch: master

commit a3b16c6b50a64df299cfde89dcbe3068777a4adb
Author: Shahbaz Youssefi <syoussefi@chromium.org>
Date: Fri Aug 28 16:14:30 2020

Vulkan: Workaround vkCmdClearAttachment bug on Pixel

Adds a workaround to use draw calls to clear color instead of
vkCmdClearAttachment when the clear happens in the middle of render
pass. On Pixel phones, vkCmdClearAttachment races with the previous
draw calls in the render pass.

Bug:

b/166809097

Change-Id: I8c96b87793da191757635658ad4ee2c3a7875aca
Reviewed-on:

https://chromium-review.googlesource.com/c/angle/angle/+/2382416
Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Tim Van Patten <timvp@google.com>

M include/platform/FeaturesVk.h
M src/libANGLE/renderer/vulkan/FramebufferVk.cpp
M src/libANGLE/renderer/vulkan/RendererVk.cpp

https://chromium-review.googlesource.com/2382416

sy...@google.com <sy...@google.com> #4Oct 9, 2020 04:30AM

The issue is not limited to color clears. The following test added in this change exposes the same bug for depth/stencil clears:

ClearTestES3.ClearMaxAttachmentsAfterDraw/ES3_Vulkan

ti...@google.com <ti...@google.com> #5Jan 14, 2022 06:55PM

The following CL is enabling using vkCmdClearAttachements for scissored clears, rather than drawing: 3388635: Vulkan: Use vkCmdClearAttachments for scissored clears | https://chromium-review.googlesource.com/c/angle/angle/+/3388635

With that CL, the following tests expose another issue with Pixel 4 Qualcomm drivers:

FramebufferMultiviewLayeredClearTest.ScissoredClear/ES3_Vulkan__vertex_shader_multiview2
FramebufferMultiviewLayeredClearTest.ScissoredClear/ES3_Vulkan__vertex_shader_multiview

QC is not interpretting the VkClearRect.layerCount value incorrectly and is clearing too many layers. Looking at FramebufferMultiviewLayeredClearTest.ScissoredClear/ES3_Vulkan__vertex_shader_multiview specifically, the vkCmdClearAttachements command is being issued with VkClearRect.layerCount = 2, but is actually clearing layers 1, 2, and 3. Layer 0 is correctly being skipped.

To work around this, preferDrawClearOverVkCmdClearAttachments will be enabled for QC in general, rather than just specific Pixel devices.

ph...@qualcomm.corp-partner.google.com <ph...@qualcomm.corp-partner.google.com> #6Jun 7, 2023 06:11PM

Hi, can someone from the "Users who can edit, comment, and view" lists help to update the assignee to the VK POS ("digishp@qualcomm.corp-partner.google.com"). Thanks.

sy...@google.com <sy...@google.com> #7Jun 7, 2023 06:39PM

Reassigned to di...@qualcomm.corp-partner.google.com.

Done, though I got a warning that digishp@ might not be able to interact with the bug.

It's a public bug though, so :shrug:

di...@qualcomm.corp-partner.google.com <di...@qualcomm.corp-partner.google.com> #8Jun 9, 2023 06:24AM

we need to check if this is still present with latest driver or not.
is there any updated steps to reproduce this problem ? (I believe angle is having some workaround which we might need to disable)

sy...@google.com <sy...@google.com> #9Jun 9, 2023 05:20PM

Yes. I hope you are familiar with building and running tests with ANGLE?

Look for preferDrawClearOverVkCmdClearAttachments. In RendererVk.cpp, you would need to change the condition from isQualcommProprietary to false. Alternatively, you should be able to modify this behavior at run time by setting:

$ adb shell setprop debug.angle.feature_overrides_disabled preferDrawClearOverVkCmdClearAttachments

before running the test.

di...@qualcomm.corp-partner.google.com <di...@qualcomm.corp-partner.google.com> #10Jun 12, 2023 05:01AM

Internal QC JIRA created GLNEXT-10445

di...@qualcomm.corp-partner.google.com <di...@qualcomm.corp-partner.google.com> #11Jul 18, 2023 04:38AM

We could not run GLES CTS test case using 64 bit angle apk to confirm this.
is there any known problem with 64 bit angle apk to run CTS ?

we checked 32 bit Angle APK we were able to run CTS test case successfully however, it was not using the VkCmdClearAttachments so couldn't confirm the issue mentioned here.

sy...@google.com <sy...@google.com> #12Jul 18, 2023 04:03PM

is there any known problem with 64 bit angle apk to run CTS ?

Not that I know of, but I'll check.

we checked 32 bit Angle APK we were able to run CTS test case successfully however, it was not using the VkCmdClearAttachments so couldn't confirm the issue mentioned here.

Did you do $ adb shell setprop debug.angle.feature_overrides_disabled preferDrawClearOverVkCmdClearAttachments first?

sy...@google.com <sy...@google.com> #13Jul 18, 2023 04:39PM

@yuxinhu, since you are running dEQP over ANGLE recently, do you think you could share some insight on what should be done for it to work?

yu...@google.com <yu...@google.com> #14Jul 18, 2023 06:42PM

Yes Shabi, here are two ways I run dEQP over ANGLE on Pixel 4:

Option 1. Build Khronos CTS with ANGLE

Step 1. Build ANGLE

gn gen out/AndroidDebug
gn args out/AndroidDebug

Add these gn args in the editor and save the file

arget_os = "android"
target_cpu = "arm64"
android32_ndk_api_level = 29
android64_ndk_api_level = 29
angle_libs_suffix = "_angle"
is_component_build = false
symbol_level = 1
is_debug = false
dcheck_always_on = true
angle_enable_vulkan = true
angle_enable_trace = true
angle_expose_non_conformant_extensions_and_versions=true

Then run the command to build ANGLE:

autoninja -C out/AndroidDebug

Create a folder arm64-v8a under out/AndroidDebug and copy the angle libs to the folder:

mkdir arm64-v8a
cd arm64-v8a
cp ../lib*_angle.so .

Step 2. Build Khronos CTS package with ANGLE as the GLES driver:

Get the khronos CTS source code from https://github.com/KhronosGroup/VK-GL-CTS.

Sample command to build CTS for Android:

python3 scripts/android/build_apk.py --target=openglcts --sdk  ~/Projects/VK-GL-CTS-Android-Deps/android-sdk --ndk ~/Projects/VK-GL-CTS-Android-Deps/android-ndk-r25b --abis arm64-v8a --angle-path ~/angle/out/AndroidDebug/

Replace the values passed to --sdk and --ndk with the actual path that stores the android-sdk and android-ndk

Step 3. Install the Khronos CTS package:

python3 scripts/android/install_apk.py --target=openglcts

Step 4. Launch the test with the command: e.g. to run the test dEQP-GLES3.functional.fragment_ops.interaction.basic_shader.11:

adb shell 'am start -n org.khronos.gl_cts/android.app.NativeActivity -e cmdLine "cts --deqp-case=dEQP-GLES3.functional.fragment_ops.interaction.basic_shader.11 --deqp-log-filename=/sdcard/deqp-debug-log.qpa"'

Option 2. Run dEQP tests with ANGLE dEQP test runner

Step 1. Build ANGLE.

This will be the same as step 1. in Option 1. above.

Step 2. Launch the test with the command:

e.g. to run the test dEQP-GLES3.functional.fragment_ops.interaction.basic_shader.11:

out/AndroidDebug/angle_deqp_gles3_tests --gtest_filter="dEQP?GLES3.functional.fragment_ops.random.1" --verbose --local-output

If you are testing with Pixel 4, please remove this line in the src/tests/deqp_support/deqp_gles3_test_expectations.txt:

5990 PIXEL4ORXL VULKAN : dEQP-GLES3.functional.fragment_ops.random.* = SKIP

di...@qualcomm.corp-partner.google.com <di...@qualcomm.corp-partner.google.com> #15Jul 20, 2023 08:56AM

Thanks for response.

in 32 bit.
>> Did you do $ adb shell setprop debug.angle.feature_overrides_disabled preferDrawClearOverVkCmdClearAttachments first?
The way we were doing this still it was not using VkCmdClearAttachments from api dump and test was passing.

in 64 bit
it was failing many test cases due to image comparison mismatch with or with or without this option.

the procedure we used was.

adb install the 64-bit angle apk

adb shell settings put global angle_debug_package org.chromium.angle
adb shell settings put global angle_gl_driver_selection_values angle
adb shell settings put global angle_gl_driver_selection_pkgs com.drawelements.deqp

adb shell "am start -n com.drawelements.deqp/android.app.NativeActivity -e cmdLine 'deqp --deqp-case=dEQP-GLES3.functional.fragment_ops.interaction.basic_shader.* --deqp-log-filename=/sdcard/TestResults.qpa'"

yu...@google.com <yu...@google.com> #16Jul 20, 2023 09:05PM

Thank you digishp@.

We managed to reproduce the bug on Pixel 4 device: https://chromium-review.googlesource.com/c/angle/angle/+/4706144?tab=checks. Can you please try below steps and see if you can repro the issue on your test device?

Build ANGLE. Command and gn args to use is the same as the step 1. option 1. in comment #14. This should build a 64-bit angle apk, because the gn args used is target_cpu = "arm64".
In ANGLE root directory: run the test with this command:

out/AndroidDebug/angle_deqp_gles3_tests --gtest_filter="dEQP?GLES3.functional.fragment_ops.interaction.basic_shader.60" --verbose --local-output --skip-clear-data

Cherry pick the change in https://chromium-review.googlesource.com/c/angle/angle/+/4706144. Re-build ANGLE.
Run the same test in step 2:

out/AndroidDebug/angle_deqp_gles3_tests --gtest_filter="dEQP?GLES3.functional.fragment_ops.interaction.basic_shader.60" --verbose --local-output --skip-clear-data

We observe the test passed after step 2, but failed after step 4. Would you please let us know if you observe the same on your test device?

yu...@google.com <yu...@google.com> #17Jul 20, 2023 09:07PM

Alternative repro method on top of comment #16.

We also attached the angle_deqp_gles3_tests-debug.apk for you to try:

Install the apk: adb install -r -d --force-queryable angle_deqp_gles3_tests-debug.apk.
Launch the test with below command:

adb shell "am start -n com.android.angle.test/com.android.angle.test.AngleUnitTestActivity -e org.chromium.native_test.NativeTest.StdoutFile /sdcard/chromium_tests_root/out.txt -e org.chromium.native_test.NativeTest.CommandLineFlags '--gtest_filter=dEQP?GLES3.functional.fragment_ops.interaction.basic_shader.60'"

Pull the qpa result file from the test device:

adb pull /data/data/com.android.angle.test/TestResults.qpa

The qpa result we got from Pixel 4 indicating the test failed. If you use this method, could you please let us know if you observe the same result on your test device?

angle_deqp_gles3_tests-debug.apk

26 MB

Download

di...@qualcomm.corp-partner.google.com <di...@qualcomm.corp-partner.google.com> #18Jul 26, 2023 09:00AM

#1
-----
Regarding

comment#17

using angle_deqp_gles3_tests-debug.apk

We are seeing following error when trying the given steps with provided apk.

I NativeTest: runTests: --gtest_filter=dEQP?GLES3.functional.fragment_ops.interaction.basic_shader.60
E chromium: Failed to open stdout file: /sdcard/chromium_tests_root/out.txt: Operation not permitted

- Do we need to push some files for the test to run?
- We tried creating /sdcard/chromium_tests_root and giving all permissions to the folder
- Given Native Test app has all permissions

#2
-----
Another thing we were trying is to compiler APK at our end.
we are ending up in some problems.

To build angle from scratch we are following the process mentioned here.

https://android.googlesource.com/platform/packages/modules/ANGLE/+/refs/heads/main/Readme.md on 18.04.1-Ubuntu
We encountered build errors almost at each step and some of which are below -

While initializing Repo Client, we were seeing python syntax related issues-

.repo/repo/main.py", line 126
except ManifestInvalidRevisionError, e:
^
SyntaxError: invalid syntax

The repo revision that cloned during initializing was way older (~ v1.75) and incompatible with python 2.7 or python 3.6.

Solution:
We had to explicitly specify rev version from the arg like below
repo init -u

https://android.googlesource.com/platform/manifest --repo-rev v2.30

We saw such python related issues at different steps too.
Currently, we are stuck at the below error

$ ninja -v -C out/Default third_party/angle:angle_apks

ninja: Entering directory `out/Default'
[1/72] touch obj/third_party/android_deps/com_google_errorprone_error_prone_check_api_java__header.stamp
[2/72] python3 ../../build/android/gyp/turbine.py --depfile=gen/build/android/bytecode/bytecode_processor_java__header.d --generated-dir=gen/build/android/bytecode/bytecode_processor_java/generated_java --jar-path=obj/build/android/bytecode/bytecode_processor_java.turbine.jar --java-srcjars=\[\] --target-name //build/android/bytecode:bytecode_processor_java__header --classpath=@FileArg$gen/build/android/bytecode/bytecode_processor_java.build_config.json:javac:interface_classpath$ --processorpath=@FileArg$gen/build/android/bytecode/bytecode_processor_java.build_config.json:javac:processor_classpath$ --processors=@FileArg$gen/build/android/bytecode/bytecode_processor_java.build_config.json:javac:processor_classes$ --turbine-jar-path ../../third_party/turbine/turbine.jar --generated-jar-path gen/build/android/bytecode/bytecode_processor_java.generated.srcjar --chromium-code=1 --warnings-as-errors @gen/build/android/bytecode/bytecode_processor_java.sources
FAILED: obj/build/android/bytecode/bytecode_processor_java.turbine.jar gen/build/android/bytecode/bytecode_processor_java.generated.srcjar
python3 ../../build/android/gyp/turbine.py --depfile=gen/build/android/bytecode/bytecode_processor_java__header.d --generated-dir=gen/build/android/bytecode/bytecode_processor_java/generated_java --jar-path=obj/build/android/bytecode/bytecode_processor_java.turbine.jar --java-srcjars=\[\] --target-name //build/android/bytecode:bytecode_processor_java__header --classpath=@FileArg$gen/build/android/bytecode/bytecode_processor_java.build_config.json:javac:interface_classpath$ --processorpath=@FileArg$gen/build/android/bytecode/bytecode_processor_java.build_config.json:javac:processor_classpath$ --processors=@FileArg$gen/build/android/bytecode/bytecode_processor_java.build_config.json:javac:processor_classes$ --turbine-jar-path ../../third_party/turbine/turbine.jar --generated-jar-path gen/build/android/bytecode/bytecode_processor_java.generated.srcjar --chromium-code=1 --warnings-as-errors @gen/build/android/bytecode/bytecode_processor_java.sources
Traceback (most recent call last):
File "../../build/android/gyp/turbine.py", line 14, in <module>
import compile_java
File "/local/mnt/workspace/ANGLE/chromium/src/build/android/gyp/compile_java.py", line 19, in <module>
import javac_output_processor
File "/local/mnt/workspace/ANGLE/chromium/src/build/android/gyp/javac_output_processor.py", line 15, in <module>
from util import dep_utils
File "/local/mnt/workspace/ANGLE/chromium/src/build/android/gyp/util/dep_utils.py", line 6
from __future__ import annotations
^
SyntaxError: future feature annotations is not defined

-Are these known problems ?
-Is there any specific python version that we should have for building angle apk?
-Are the steps mentioned at

https://android.googlesource.com/platform/packages/modules/ANGLE/+/refs/heads/main/Readme.md updated?

yu...@google.com <yu...@google.com> #19Jul 28, 2023 12:59AM

Hi digishp@,

Thank you for the update.

#1 Apologize for the missing information in the instruction. It turns out some additional files and permission settings are required when running the test directly from the apk. Please try below steps:

Enable adb root on the testing device: adb root.
Install the test apk: adb install -r -d -g --force-queryable ~/Downloads/angle_deqp_gles3_tests-debug.apk
Set storage permission of the test app: adb shell appops set com.android.angle.test MANAGE_EXTERNAL_STORAGE allow || true
Download the attached zip file chromium_tests_root.zip and unzip the file.
Push the unzipped file to the /sdcard/ directory of your testing device: adb push chromium_tests_root /sdcard/
Launch test:

adb shell "am start -n com.android.angle.test/com.android.angle.test.AngleUnitTestActivity -e org.chromium.native_test.NativeTest.StdoutFile /sdcard/chromium_tests_root/out.txt -e org.chromium.native_test.NativeTest.CommandLineFlags '--gtest_filter=dEQP?GLES3.functional.fragment_ops.interaction.basic_shader.60'"

Pull the test results from the testing device:

adb pull /data/data/com.android.angle.test/TestResults.qpa

#2 The doc you referenced to https://android.googlesource.com/platform/packages/modules/ANGLE/+/refs/heads/main/Readme.md are outdated. Could you please try following below steps instead?

Refer to DevSetup to sync ANGLE source code and setup build tools, specifically, follow the steps in Version Control, Required First Setup, and Getting the source.
Refer to Setting up the ANGLE build for Android to download Android build dependencies and setup android build args.
Build ANGLE for Android with command autoninja -C out/Android.
Launch the test with command out/Android/angle_deqp_gles3_tests --gtest_filter="dEQP?GLES3.functional.fragment_ops.interaction.basic_shader.60" --verbose --local-output --skip-clear-data

Please let us know if this works for you.

chromium_tests_root.zip

458 KB

Download

ma...@gmail.com <ma...@gmail.com> #20Jan 20, 2024 06:51AM

please give me angle apk for android 10

IssueTracker

This issue is a part of the ANGLE tracker.