QualitySelector fails to use UHD when a fallback strategy is enabled [230651237]

Fixed

Bug

Status Update

No update yet.

Description

da...@gmail.com

created issue #1

Apr 27, 2022 09:12PM

When we create a QualitySelector with the following code, it doesn't use UHD even when UHD is supported:

QualitySelector.fromOrderedList(Quality.UHD, FallbackStrategy.lowerQualityOrHigherThan(Quality.UHD)

We had to implement a workaround for this in our app:

https://github.com/GrapheneOS/Camera

// work around CameraX QualitySelector bug
private fun Quality.withFallbackIfUnsupported(): QualitySelector {
    return if (QualitySelector.isQualitySupported(camera!!.cameraInfo, this)) {
        QualitySelector.from(this)
    } else {
        QualitySelector.from(this, FallbackStrategy.higherQualityOrLowerThan(this))
    }
}

This works but led to a regression (a crash) because camera is null when the app is being opened immediately to video mode via the standard Android video capture intent. We'll try some other approaches to using QualitySector but it doesn't seem that FallbackStrategy works correctly at the moment.

Our app is open source if for some reason you can't reproduce this yourselves.

https://github.com/GrapheneOS/Camera

Comments

da...@gmail.com <da...@gmail.com> #2Apr 27, 2022 10:33PM

This was partially addressed by

https://android-review.googlesource.com/c/platform/frameworks/support/+/2076902, available now in 1.1 RC02, see

b/230665435

. As compilation on user builds requires a full target reinstall (which is a behavior change), we offer an opt out, which can be used to accomplish this feature request: 1) configure every macrobench to use `CompilationMode.Full()` 1) manually issuing the compile command `cmd package compile -f -m speed <packagename>` for your target 1) pass the instrumentation arg `androidx.benchmark.compilation.enable` = `false` to skip compilation/reinstall for each macrobenchmark. This should still give you the numbers you've been seeing, while avoiding the cost of a large AOT each test. Leaving this bug open, since in general we should be able to do this more automatically for everything without warmup driven profiles. (Somewhat related bug - there have been excess compilations issued specifically for `Compilation.None`, `StartupMode.COLD` benchmarks, which has been fixed, but not shipped publicly yet:

b/231976084

)

da...@gmail.com <da...@gmail.com> #3Apr 27, 2022 10:39PM

Note: ideally there's be a way to turn off all install related behavior without having to pass in an arg flag. If I want to handle installation & optimization without benchmark getting in the way, there should be an API for that. Arguably that's what Compilation.None should do, or there should be a Compilation.Interpreted and a Compilation.NoOp.

da...@gmail.com <da...@gmail.com> #4Apr 27, 2022 11:40PM

Also, generally clearing profiles doesn't make much sense for Full compilation mode. I understand this makes sense for profile based benchmarks, but not really for full where the output shouldn't vary.

wu...@google.com <wu...@google.com> #5Apr 28, 2022 06:44AM

Reassigned to da...@gmail.com.

> ideally there's be a way to turn off all install related behavior without having to pass in an arg flag. We can't change APIs in an RC, but we'll consider it for 1.2 > Arguably that's what Compilation.None should do, or there should be a Compilation.Interpreted and a Compilation.NoOp. `None` means "none of my app should be compiled", so at least a reset (if compilation has happened before) should occur. `Compilation.NoOp` or equivalent sounds reasonable. > Also, generally clearing profiles doesn't make much sense for Full compilation mode. I assume so too, I would want to validate that with ART and test it, but either way, we opted for full reinstallation for every test in 1.1 since it's predictable and consistent.

da...@gmail.com <da...@gmail.com> #6Apr 28, 2022 06:47AM

Project: platform/frameworks/support
Branch: androidx-main

commit 78a3847fafbb5a810b60b27b1361400bf8dcfa58
Author: Marcello Albano <maralb@google.com>
Date: Mon Jul 11 14:07:07 2022

Added a new compilation mode `CompilationMode.Ignore` to skip profile reset and compilation.

Bug: 230453509
Test: ./gradlew :benchmark:integration-tests:macrobenchmark:cC -Pandroid.testInstrumentationRunnerArguments.class=androidx.benchmark.integration.macrobenchmark.CompilationModeTest

This change adds a new compilation mode `CompilationMode.Ignore` that will skip profile reset and compilation. This can be used to customize the benchmark loop.

Change-Id: Ibbcf8e8c627d227e0f290bddf464b67d9b6653bc
Relnote: Added a new compilation mode `CompilationMode.Ignore` to skip profile reset and compilation.

A benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/ExperimentalMacrobenchmarkApi.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/CompilationMode.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MacrobenchmarkTest.kt
A benchmark/integration-tests/macrobenchmark/src/androidTest/java/androidx/benchmark/integration/macrobenchmark/CompilationModeTest.kt
M benchmark/benchmark-macro/api/public_plus_experimental_current.txt

https://android-review.googlesource.com/2150519

wu...@google.com <wu...@google.com> #7Apr 28, 2022 06:51AM

Reassigned to ca...@google.com.

Thanks for the update. We will take a look and update the status here!

le...@google.com <le...@google.com> #8Apr 28, 2022 11:25AM

Hi,

Thanks for the feedback. QualitySelector.isQualitySupported() can only know whether the device support this quality, but the final selected quality is still constrained by UseCase combinations when ProcessCameraProvider.bindToLifecycle is called. It depends on camera hardware level and the supported surface combinations defined in camera2 CameraDevice regular-capture.

For example, assume isQualitySupported(UHD) returns true on a device. When trying to bind use cases with QualitySelector only contains UHD, you may get below results for different combinations

Preview + VideoCapture(UHD) => success
Preview + ImageCapture + VideoCapture(UHD) => fail
Preview + ImageCapture + VideoCapture(FHD) => success

In your case, I think you want to know in advance what qualities are supported under a certain use cases combination (ex: Preview + VideoCapture). However, there is not yet a convenient API to query about this. Currently you can create a VideoCapture with a single quality (QualitySelector.from(quality)) and then bind the use cases you want, use try-catch on ProcessCameraProvider.bindToLifecycle() to know if the combination is supported (ProcessCameraProvider.bindToLifecycle() will throw an exception if the combination is not supported).

However, this method will open camera when the combination is supported. It would be better to have an API that only checks for support without opening camera. There is a bug related to such API requirement. See https://issuetracker.google.com/204274197. Add you to the CC list for updates.

Let us know if you have any questions.

Leo

da...@gmail.com <da...@gmail.com> #9Apr 28, 2022 11:01PM

Pixels appear to be capable of taking photos while recording 4k video and we're still seeing 4k resolution not being used on them when we try to use QualitySelector with UHD as the main choice and FHD as a fallback. It's likely that what you're describing does occur on some devices but I don't think it explains the QualitySelector issue we're seeing on Pixels.

da...@gmail.com <da...@gmail.com> #10Apr 28, 2022 11:08PM

If you try our app from https://github.com/GrapheneOS/Camera, it's able to use 4k recording on Pixels while doing image capture. We do fall back to video + preview if video + preview + image capture fails to load, but it doesn't happen for 4k on Pixels when using QualitySelector.from(Quality.UHD), only when doing QualitySelector.from(Quality.UHD, fallback) or using a list of qualities. It seems like QualitySelector never uses the first choice when there are fallbacks. That's the behavior we're seeing at least. Are you not able to replicate it?

le...@google.com <le...@google.com> #11Apr 29, 2022 07:58AM

Reassigned to le...@google.com.

This sounds like a bug. I will investigate it and get back to you.

Thanks

le...@google.com <le...@google.com> May 3, 2022 05:56AM

Accepted by le...@google.com.

le...@google.com <le...@google.com> #12May 3, 2022 12:04PM

The issue happens when combining ImageCapture + VideoCapture.

In a FULL level camera device, there are two possible combinations for Preview + VideoCapture + ImageCapture. (refer to CameraDevice regular-capture.)

(PRIV, PREVIEW) + (PRIV, RECORD) + (JPEG, RECORD)
(PRIV, PREVIEW) + (PRIV, PREVIEW) + (JPEG, MAXIMUM)

MAXIMUM > RECORD > PREVIEW

In the current CameraX settings, ImageCapture has higher surface occupancy priority than VideoCapture by default. That means ImageCapture has higher priority to choose resolution than VideoCapture. When QualitySelector contains multiple qualities as candidates, such as

QualitySelector.from(Quality.UHD, FallbackStrategy.lowerQualityOrHigherThan(Quality.UHD)))

It means VideoCapture can accept UHD, FHD, HD, SD. CameraX auto-resolution mechanism will then adopt combination (2) because ImageCapture has higher priority. i.e. ImageCapture gets MAXIMUM size and VideoCapture gets Preview size (FHD).

For

QualitySelector.from(Quality.UHD)

Since there is only one option UHD (RECORD size) for VideoCapture, combination (2) can't be met and combination (1) is adopted. i.e. both ImageCapture and VideoCapture gets RECORD size.

will have a solution to fix this.

ap...@google.com <ap...@google.com> #13May 5, 2022 01:02PM

Project: platform/frameworks/support
Branch: androidx-main

commit 76d3d536e73a8c119534d7fbb2110e29bacc3706
Author: leo huang <leohuang@google.com>
Date: Tue May 03 00:54:01 2022

Increase VideoCapture surface occupancy priority over ImageCapture

Root Cause:
In a FULL (or higher) hardware level camera device, there are two possible combinations for Preview + VideoCapture + ImageCapture
(1) (PRIV, PREVIEW) + (PRIV, RECORD) + (JPEG, RECORD)
(2) (PRIV, PREVIEW) + (PRIV, PREVIEW) + (JPEG, MAXIMUM)

In the current code, ImageCapture has higher surface occupancy priority than VideoCapture by default. When QualitySelector contains multiple qualities as candidates, such as using QualitySelector.from(Quality.UHD, FallbackStrategy.lowerQualityOrHigherThan(Quality.UHD))) as the bug mentioned. The CameraX auto-resolution mechanism will give ImageCapture higher priority than VideoCapture to select the largest resolution it can use. So combination (2) is adopted and VideoCapture obtains PREVIEW size, i.e VideoCapture gets FHD but not UHD.

Solution:
For scenario using VideoCapture + ImageCapture, it makes sense to give VideoCapture higher surface occupancy priority than ImageCapture. Then combination (1) can be adopted for the issue case.

The CL also removes impossible resolutions from prioritized resolution list to prevent auto-resolution mechanism from incorrectly assuming that VideoCapture might use them. This ensure ImageCapture can still get the MAXIMUM size when VideoCapture only requests a surface resolution under PREVIEW size for combination (2).

Relnote: "Fixed QualitySelector fails to record a UHD video when a fallback strategy is enabled. The issue happens when VideoCapture is bound with ImageCapture and Preview on a FULL or higher hardware level camera device. A fallback strategy of QualitySelector causes VideoCapture incorrectly to get a FHD resolution. UHD resolution is actually supported for this use case combination and should be adopted."

Bug:

b/230651237

Test: ./gradlew camera:camera-video:testDebug
Change-Id: I0e788060512feef5fa934d7f35374da218ffe8e8

M camera/camera-camera2/build.gradle
M camera/camera-video/src/main/java/androidx/camera/video/VideoCapabilities.java
M camera/camera-camera2/src/test/java/androidx/camera/camera2/internal/SupportedSurfaceCombinationTest.java
M camera/camera-video/src/main/java/androidx/camera/video/VideoCapture.java
M camera/camera-video/src/test/java/androidx/camera/video/VideoCaptureTest.kt

https://android-review.googlesource.com/2083983

le...@google.com <le...@google.com> #14May 10, 2022 03:28AM

Marked as fixed.

The fix will be included in June release.