Capture YUV frames at higher resolution [192911449]

Fixed

Bug

Status Update

No update yet.

Description

oa...@gmail.com

created issue #1

Jul 7, 2021 06:49AM

I understand that for capturing YUV frames the ImageAnalysis use case should be used. I saw that it's output is limited to 1920x1080 resolution.

Now we are trying to switch from Camera2 API to CameraX.
Previous with Camera2 I was able to choose a higher resolution for the YUV images. We use those images with MediaCodec and MediaMuxer to make an mp4 and is important for us to have images with higher resolution.
Also we checked the available resolution using this API before setting one of them:

val map = characteristics.get(
CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP
val resolutions: Array<Size> = map.getOutputSizes(ImageFormat.YUV_420_888)

Is there a possibility to achieve this with CameraX? Or will it be available in the future a way to capture YUV images with a higher resolution?

Comments

wu...@google.com <wu...@google.com> Jul 7, 2021 07:36AM

Reassigned to ch...@google.com.

ch...@google.com <ch...@google.com> #2Jul 9, 2021 02:12AM

Reassigned to oa...@gmail.com.

Project: platform/frameworks/support
Branch: androidx-master-dev

commit c60f33e229e31ab328ef6b59dab63b264954831c
Author: Alexandre Elias <aelias@google.com>
Date: Fri Jul 10 16:23:09 2020

Semantics no-op cleanups

Partly in response to lmr's broad code review, I did a pass of
superficial API/implementation cleanup. The main changes are:

- I changed each Boolean SemanticsProperty where false is equivalent to
not being present to take "Unit" instead. This is conceptually
clearer: it avoids questions like "can I cancel out a semantics from a
merged child by setting it to false?" Because "property = Unit" looks
weird, I also changed the style of these to "property()".

- I moved the Semantics id generator closer to where it's used, in
SemanticsModifierCore. I made it internal and an AtomicInt.
(Note that integer ids are heavily used in the Android
AccessibilityNodeInfo APIs so I can't simply remove them entirely.)

- I deleted dead code. Some examples include SemanticsHintOverrides,
a public API not connected to anything, and SemanticsPropertyKey
merge() open method which is never called. (In both cases I have
a different plan in mind for accessibility.)

Fixes: 145951226
Fixes: 145955412
Test: existing tests
Relnote: "Single-value semantics properties now use a calling style.
For example, 'semantics { hidden = true }' is now written as:
'semantics { hidden() }'."

Change-Id: Ic1afd12ea22c926babc9662f1804d80b33aa0cfc

M ui/integration-tests/benchmark/src/androidTest/java/androidx/ui/benchmark/test/LayoutNodeModifierBenchmark.kt
M ui/ui-core/api/0.1.0-dev15.txt
M ui/ui-core/api/current.txt
M ui/ui-core/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-core/api/public_plus_experimental_current.txt
M ui/ui-core/api/restricted_0.1.0-dev15.txt
M ui/ui-core/api/restricted_current.txt
M ui/ui-core/src/androidAndroidTest/kotlin/androidx/ui/graphics/vector/VectorTest.kt
M ui/ui-core/src/androidAndroidTest/kotlin/androidx/ui/semantics/SemanticsTests.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidActuals.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidComposeView.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidComposeViewAccessibilityDelegateCompat.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidPopup.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/Expect.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsConfiguration.kt
D ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsHintOverrides.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsModifier.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsNode.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsOwner.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsWrapper.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/semantics/SemanticsProperties.kt
M ui/ui-foundation/api/0.1.0-dev15.txt
M ui/ui-foundation/api/current.txt
M ui/ui-foundation/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-foundation/api/public_plus_experimental_current.txt
M ui/ui-foundation/api/restricted_0.1.0-dev15.txt
M ui/ui-foundation/api/restricted_current.txt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Clickable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Dialog.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Scroller.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/selection/Selectable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/selection/Toggleable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/semantics/FoundationSemanticsProperties.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ButtonTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/CardTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/CheckboxScreenshotTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/RadioButtonScreenshotTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ScaffoldTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/SnackbarTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/SurfaceTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ripple/RippleIndicationTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/textfield/TextFieldScreenshotTest.kt
M ui/ui-material/src/main/java/androidx/ui/material/AppBar.kt
M ui/ui-material/src/main/java/androidx/ui/material/TextFieldImpl.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/AssertsTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/CallSemanticsActionTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/ErrorMessagesTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/FindersTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/PrintToStringTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/ScrollToTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/TextActionsTest.kt
M ui/ui-test/src/main/java/androidx/ui/test/Actions.kt
M ui/ui-test/src/main/java/androidx/ui/test/Filters.kt
M ui/ui-text/api/0.1.0-dev15.txt
M ui/ui-text/api/current.txt
M ui/ui-text/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-text/api/public_plus_experimental_current.txt
M ui/ui-text/api/restricted_0.1.0-dev15.txt
M ui/ui-text/api/restricted_current.txt
M ui/ui-text/src/commonMain/kotlin/androidx/ui/text/CoreTextField.kt
M ui/ui-text/src/commonMain/kotlin/androidx/ui/text/TextSemanticsProperties.kt

https://android-review.googlesource.com/1360099

oa...@gmail.com <oa...@gmail.com> #3Jul 12, 2021 08:14AM

Hi charcoalchen,

Thank you for your quick response!

The app is used to record trips. We display to the user the camera preview first. The user can start recording a trip, then we start encoding images with ImageAnalysis at a given interval in an mp4 file using MediaMuxer and MediaCodec to keep the storage size small and create a video with the whole trip. Later, users have the possibility to get the video file or a specific image from the video which is important to have a higher resolution.

1. We are using this ImageAnalysis with the Preview use case. But we are open to other approaches too in order to maintain a higher resolution.
2. For the hardware level we didn't limited the supported devices. Our min sdk version is 22. In the previous version of the app, we used camera1 API with devices having the hardware level LEGACY, because the performance was better even if the camera2 API is exposing approximately the same capabilities and for the other hardware levels we used camera2 API. So, a combination of both APIs depending on the hardware level.

We don't need to record an actual video, just some frames that we can trigger when should be taken by our logic. Is this possible maybe using other use cases like VideoCapture? (Also, there is no need for audio.) Or what is the hidden API to remove the limitation for 1080p? I've tested the ImageAnalysis use case alone without a preview and is still limited to 1080p.

Testing device: Samsung S10 SM-G973F, Android 11

ch...@google.com <ch...@google.com> #4Jul 12, 2021 08:40AM

Thanks for providing the information.

We're discussing to remove the 1080p restriction from ImageAnalysis. Will let you know when we have the final conclusion.

But, even if only Preview + ImageAnalysis are bound, a LEGACY-level device can only support the following configuration.

PRIV/PREVIEW + YUV/PREVIEW		Preview plus in-app processing.

Some devices might be able to support extra (larger) configurations, but not all devices. There is no API to know that. Therefore, CameraX can only select a PREVIEW size resolution for ImageAnalysis. Didn't you encounter some LEGACY-level devices that can't support 1080p above resolutions when using Camera 1 API?

oa...@gmail.com <oa...@gmail.com> #5Jul 12, 2021 10:02AM

Actually all the LEGACY devices that we used with camera1 API didn't support a higher resolution than 1080p because the frames were tight with the preview stream. This is not a problem, but we want to allow for those devices that have the capability, to select a higher resolution.

With camera2 we used to display all the possible resolutions that are 16:9 for the YUV format using:

val map = characteristics.get(
CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP
val resolutions: Array<Size> = map.getOutputSizes(ImageFormat.YUV_420_888)

So with this the user could select a higher resolution than 1080p if the device had that capability.

ch...@google.com <ch...@google.com> #6Jul 12, 2021 10:21AM

Reassigned to ch...@google.com.

Thanks for the clarification. It looks that the Camera 2 LEGACY-level configuration restriction won't impact your app. I will let you know the result of whether ImageAnalysis will remove the 1080p restriction.

oa...@gmail.com <oa...@gmail.com> #7Jul 12, 2021 07:36PM

Thanks! I'm waiting for a better solution for this. In the meantime we'll use the restricted API ImageCapture.Builder#setBufferFormat(int) with the YUV format.

ap...@google.com <ap...@google.com> #8Jul 15, 2021 03:36AM

Project: platform/frameworks/support
Branch: androidx-main

commit dc02f68f180b9b884c3ae361ab2cceb9daddb69c
Author: Charcoal Chen <charcoalchen@google.com>
Date: Fri Jul 09 17:08:41 2021

Allows ImageAnalysis to select a resolution larger than 1080p

Remove the 1080p max resolution setting from ImageAnalysis.

- A LEGACY-level device can still only support PRIV/PREVIEW + YUV/PREVIEW + JPEG/MAXIMUM configuration for Preview + ImageAnalysis + ImageCapture. A resolution larger than 1080p can't be selected is due to the device capability but not the restriction of CameraX.
- LIMITED-level above devices can support PRIV/PREVIEW + YUV/RECORD + JPEG/RECORD configuration for Preview + ImageAnalysis + ImageCapture. Since this configuration is listed in the guaranteed supported configurations tables, it should be safe to use a larger resolution for ImageAnalysis.
- The default 640x480 resolution setting of ImageAnalysis is not changed. This CL should not impact the selected resolutions of all use cases when the application doesn't explicitly specify they want a larger resolution for ImageAnalysis.
- The selected resolution of ImageCapture will only be affected when the application explicitly specifies that it wants a larger resolution for ImageAnalysis rather than a MAXIMUM resolution for ImageCapture.

Relnote: "Allows ImageAnalysis to select a resolution larger than 1080p. A LIMITED-level above device can support a RECORD size resolution for ImageAnalysis when it is bound together with Preview and ImageCapture. The trade-off is the selected resolution for the ImageCapture will also need to be a RECORD size resolution. To successfully select a RECORD size resolution for ImageAnalysis, a RECORD size target resolution should be set on both ImageCapture and ImageAnalysis. This indicates that the application clearly understand the trade-off and prefer the ImageAnalysis to have a larger resolution rather than the ImageCapture to have a MAXIMUM resolution. For the definitions of RECORD, MAXIMUM sizes and more details see

https://developer.android.com/reference/android/hardware/camera2/CameraDevice#regular-capture. The RECORD size refers to the camera device's maximum supported recording resolution, as determined by CamcorderProfile. The MAXIMUM size refers to the camera device's maximum output resolution for that format or target from StreamConfigurationMap.getOutputSizes(int)."

Bug: 192911449
Test: ./gradlew bOS
Change-Id: I1ee973a50f9605324f06f406ca2230a464ed2363

M camera/camera-camera2/src/test/java/androidx/camera/camera2/internal/SupportedSurfaceCombinationTest.java
M camera/camera-core/src/main/java/androidx/camera/core/ImageAnalysis.java
M camera/camera-core/src/main/java/androidx/camera/core/ImageCapture.java
M camera/camera-core/src/main/java/androidx/camera/core/Preview.java

https://android-review.googlesource.com/1762582

oa...@gmail.com <oa...@gmail.com> #9Jul 15, 2021 07:06AM

Thank you charcoalchen for this quick change!

I have some questions, though, when not all 3 use cases are bound to the camera.
Currently the config for PRIV + YUV looks like:
LEGACY: PRIV/PREVIEW + YUV/PREVIEW
LIMITED: PRIV/PREVIEW + YUV/RECORD
FULL: PRIV/PREVIEW + YUV/MAXIMUM

1. So, after this commit will this config be available?
LEGACY: PRIV/PREVIEW + YUV/PREVIEW
LIMITED and FULL: PRIV/PREVIEW + YUV/MAXIMUM

I'm interested if for LIMITED devices now I can select the resolution from StreamConfigurationMap.getOutputSizes(int) as it was before with camera 2 API or is still by using CamcorderProfile.

2. Do you have a timeline when this change will be released to the public? This is a release blocker for us.

ch...@google.com <ch...@google.com> #10Jul 15, 2021 07:47AM

Marked as fixed.

For question 1:

LEGACY: PRIV/PREVIEW + YUV/PREVIEW => Yes
LIMITED and FULL: PRIV/PREVIEW + YUV/MAXIMUM => Only FULL-level devices can support the combination. The LIMITED-level devices can only support PRIV/PREVIEW + YUV/RECORD.

For LIMITED device, the YUV stream size is still restricted by CamcorderProfile. StreamConfigurationMap.getOutputSizes(int) provides all supported sizes without considering the streams combination. It will return the MAXIMUM size, but the resolution can only be available when binding only one ImageAnalysis (YUV) use case. The configuration is listed in the LEGACY-level guaranteed configurations table.

YUV	MAXIMUM			In-application video/image processing.

For question 2:

This commit will be included in camera-core 1.1.0-alpha08 which will be release around end of August. As a workaround, you may be able to use the restricted ImageAnalysis.Builder#setMaxResolution(Size) to set MAXIMUM resolution to the ImageAnalysis before 1.1.0-alpha08 is released. After 1.1.0-alpha08 is release, please remove the workaround since the restricted API might be changed or removed in the future.

oa...@gmail.com <oa...@gmail.com> #11Jul 15, 2021 11:53AM

Thanks for the info, it sounds really good!
Great job guys!