Capture YUV frames at higher resolution [192911449]

Fixed

Bug

Status Update

No update yet.

Description

oa...@gmail.com

created issue #1

Jul 7, 2021 06:49AM

I understand that for capturing YUV frames the ImageAnalysis use case should be used. I saw that it's output is limited to 1920x1080 resolution.

Now we are trying to switch from Camera2 API to CameraX.
Previous with Camera2 I was able to choose a higher resolution for the YUV images. We use those images with MediaCodec and MediaMuxer to make an mp4 and is important for us to have images with higher resolution.
Also we checked the available resolution using this API before setting one of them:

val map = characteristics.get(
CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP
val resolutions: Array<Size> = map.getOutputSizes(ImageFormat.YUV_420_888)

Is there a possibility to achieve this with CameraX? Or will it be available in the future a way to capture YUV images with a higher resolution?

Comments

wu...@google.com <wu...@google.com> Jul 7, 2021 07:36AM

Reassigned to ch...@google.com.

ch...@google.com <ch...@google.com> #2Jul 9, 2021 02:12AM

Reassigned to oa...@gmail.com.

Hi Ed, Thank you so much for these suggestions. I've been reviewing them and merging them in. Hopefully it should be live. I've included a thank you note too in the article.

oa...@gmail.com <oa...@gmail.com> #3Jul 12, 2021 08:14AM

Great! Thanks a lot, I'll look for the live updates soon!

ch...@google.com <ch...@google.com> #4Jul 12, 2021 08:40AM

Thanks for providing the information.

We're discussing to remove the 1080p restriction from ImageAnalysis. Will let you know when we have the final conclusion.

But, even if only Preview + ImageAnalysis are bound, a LEGACY-level device can only support the following configuration.

PRIV/PREVIEW + YUV/PREVIEW		Preview plus in-app processing.

Some devices might be able to support extra (larger) configurations, but not all devices. There is no API to know that. Therefore, CameraX can only select a PREVIEW size resolution for ImageAnalysis. Didn't you encounter some LEGACY-level devices that can't support 1080p above resolutions when using Camera 1 API?

oa...@gmail.com <oa...@gmail.com> #5Jul 12, 2021 10:02AM

Actually all the LEGACY devices that we used with camera1 API didn't support a higher resolution than 1080p because the frames were tight with the preview stream. This is not a problem, but we want to allow for those devices that have the capability, to select a higher resolution.

With camera2 we used to display all the possible resolutions that are 16:9 for the YUV format using:

val map = characteristics.get(
CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP
val resolutions: Array<Size> = map.getOutputSizes(ImageFormat.YUV_420_888)

So with this the user could select a higher resolution than 1080p if the device had that capability.

ch...@google.com <ch...@google.com> #6Jul 12, 2021 10:21AM

Reassigned to ch...@google.com.

Thanks for the clarification. It looks that the Camera 2 LEGACY-level configuration restriction won't impact your app. I will let you know the result of whether ImageAnalysis will remove the 1080p restriction.

oa...@gmail.com <oa...@gmail.com> #7Jul 12, 2021 07:36PM

Thanks! I'm waiting for a better solution for this. In the meantime we'll use the restricted API ImageCapture.Builder#setBufferFormat(int) with the YUV format.

ap...@google.com <ap...@google.com> #8Jul 15, 2021 03:36AM

Project: platform/frameworks/support
Branch: androidx-main

commit dc02f68f180b9b884c3ae361ab2cceb9daddb69c
Author: Charcoal Chen <charcoalchen@google.com>
Date: Fri Jul 09 17:08:41 2021

Allows ImageAnalysis to select a resolution larger than 1080p

Remove the 1080p max resolution setting from ImageAnalysis.

- A LEGACY-level device can still only support PRIV/PREVIEW + YUV/PREVIEW + JPEG/MAXIMUM configuration for Preview + ImageAnalysis + ImageCapture. A resolution larger than 1080p can't be selected is due to the device capability but not the restriction of CameraX.
- LIMITED-level above devices can support PRIV/PREVIEW + YUV/RECORD + JPEG/RECORD configuration for Preview + ImageAnalysis + ImageCapture. Since this configuration is listed in the guaranteed supported configurations tables, it should be safe to use a larger resolution for ImageAnalysis.
- The default 640x480 resolution setting of ImageAnalysis is not changed. This CL should not impact the selected resolutions of all use cases when the application doesn't explicitly specify they want a larger resolution for ImageAnalysis.
- The selected resolution of ImageCapture will only be affected when the application explicitly specifies that it wants a larger resolution for ImageAnalysis rather than a MAXIMUM resolution for ImageCapture.

Relnote: "Allows ImageAnalysis to select a resolution larger than 1080p. A LIMITED-level above device can support a RECORD size resolution for ImageAnalysis when it is bound together with Preview and ImageCapture. The trade-off is the selected resolution for the ImageCapture will also need to be a RECORD size resolution. To successfully select a RECORD size resolution for ImageAnalysis, a RECORD size target resolution should be set on both ImageCapture and ImageAnalysis. This indicates that the application clearly understand the trade-off and prefer the ImageAnalysis to have a larger resolution rather than the ImageCapture to have a MAXIMUM resolution. For the definitions of RECORD, MAXIMUM sizes and more details see

https://developer.android.com/reference/android/hardware/camera2/CameraDevice#regular-capture. The RECORD size refers to the camera device's maximum supported recording resolution, as determined by CamcorderProfile. The MAXIMUM size refers to the camera device's maximum output resolution for that format or target from StreamConfigurationMap.getOutputSizes(int)."

Bug: 192911449
Test: ./gradlew bOS
Change-Id: I1ee973a50f9605324f06f406ca2230a464ed2363

M camera/camera-camera2/src/test/java/androidx/camera/camera2/internal/SupportedSurfaceCombinationTest.java
M camera/camera-core/src/main/java/androidx/camera/core/ImageAnalysis.java
M camera/camera-core/src/main/java/androidx/camera/core/ImageCapture.java
M camera/camera-core/src/main/java/androidx/camera/core/Preview.java

https://android-review.googlesource.com/1762582

oa...@gmail.com <oa...@gmail.com> #9Jul 15, 2021 07:06AM

Thank you charcoalchen for this quick change!

I have some questions, though, when not all 3 use cases are bound to the camera.
Currently the config for PRIV + YUV looks like:
LEGACY: PRIV/PREVIEW + YUV/PREVIEW
LIMITED: PRIV/PREVIEW + YUV/RECORD
FULL: PRIV/PREVIEW + YUV/MAXIMUM

1. So, after this commit will this config be available?
LEGACY: PRIV/PREVIEW + YUV/PREVIEW
LIMITED and FULL: PRIV/PREVIEW + YUV/MAXIMUM

I'm interested if for LIMITED devices now I can select the resolution from StreamConfigurationMap.getOutputSizes(int) as it was before with camera 2 API or is still by using CamcorderProfile.

2. Do you have a timeline when this change will be released to the public? This is a release blocker for us.

ch...@google.com <ch...@google.com> #10Jul 15, 2021 07:47AM

Marked as fixed.

For question 1:

LEGACY: PRIV/PREVIEW + YUV/PREVIEW => Yes
LIMITED and FULL: PRIV/PREVIEW + YUV/MAXIMUM => Only FULL-level devices can support the combination. The LIMITED-level devices can only support PRIV/PREVIEW + YUV/RECORD.

For LIMITED device, the YUV stream size is still restricted by CamcorderProfile. StreamConfigurationMap.getOutputSizes(int) provides all supported sizes without considering the streams combination. It will return the MAXIMUM size, but the resolution can only be available when binding only one ImageAnalysis (YUV) use case. The configuration is listed in the LEGACY-level guaranteed configurations table.

YUV	MAXIMUM			In-application video/image processing.

For question 2:

This commit will be included in camera-core 1.1.0-alpha08 which will be release around end of August. As a workaround, you may be able to use the restricted ImageAnalysis.Builder#setMaxResolution(Size) to set MAXIMUM resolution to the ImageAnalysis before 1.1.0-alpha08 is released. After 1.1.0-alpha08 is release, please remove the workaround since the restricted API might be changed or removed in the future.

oa...@gmail.com <oa...@gmail.com> #11Jul 15, 2021 11:53AM

Thanks for the info, it sounds really good!
Great job guys!