IssueTracker

yes the app used androidx.camera:camera-extensions:1.0.0-alpha24

Hi lixw1021,

Could you help to provide the following information?

1. What extension mode was set when the issue happened?
2. What is the issue occurrence rate?
3. Do you know any reproduce steps that might be easier to reproduce the issue?

Hi Trevor,

I didn't see this issue before. I have Samsung Note10+ and S20+ on my hand. I'll try to reproduce the issue.

What extension mode was set when the issue happened?

the only extension mode was enabled is HdrImageCaptureExtender

What is the issue occurrence rate?

There is no specific occurrence rate, but I think it less than 1%, because not every Samsung Galaxy S21/Note20 5G/ Note10+ crash.

Do you know any reproduce steps that might be easier to reproduce the issue?

Unfortunately, I don't have the specific device to test or reproduce the crash.

I have attached the code where we apply the HdrImageCaptureExtender, hope it helps.

                val cameraSelector = CameraSelector.Builder().requireLensFacing(lensFacing).build()

                // Set up the preview use case to display camera preview.
                val preview = Preview.Builder()
                    // We request aspect ratio but no resolution
                    .setTargetAspectRatio(screenAspectRatio)
                    // Set initial target rotation
                    .setTargetRotation(rotation)
                    .build()

                imageCapture = ImageCapture.Builder()
                    .setCaptureMode(ImageCapture.CAPTURE_MODE_MAXIMIZE_QUALITY)
                    .setTargetAspectRatio(screenAspectRatio)
                    .setTargetRotation(rotation)
                    .apply {
                        val hdrImageCapture = HdrImageCaptureExtender.create(this)
                        if (hdrImageCapture.isExtensionAvailable(cameraSelector)) {
                            hdrImageCapture.enableExtension(cameraSelector)
                        }
                    }.build()

Thanks lixw1021,

I can not reproduce the issue yet. But I have some questions about the code provided in comment#6.

1. Do you call the ExtensionsManager#init(Context) API to do the extensions initialization process? It should be the first call before making other calls to the extensions module.
2. Please refer to ImageCaptureExtender#enableExtension()'s javadoc. Please also enable the HDR extension mode for the Preview to make sure the extensions function can work well.

Image capture extension has dependence on preview extension. A PREVIEW_EXTENSION_REQUIRED error will be thrown if corresponding preview extension is not enabled together.

BTW, the original extension enabling approach (via *ImageCaptureExtender and *PreviewExtender) is not so good and it is easy to miss enabling the extension mode to Preview or ImageCapture. We have plan to improve the extensions APIs in the following releases. The *ImageCaptureExtender and *PreviewExtender classes and some other related APIs in ExtensionsManager class will be deprecated in the next release. A new mechanism will be added to make developers easier to enable the extension modes. Please refer to the following new APIs and they will be included in the next CameraX formal release which should be published soon.

• ExtensionsManager#getInstance(Context)
• ExtensionsManager#getExtensionEnabledCameraSelector(CameraProvider, CameraSelector, int)
• ExtensionsManager#isExtensionAvailable(CameraProvider, CameraSelector, int)

New APIs sample code: The HDR extension effect will be enabled for the bound Preview and ImageCapture when binding the use cases with an extension enabled CameraSelector. Applications don't need to enable the extension mode onto each use case separately by the Extender classes.

ProcessCameraProvider mCameraProvider = ...

// Calls the init function to retrieve a ListenableFuture object
ListenableFuture<ExtensionsManager> extensionsFuture =
                ExtensionsManager.getInstance(getApplicationContext());

// Adds callback function to wait for the initialization process complete
Futures.addCallback(extensionsFuture, new FutureCallback<ExtensionsManager>() {
    @Override
    public void onSuccess(@Nullable ExtensionsManager extensionsManager) {
        CameraSelector selector = CameraSelector.DEFAULT_BACK_CAMERA; 
                                    
        // Checks whether HDR is supported
        if (extensionsManager.isExtensionAvailable(mCameraProvider, selector, 
                    ExtensionMode.HDR)) {
            // Gets the HDR enabled camera selector if HDR is supported
            selector = extensionsManager.getExtensionCameraSelector(selector, 
                                                 ExtensionMode.HDR);

            // Binds use case with the HDR enabled camera selector
            mCameraProvider.bindToLifecycle(mLifecycleOwner, selector, useCases);
        }
    }

    @Override
    public void onFailure(@NonNull Throwable throwable) {
    }
}, ContextCompat.getMainExecutor(CameraExtensionsActivity.this));

Ah, the fact that this is happening with the HDR extender (which will increase image contrast/detail) and using mode CAPTURE_MODE_MAXIMIZE_QUALITY (which will decrease compression/increase file size) makes sense.

Charcoal, I would recommend attempting to reproduce by ensuring CAPTURE_MODE_MAXIMIZE_QUALITY is set with the HDR extension and try to take photos of images that are hard for JPEG to compress. Some inspiration may be found here.

It's possible the buffers we are allocating in the YuvToJpegProcessor's output ImageReader are not accounting for large JPEG image data plus all the required EXIF metadata. We may need to revisit that calculation.

Thanks Trevor.

The issue can be reproduced when taking pictures of this with HDR and CAPTURE_MODE_MAXIMIZE_QUALITY enabled. It is not 100%. You may need to adjust the angle, distance and the focused area to reproduce the issue.

I do some investigations.

The buffer size of the Image dequeued from the JPEG ImageWriter will be equal to the width multiplied by the height of its target surface. It seems there is no way to force enlarge the buffer size.

For the YuvImage#compressToJpeg() function, it will directly write the compressed result to the output stream. There should be no way to know the final compressed size in advance.

For the issue, the other idea is to decrease the compress quality level when the exception happens. I took some 4032x3024 pictures on S20+ device. The available buffer size will be 12192768. When the quality level setting is 100, the compressed result might be very close to 12192768. When the quality level setting is 98, the compressed result might be around 9500000. When the quality level setting is 95, the compressed result might be around 6800000. yuvImage.compressToJpeg might take around 100ms.

Do you think it makes sense if we try to decrease the compress quality level to 98 when the result size of quality 100 exceeds the available buffer size and re-compress the yuv buffer again to generate the output Jpeg Image object? Maybe we can try 99 then 98, but it might take more time to get the output result in some conditions. Or do you have other ideas/suggestions for this issue?

Thanks guys, amazing job! I think the quick solution is do not enable HDR CAPTURE_MODE_MAXIMIZE_QUALITY together, right? any other solution I can use to solve this crash?

Yes, it is likely for now that will work around the issue. MAXIMIZE_QUALITY uses a JPEG quality of 100 whereas without it the jpeg quality will be 95. There should be very little perceptual difference in quality, but the JPEG filesize should be significantly smaller, so this issue will be much less likely to reproduce.

Charcoal, I think our code for choosing the JPEG buffer size is a bit naive and could be improved. Here is where we set the buffer size to width * height: https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:camera/camera-core/src/main/java/androidx/camera/core/ProcessingImageReader.java;l=226;drc=1892e54378743a9c1ff62dca8680d89ac8b0d6a5

If you look at the camera service code, there is a much more involved process for choosing the buffer size. While the input dimensions do factor in to the calculation, there are other factors we need to consider. That code can be found here: https://cs.android.com/android/platform/superproject/+/master:frameworks/av/services/camera/libcameraservice/device3/Camera3Device.cpp;l=605;drc=master

We may also need to consider adjusting this if we plan on including any XMP metadata in the EXIF in the future. Though I'm not sure if that's in the plan for extensions or not.

Hi Trevor,

Thanks for providing the information.

In the Camera3Device#getJpegBufferSize() function, there are four factors to determine the buffer size.

• Target Jpeg resolution: YuvToJpegProcessor knows the information
• Max Jpeg resolution: Can be obtained from StreamConfigurationMap
• kMinJpegBufferSize: This should be a constant value (here) and we can also define it in CameraX.
• maxJpegBufferSize: Stored in CameraCharacteristics (ANDROID_JPEG_MAX_SIZE) but it seems a system protected information (here) and can’t be obtained in Java layer.

I also tried to search whether there is any other approach to estimate the JPEG compression buffer size. Someone mentioned that it shall not exceed width * height * 3 / 2 after compression (here). This looks like one line of his code in here. But I’m not sure how he got the estimation equation.

I’ll keep studying whether it is possible to know the ANDROID_JPEG_MAX_SIZE information in the Java layer.

Hi lixw1021,

https://android-review.googlesource.com/c/platform/frameworks/support/+/1759789 has been merged to fix the issue. The fix will be included in the camera-core module of next CameraX's release. Thanks.