Status Update
Comments
tr...@google.com <tr...@google.com> #2
[Deleted User] <[Deleted User]> #3
li...@gmail.com <li...@gmail.com> #4
yes the app used androidx.camera:camera-extensions:1.0.0-alpha24
ch...@google.com <ch...@google.com> #5
Hi lixw1021,
Could you help to provide the following information?
- What extension mode was set when the issue happened?
- What is the issue occurrence rate?
- Do you know any reproduce steps that might be easier to reproduce the issue?
Hi Trevor,
I didn't see this issue before. I have Samsung Note10+ and S20+ on my hand. I'll try to reproduce the issue.
li...@gmail.com <li...@gmail.com> #6
What extension mode was set when the issue happened?
the only extension mode was enabled is HdrImageCaptureExtender
What is the issue occurrence rate?
There is no specific occurrence rate, but I think it less than 1%, because not every Samsung Galaxy S21/Note20 5G/ Note10+ crash.
Do you know any reproduce steps that might be easier to reproduce the issue?
Unfortunately, I don't have the specific device to test or reproduce the crash.
I have attached the code where we apply the HdrImageCaptureExtender
, hope it helps.
val cameraSelector = CameraSelector.Builder().requireLensFacing(lensFacing).build()
// Set up the preview use case to display camera preview.
val preview = Preview.Builder()
// We request aspect ratio but no resolution
.setTargetAspectRatio(screenAspectRatio)
// Set initial target rotation
.setTargetRotation(rotation)
.build()
imageCapture = ImageCapture.Builder()
.setCaptureMode(ImageCapture.CAPTURE_MODE_MAXIMIZE_QUALITY)
.setTargetAspectRatio(screenAspectRatio)
.setTargetRotation(rotation)
.apply {
val hdrImageCapture = HdrImageCaptureExtender.create(this)
if (hdrImageCapture.isExtensionAvailable(cameraSelector)) {
hdrImageCapture.enableExtension(cameraSelector)
}
}.build()
ch...@google.com <ch...@google.com> #7
Thanks lixw1021,
I can not reproduce the issue yet. But I have some questions about the code provided in
- Do you call the
API to do the extensions initialization process? It should be the first call before making other calls to the extensions module.ExtensionsManager#init(Context) - Please refer to
's javadoc. Please also enable the HDR extension mode for the Preview to make sure the extensions function can work well.ImageCaptureExtender#enableExtension()
Image capture extension has dependence on preview extension. A PREVIEW_EXTENSION_REQUIRED error will be thrown if corresponding preview extension is not enabled together.
BTW, the original extension enabling approach (via *ImageCaptureExtender and *PreviewExtender) is not so good and it is easy to miss enabling the extension mode to Preview or ImageCapture. We have plan to improve the extensions APIs in the following releases. The *ImageCaptureExtender and *PreviewExtender classes and some other related APIs in ExtensionsManager class will be deprecated in the next release. A new mechanism will be added to make developers easier to enable the extension modes. Please refer to the following new APIs and they will be included in the next CameraX formal release which should be published soon.
ExtensionsManager#getInstance(Context) ExtensionsManager#getExtensionEnabledCameraSelector(CameraProvider, CameraSelector, int) ExtensionsManager#isExtensionAvailable(CameraProvider, CameraSelector, int)
New APIs sample code: The HDR extension effect will be enabled for the bound Preview and ImageCapture when binding the use cases with an extension enabled CameraSelector. Applications don't need to enable the extension mode onto each use case separately by the Extender classes.
ProcessCameraProvider mCameraProvider = ...
// Calls the init function to retrieve a ListenableFuture object
ListenableFuture<ExtensionsManager> extensionsFuture =
ExtensionsManager.getInstance(getApplicationContext());
// Adds callback function to wait for the initialization process complete
Futures.addCallback(extensionsFuture, new FutureCallback<ExtensionsManager>() {
@Override
public void onSuccess(@Nullable ExtensionsManager extensionsManager) {
CameraSelector selector = CameraSelector.DEFAULT_BACK_CAMERA;
// Checks whether HDR is supported
if (extensionsManager.isExtensionAvailable(mCameraProvider, selector,
ExtensionMode.HDR)) {
// Gets the HDR enabled camera selector if HDR is supported
selector = extensionsManager.getExtensionCameraSelector(selector,
ExtensionMode.HDR);
// Binds use case with the HDR enabled camera selector
mCameraProvider.bindToLifecycle(mLifecycleOwner, selector, useCases);
}
}
@Override
public void onFailure(@NonNull Throwable throwable) {
}
}, ContextCompat.getMainExecutor(CameraExtensionsActivity.this));
tr...@google.com <tr...@google.com> #8
Ah, the fact that this is happening with the HDR extender (which will increase image contrast/detail) and using mode CAPTURE_MODE_MAXIMIZE_QUALITY
(which will decrease compression/increase file size) makes sense.
Charcoal, I would recommend attempting to reproduce by ensuring CAPTURE_MODE_MAXIMIZE_QUALITY
is set with the HDR extension and try to take photos of images that are hard for JPEG to compress. Some inspiration may be found
It's possible the buffers we are allocating in the YuvToJpegProcessor's output ImageReader are not accounting for large JPEG image data plus all the required EXIF metadata. We may need to revisit that calculation.
ch...@google.com <ch...@google.com> #9
Thanks Trevor.
The issue can be reproduced when taking pictures of HDR
and CAPTURE_MODE_MAXIMIZE_QUALITY
enabled. It is not 100%. You may need to adjust the angle, distance and the focused area to reproduce the issue.
I do some investigations.
The buffer size of the Image dequeued from the JPEG ImageWriter will be equal to the width multiplied by the height of its target surface. It seems there is no way to force enlarge the buffer size.
For the YuvImage#compressToJpeg() function, it will directly write the compressed result to the output stream. There should be no way to know the final compressed size in advance.
For the issue, the other idea is to decrease the compress quality level when the exception happens. I took some 4032x3024 pictures on S20+ device. The available buffer size will be 12192768. When the quality level setting is 100, the compressed result might be very close to 12192768. When the quality level setting is 98, the compressed result might be around 9500000. When the quality level setting is 95, the compressed result might be around 6800000. yuvImage.compressToJpeg might take around 100ms.
Do you think it makes sense if we try to decrease the compress quality level to 98 when the result size of quality 100 exceeds the available buffer size and re-compress the yuv buffer again to generate the output Jpeg Image object? Maybe we can try 99 then 98, but it might take more time to get the output result in some conditions. Or do you have other ideas/suggestions for this issue?
li...@gmail.com <li...@gmail.com> #10
Thanks guys, amazing job!
I think the quick solution is do not enable HDR
CAPTURE_MODE_MAXIMIZE_QUALITY
together, right?
any other solution I can use to solve this crash?
tr...@google.com <tr...@google.com> #11
Yes, it is likely for now that will work around the issue. MAXIMIZE_QUALITY uses a JPEG quality of 100 whereas without it the jpeg quality will be 95. There should be very little perceptual difference in quality, but the JPEG filesize should be significantly smaller, so this issue will be much less likely to reproduce.
Charcoal, I think our code for choosing the JPEG buffer size is a bit naive and could be improved. Here is where we set the buffer size to width * height:
If you look at the camera service code, there is a much more involved process for choosing the buffer size. While the input dimensions do factor in to the calculation, there are other factors we need to consider. That code can be found here:
We may also need to consider adjusting this if we plan on including any XMP metadata in the EXIF in the future. Though I'm not sure if that's in the plan for extensions or not.
ch...@google.com <ch...@google.com> #12
Hi Trevor,
Thanks for providing the information.
In the
- Target Jpeg resolution: YuvToJpegProcessor knows the information
- Max Jpeg resolution: Can be obtained from StreamConfigurationMap
- kMinJpegBufferSize: This should be a constant value (
) and we can also define it in CameraX.here - maxJpegBufferSize: Stored in CameraCharacteristics (
ANDROID_JPEG_MAX_SIZE
) but it seems a system protected information ( ) and can’t be obtained in Java layer.here
I also tried to search whether there is any other approach to estimate the JPEG compression buffer size. Someone mentioned that it shall not exceed width * height * 3 / 2
after compression (
I’ll keep studying whether it is possible to know the ANDROID_JPEG_MAX_SIZE information in the Java layer.
ap...@google.com <ap...@google.com> #13
Branch: androidx-main
commit ebe604adc1fce7d57b16c7bfd7b35fc17ebad922
Author: Charcoal Chen <charcoalchen@google.com>
Date: Thu Jul 08 18:46:15 2021
Fix YuvToJpegProcessor EOFException issue.
The issue should only happens when extension mode is enabled and ImageCapture#CAPTURE_MODE_MAX_QUALITY mode is set. The JPEG compression quality setting will be 100. The compressed byte buffer data might exceed the image's width * height. The solution is to enlarge the byte buffer to 1.5 times of width * height plus extra exif metadata maximum size. 1.5 times of width * height is the size of YUV_420_888 byte buffer. The compressed JPEG data shouldn't exceed it.
The ProcessingImageReader constructors have accepted too many parameters. Change to use the bulder pattern to build a ProcessingImageReader object.
Relnote: "Fixed YuvToJpegProcessor EOFException issue when extension mode is enabled and ImageCapture#CAPTURE_MODE_MAX_QUALITY mode is set."
Bug: 192017012
Test: ./gradlew bOS && manual test on real device
Change-Id: I538bdbc1e460cb5ad3d98cf87017127d7e68f131
M camera/camera-core/src/androidTest/java/androidx/camera/core/ProcessingImageReaderDeviceTest.kt
M camera/camera-core/src/main/java/androidx/camera/core/ImageCapture.java
M camera/camera-core/src/main/java/androidx/camera/core/ProcessingImageReader.java
M camera/camera-core/src/main/java/androidx/camera/core/internal/YuvToJpegProcessor.java
M camera/camera-core/src/test/java/androidx/camera/core/ProcessingImageReaderTest.java
ch...@google.com <ch...@google.com> #14
Hi lixw1021,
Description
CAMERAX VERSION (1.0.0)
ANDROID OS BUILD NUMBER: Android 11
DEVICE NAME: Samsung Galaxy S21, Samsung Galaxy Note20 5G, Samsung Galaxy Note10+
DESCRIPTION:
log:
ADDITIONAL INFORMATION:
Looks like it only happens on Samsung device with android 11
CODE FRAGMENTS (this will help us troubleshoot your issues):