IssueTracker

Hi Ed, Thank you so much for these suggestions. I've been reviewing them and merging them in. Hopefully it should be live. I've included a thank you note too in the article.

To clarify,  so the sharpness/quality is not an issue .  right ?

But you have another issue, which happens on Xioami mi A2 where the image is weird .
by the way,  how do you exactly do the YuvToRgb conversion on Xiaomi mi A2.   I saw you mentioned YUV to NV21 , then NV21 to JPG in the stackoverflow, if this is the way you did the conversion,  have you tried the YuvToRgbConverter ( https://github.com/android/camera-samples/blob/3730442b49189f76a1083a98f3acf3f5f09222a3/CameraUtils/lib/src/main/java/com/example/android/camera/utils/YuvToRgbConverter.kt )  ?

yes, quality is ok. Yes I use the one from stackoverflow. I looked at your sample, but it was annotated that it is just a sample and has not been tested on many devices, so I feel like this is not production tested code. If I am wrong, please correct me. Also, I was unsure how should you call that method. I see that you need to pass output: Bitmap as parameter, but what does that mean? Should I create a default bitmap and then transform it with code?
I am attaching my solution with edited bitmap. However, I don't get correct rotation and don't know how should I correctly rotate bitmap with this method.

Actually YuvToRgbConverter is almost identical to your CustomBitmapConverter except that YuvToRgbConverter requires user to create a bitmap for output and pass it to the converter.  

And YuvToRgbConverter is our official recommended way to convert a YUV image to a RGB image.  We've tested a lot but certainly cannot guaranteed 100% correctness on all devices.  If YuvToRgbConverter did have problem on some devices, please let us know we will fix it soon.    

To confirm, the Xioami mi A2 issue does happen  using the YuvToRgbConverter method ,  correct ?  

No, issue in the Xioami mi A2 occurred only with stackoverflow code. It worked well with CustomBitmapConverter, I just was not sure if it is official way of getting bitmap. Thanks for clarification. Yes, I know that CustomBitmapConverter is same as YuvToRgbConverter, I only changed bitmap created part and moved it inside of method. Thanks for clarifying that it is the same, so I can now revert to YuvToRgbConverter and just pass output as function parameter.

Now I have only one remaning question. Should I use YuvToRgbConverter or rather getting bitmap with previewView.getBitmap()?

Currently our codebase uses  previewView.getBitmap() and we have not experienced any issues. The only issue was that initial frames look darker, it could be because of Auto-Exposure, because I call previewView.getBitmap() inside of Analyzer callback. Is that correct or should I rather wait for camera to load? How much should I wait or perhaps there is some callback, which would indicate if camera is already loaded?

previewView.getBitmap is for different purpose and is much slower than YuvToRgbConverter.
So if the purpose is to convert the YUV image in Analyzer to RGB bitmap,  you should definitely use YuvToRgbConverter.
And previewView.getBitmap is for saving a snapshot of current preview for later use.  It is not designed to be fetched continuously.

As for the auto-exposure issue,   Analyzer is supposed to analyze the image continuously and when analyzing continuously,  the exposure will becomes normal for a while so it is not a problem.    

Are you looking for one-time capture using ImageAnalysis ?   What is your exact use case ?  

Thanks for clarification again! We actually designed our camera in a way that user presses button and then we wait for the ImageAnalysis to emit a latest ImageProxy and then we call preview.getBitmap() to get snapshot. As far as I understood, we should better use YuvToRgbConverter for getting bitmap of current camera preview.

We have decided to do it, because taking image with imageCapture seems very slow on so called "budget" devices, like Xiaomi mi A2 ( it really lags and takes ±1 sec).
So by capturing current camera preview we managed to get picture fast.

The code we used to actually capture image, without taking camera preview frame was from your sample also.

Just to be clear regarding lagging. For instance onImageSaved callback is called ±800ms after actually calling imageCapture.takePicture. I have tried on memory photo taking, but run into other exceptions, like camera was already closed.

Since the capture.is initiated by user clicking,   I think the user probably will wait until the preview becomes brighter to take a picture.
if he/she doesn't wait,  then it is expected that the captured image could be dark because this is the same as the preview.

As for the lag image capture issue in Xiaomi Mi A2,  we will take a look and see if we can improve the ImageCapture latency.

Yes, you are correct :) I have over-tested perhaps, without taking into account real world scenario.
I really appreciate that you will take a look at lag issue with image capture. Perhaps speed could be improved. I think that drop in 200-300 ms would make an actual difference to the user experience.

Just to clarify one more time, before closing this issue. By using preview.getBitmap() for user initiated action would be correct in our case and would result in same result as convert ImageProxy to Bitmap with  YuvToRgbConverter if our final goal is to get bitmap as quick as possible.

I am hesitant to use YuvToRgbConverter, because of lack of testing on our side as well and code complexity in YuvToRgbConverter. On the other hand, we have not had any issue with androidx.camera.view.PreviewView.getBitmap()

Another drawback of preview.getBitmap() is the resolution is just the size of the PreviewView.   So if the device has smaller resolution,   then the user will get a low resolution photo.  
(But ImageAnalysis has the same problem as the default resolution of ImageAnalysis is 640x480 unless you set the higher resolution )

so if the low resolution doesn't bother you , then I think it should fit your need.      

and by the way,  you mentioned that  "I have tried on memory photo taking, but run into other exceptions, like camera was already closed."  
Does the crash happen always ?  
Does the crash look like the following ?
    java.lang.IllegalStateException: maxImages (2) has already been acquired, call #close before acquiring more.
        at android.media.ImageReader.acquireNextImage(ImageReader.java:513)
        at androidx.camera.core.AndroidImageReaderProxy.acquireNextImage(AndroidImageReaderProxy.java:79)
        at androidx.camera.core.MetadataImageReader.imageIncoming(MetadataImageReader.java:318)
        at androidx.camera.core.MetadataImageReader$2.onImageAvailable(MetadataImageReader.java:67)

I will go back later tomorrow regarding issue with memory photo taking. Today we have transitioned to YuvToRgbConverter and have tested on several devices. However, we found that quality is indeed worse than preview.getBitmap() on device like Xioami mi A2. The issue was specifically with letters ( I took image of passport). Letters in image taken with YuvToRgbConverter had small empty spaces which greatly reduced overall readability.

One of the colleagues tested on Huawei p20 lite. He also received same issue, but then he experimented with  ImageAnalysis. Previously he used .setTargetAspectRatio(screenAspectRatio) and colleague faced same issue. Then he switched to   .setTargetResolution(resolution) and provided resolution with following code:

val metrics = DisplayMetrics().also { viewFinder!!.display.getMetrics(it) }
val resolution = Size(metrics.widthPixels, metrics.heightPixels)

This indeed improved outputted bitmap and issues with quality were solved. This however did not solved photo quality issue with my Xioami phone. We have tried as much as we could and this is all we got. It would be grateful if you could double check bitmap quality after applying conversion.

Sincerely, Viktor

Can you log the width/height of the imageProxy in Analyzer on Xiaomi mi A2 (after setting TargetResolution) ?

I think the problem is the resolution selected for the ImageAnalysis is too small.   Resolution is determined by multiple factors including supported resolution on devices.   so the resulted resolution could vary across different devices.

Ok, I did and it seems this is a reason, as you said.
Values of images are:
2020-06-30 15:56:10.979 7436-7669/com.idenfy.app.new D/CameraLog: imageProxyImmutableImageInfo{tag=null, timestamp=288133495824503, rotationDegrees=90}width:864 height: 480

2020-06-30 15:56:10.979 7436-7669/com.idenfy.app.new D/CameraLog: bitmap from preview.getBitmap() width:1080 height:1944

Resolution seems very small and this is the case. The width is 864(Image is rotated, so actual width is 480). While preview.getBitmap() returns much better resolution on this device.

If I set .setTargetAspectRatio(screenAspectRatio) instead of .setTargetResolution(resolution) (which is 1080x2016 on Xiaomi mi A2) I still get same value from analyzer.

I think that Xiaomi mi A2 is unique device, or perhaps camerax fails to get better resolution, if it is even possible for ImageAnalysis.

Hello, any news? :)

Hi Viktor,
To be clear,  
you set setTargetAspectRatio to RATIO_16_9 or or 4_3  on Xiaomi mi A2 ,  and still get the ImageAnalysis image size = 864 x480  ?  

Yes, exactly :)

Thanks for the info.
+ Charcoal,
This looks like a bug in CameraX.  Can you help clarify it ?

Yes, I can assist in any way you need. How can I help? What information you
need?

On Mon, Jul 6, 2020, 18:05 <buganizer-system@google.com> wrote:

--
UAB "Identifikaciniai Projektai", iDenfy™ is compliant with the General
Data Protection Regulation. To learn more about how we collect, keep, and
process your private information in compliance with GDPR, please view our
privacy policy <https://www.idenfy.com/privacy-policy/>. This policy was
last updated on 2019-10-11.


Information contained in this email and any
attachments may be privileged or confidential and intended for the
exclusive use of the original recipient. If you have received this email by
mistake, please advise the sender immediately and delete the email,
including emptying your deleted email box.

Hi Viktor,

Could you help to provide the supported output sizes of YUV_420_888 format on the Xiaomi mi A2 device?
The info can be obtained by StreamConfigurationMap#getOutputSizes.

Ok, so this is how I did it on Pixel XL 3 and also Xiaomi mi A2.

camera = cameraProvider.bindToLifecycle(
                            this, cameraSelector, preview, imageAnalyzer, imageCapture)
                    val cameraId = Camera2CameraInfo.extractCameraId(camera!!.cameraInfo)
                    val cameraManager = context!!.getSystemService(Context.CAMERA_SERVICE) as CameraManager
                    val characteristics = cameraManager.getCameraCharacteristics(cameraId)
                    val streamConfigurationMap = characteristics.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP)
                    val outputSizes = streamConfigurationMap.getOutputSizes(ImageFormat.YUV_420_888)
                    for(size in outputSizes){
                        Log.d("CameraLog", "outPut size: ${size.toString()}")
                    }

By the way, this is how I set aspectRatio, which I use inImageAnalysis.Builder()  perhaps there is issue there( I doubt it):

 val metrics = DisplayMetrics().also { viewFinder!!.display.getMetrics(it) }
        val screenAspectRatio = getCameraAspectRatio(metrics.widthPixels, metrics.heightPixels)

 fun getCameraAspectRatio(width: Int, height: Int): Int{
        val previewRatio = Math.max(width, height).toDouble() / Math.min(width, height)
        if (Math.abs(previewRatio - 4.0 / 3.0) <= kotlin.math.abs(previewRatio - 16.0 / 9.0)) {
            return AspectRatio.RATIO_4_3
        }
        return AspectRatio.RATIO_16_9
    }


Now regarding output sizes results on Pixel XL3:

2020-07-07 11:29:57.498 12722-12722/com.idenfy.app.new D/CameraLog: outPut size: 1280x720
2020-07-07 11:29:57.498 12722-12722/com.idenfy.app.new D/CameraLog: outPut size: 640x480
2020-07-07 11:29:57.498 12722-12722/com.idenfy.app.new D/CameraLog: outPut size: 320x240
2020-07-07 11:29:57.499 12722-12722/com.idenfy.app.new D/CameraLog: outPut size: 176x144
2020-07-07 11:29:57.499 12722-12722/com.idenfy.app.new D/CameraLog: outPut size: 1920x1080
2020-07-07 11:30:05.669 12722-13019/com.idenfy.app.new D/CameraLog: imageProxy info: ImmutableImageInfo{tag=null, timestamp=17810298350440, rotationDegrees=90} width:1280 height:720

Now same code on Xiaomi mi A2:

2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 2048x1536
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 1920x1440
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 2048x1080
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 1920x1080
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 2560x800
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 1440x1080
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 1280x960
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 1440x720
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 1280x720
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 1280x480
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 1280x400
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 864x480
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 640x640
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 800x480
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 720x480
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 768x432
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 640x480
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 480x640
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 576x432
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 640x360
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 480x360
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 480x320
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 384x288
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 352x288
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 320x240
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 240x320
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 240x160
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 176x144
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 144x176
2020-07-07 11:35:27.512 31102-31102/com.idenfy.app.new D/CameraLog: outPut size: 160x120
2020-07-07 11:35:38.932 31102-31707/com.idenfy.app.new D/CameraLog: imageProxy info: ImmutableImageInfo{tag=null, timestamp=877300072296933, rotationDegrees=90} width:864 height:480

As you can see device has better resolution. For instance we currently use https://github.com/RedApparat/Fotoapparat camera library and it produces much better images using frames.

Thanks for providing the info. I can reproduce the issue by robolectric test. I still need more time to clarify and fix the issue.

If your major requirement is to obtain a better quality result by converting ImageAnalysis's outputs, I'll suggest to directly set the target resolution to the maximal size that ImageAnalysis can support. The maximal supported size is 1920x1080. But please be noticed that the target resolution setting needs to match the target rotation. It means that it should be set as 1080x1920 when a phone device is in its natural portrait (ROTATION_0) orientation and should be set as 1920x1080 when the phone device is in landscape orientation.

It is great that you can reproduce issue!
Regarding resolution, we only use camera on portrait mode, so it is even
user for us.

Now, not all phones have 1080x1920. Will it be a problem if I set the
highest resolution, but device does not support it?  Will the camerax
fallback to other resolution automatically?

If not, perhaps I can update cameraX resolution at runtime, because I do
get output sizes after binding camera to lifecycle. However, I am not sure
how can I do the following in code.






On Wed, Jul 8, 2020, 05:46 <buganizer-system@google.com> wrote:

--
UAB "Identifikaciniai Projektai", iDenfy™ is compliant with the General
Data Protection Regulation. To learn more about how we collect, keep, and
process your private information in compliance with GDPR, please view our
privacy policy <https://www.idenfy.com/privacy-policy/>. This policy was
last updated on 2019-10-11.


Information contained in this email and any
attachments may be privileged or confidential and intended for the
exclusive use of the original recipient. If you have received this email by
mistake, please advise the sender immediately and delete the email,
including emptying your deleted email box.

When setting the target resolution as 1080x1920, if the size is not supported for the device, the next smaller size with the same aspect ratio will be selected to use. If the device does not support 16:9 sizes, the other sizes with aspect ratio that is nearest to the target resolution will still be selected to use. Therefore, I think you can adopt the 1080x1920 target resolution setting approach.

The difference between setting target aspect ratio as ApsectRatio.RATIO_16_9 and setting target resolution as 1080x1920 for ImageAnalysis is:
 -> Setting target aspect ratio as ApsectRatio.RATIO_16_9 for ImageAnalysis will select the smallest 16:9 size that can box 640x480.
 -> Setting target resolution as 1080x1920 for ImageAnalysis will select the biggest 16:9 available size that is under 1080x1920.

The difference is caused by CameraX has default 640x480 size setting for ImageAnalysis. For most analysis requirement, 640x480 size has been big enough. Analyzing larger image costs time. Therefore, developers need to set the target resolution explicitly to obtain larger ImageAnalysis output.

Hello, I tested with resolution. Perhaps you made a mistake with size and
swapped values? I tested with : *Size(width:1920, height:1080) *as
resolution and it did not work. As soon as I switched to *Size(width:1080,
height:1920)* for ROTATION_0 iamgeAnalyzer returned imageProxy with high
resolution even on Xioami mi A2.

On Wed, Jul 8, 2020 at 9:57 AM <buganizer-system@google.com> wrote:

--
UAB "Identifikaciniai Projektai", iDenfy™ is compliant with the General
Data Protection Regulation. To learn more about how we collect, keep, and
process your private information in compliance with GDPR, please view our
privacy policy <https://www.idenfy.com/privacy-policy/>. This policy was
last updated on 2019-10-11.


Information contained in this email and any
attachments may be privileged or confidential and intended for the
exclusive use of the original recipient. If you have received this email by
mistake, please advise the sender immediately and delete the email,
including emptying your deleted email box.

Please refer to https://developer.android.com/reference/androidx/camera/core/ImageAnalysis.Builder#setTargetResolution(android.util.Size).

=> The resolution Size should be expressed in the coordinate frame after rotating the supported sizes by the target rotation. For example, a device with portrait natural orientation in natural target rotation requesting a portrait image may specify 480x640, and the same device, rotated 90 degrees and targeting landscape orientation may specify 640x480.

Therefore, the target resolution setting should correspond to the target rotation setting. Just like what I mentioned in comment#25.
=> it should be set as 1080x1920 when a phone device is in its natural portrait (ROTATION_0) orientation and should be set as 1920x1080 when the phone device is in landscape orientation.

Hello, perhaps there are some updates regarding this issue? Or should we rather use (width:1920, height:1080) for our case, because it does work correctly? :)
Sincerely, Viktor

Hi Viktor,

After clarifying, the root causes of why 864x480 is selected are listed as the following.

Case 1: When setting target aspect ratio as ApsectRatio.RATIO_16_9 for ImageAnalysis
Reason: For ImageAnalysis, a size closest to 640x480 with the target aspect ratio 16:9 will be selected. 864x480 is a mod16 alignment size that matches 16:9 aspect ratio. Therefore, 864x480 is selected when setting target aspect ratio as ApsectRatio.RATIO_16_9 for ImageAnalysis.

Case 2: When setting target resolution as 1080x2016 as mentioned in comment#17
Reason: 864x480 is considered to be a mod16 alignment size to match the aspect ratio of 2016x1080. But other 16:9 sizes do not match the aspect ratio of 2016x1080. CameraX will selected a size matches or its aspect ratio is nearest to the aspect ratio of the target resolution setting. Therefore,  864x480 is selected when setting target resolution as 1080x2016.

For the case 2, I think it should be a bug. The 864x480 size should be grouped into 16:9 aspect ratio sizes group but not to be a size matching the aspect ratio of 2016x1080. I'm working to have a solution for the case 2, then, you will be able to directly set the target resolution as the display's size after the solution is merged.

Project: platform/frameworks/support
Branch: androidx-master-dev

commit fd15f09dce5fb49a871e544ddbc812271883af81
Author: charcoalchen <charcoalchen@google.com>
Date:   Mon Jul 13 13:55:04 2020

    Fixed SupportedSurfaceCombination aspect ratio size grouping issue
   
    Group all available sizes by aspect ratio first and then sort the groups by the distance to the target aspect ratio to find the best suitable size.
   
    The developers may set the target resolution as any arbitrary size. Original logic directly used the aspect ratio of the target resolution setting to find the supported sizes that match the aspect ratio under mod16 mechanism. It made some supported sizes that should belong to other aspect ratio were incorrectly grouped under mod16 mechanism, and then considered to match the aspect ratio of the target resolution setting. For example, 864x480 should be a 16:9 size, but it was grouped to match the aspect ratio of 2016x1080 target resolution setting. To fix this issue, the all supported sizes will be grouped first. It can avoid a size to be grouped to match a strange aspect ratio. Then, the auto-resolution mechanism will sort all sizes by the aspect ratio difference distance of the groups to select the suitable supported size for the target resolution setting.
   
    Relnote:"Fixed auto-resolution aspect ratio size grouping issue that a 16:9 mod16 size (864x480) is selected when the target resolution setting is 2016x1080 and there is another 1920x1080 16:9 size supported."
   
    Bug: 159363774
    Test: run SupportedSurfaceCombinationTest
    Change-Id: I531671972a05bf01e2dea1f96da5bc3d6655c391

M       camera/camera-camera2/src/main/java/androidx/camera/camera2/internal/SupportedSurfaceCombination.java
M       camera/camera-camera2/src/test/java/androidx/camera/camera2/internal/SupportedSurfaceCombinationTest.java

https://android-review.googlesource.com/1360358

A solution has been merged to make 1920x1080 can be selected when the target resolution is set as 1080x2016. Please wait for CameraX's next release.

Amazing, thank you for your work!