IssueTracker

Hey all!

I have a Camera2 app where I built a custom VideoPipeline that effectively does this (pseudo code):

val usage = getHardwareBufferUsageFlagsIfSupported() // <-- USAGE_GPU_SAMPLED | CPU
val format = ImageFormat.YUV_420_888 // <-- sometimes I can also use ImageFormat.PRIVATE

// 1. Create an ImageReader with GPU flag if possible
val imageReader = ImageReader.newInstance(width, height, format, 3, usage)

// 2. Listen to Frames (could be up to 60 or even 120 FPS)
imageReader.setOnImageAvailableListener({ reader -> 
  val image = reader.acquireNextImage() ?: return@setOnImageAvailableListener

  // 4. Run the fast real-time frame processing (e.g. blazeface ML)
  frameProcessor.run(image)

  // 5. Render the image to the OpenGL pipeline. I have two implementations of this:
  //    a) One with ImageWriter.queueInputImage(..) (should be efficient)
  //    b) One where I use the HardwareBuffer as an EGLImageKHR input texture
  renderImageToOpenGL(image)
}, videoQueue)

// 3. Create the Camera2 CaptureSession with the ImageReader.surface as an output target.
//    The ImageReader.surface target is treated like a video/RECORD target.
createCamera(imageReader.surface)

And then in renderImageToOpenGL I effectively render a filter above the user's face, and finally render the whole image with filter to two output surfaces:

• Preview
• MediaRecorder (Encoder)

I wanted to rebuild this in CameraX, but haven't found a way to do so. From my understanding this would require me to either extend UseCase or VideoOutput, but I couldn't get either to work.

I thought that this might actually be quite a common use-case nowadays, since the ImageAnalysis has a few disadvantages:

• If I use Preview, ImageAnalysis and VideoCapture, the ImageAnalysis use-case runs in parallel, so the face filters would not be perfectly in sync for each frame.
• With ImageAnalysis I have seen resolution limits (e.g. not being able to stream 4k images) as well as FPS limits (e.g. not being able to stream at 60 FPS or even 90 FPS)
• With ImageAnalysis I was not able to use ImageFormat.PRIVATE.
• With ImageAnalysis, I would still need to create my own MediaRecorder/Encoder instead of being able to re-use the Recorder/Recording from CameraX.

So this is why I created this feature request - to have an additional way of processing frames in a synchronous pipeline which is treated like a realtime video stream.

I have thought of something like this:

val recorder = Recorder.Builder().also { recorder ->
  recorder.setQualitySelector(format.videoQualitySelector)
}.build()

val video = VideoCapture.Builder(recorder).also { video ->
  video.setMirrorMode(MirrorMode.MIRROR_MODE_ON_FRONT_ONLY)
  video.setVideoStabilizationEnabled(format.enableVideoStabilization)
  video.setTargetFrameRate(fpsRange)
  video.setResolutionSelector(format.videoResolutionSelector)
  if (videoConfig.config.enableHdr) {
    video.setDynamicRange(DynamicRange.HDR_UNSPECIFIED_10_BIT)
  }
}.build()

// NEW API:
video.setRealtimeFrameProcessor({ frame, output ->
  // this is called before each video frame is being forwarded to the output.
  //   - `frame` is something like a HardwareBuffer, Image, or a custom type. 
  //     in my case I don't always need CPU access, sometimes I can use GPU HardwareBuffers.
  //   - `output` is the `Recorder`, with a way to get access to it's surface, so
  //     I can render into it with OpenGL
})

In my experience, using HardwareBuffers instead of Images was a bit tricky to get working, but it was a much more flexible approach than ImageReader/ImageWriters as OpenGL could handle scaling automatically. Not sure which one is faster/more suitable for actual 60 FPS - 120 FPS realtime use though.

If this is not something you are interested in building, feel free to close this feature request.

If this is something that is already possible in CameraX, please let me know!

Thanks!