IssueTracker

CAMERAX VERSION 1.4.0-alpha04

CAMERA APPLICATION NAME AND VERSION: 9.1.0.338

ANDROID OS BUILD NUMBER: 29

DEVICE NAME: Huawei P10 (VTR-L29)

DESCRIPTION:

I have my own custom VideoPipeline which first analyzes frames using CPU access (ImageReader) and then passes them on to a OpenGL pipeline to draw stuff onto the Frame, and finally renders the modified frame to an output surface (MediaRecorder).

This is my code:

class VideoPipeline(
  private val format = ImageFormat.YUV_420_888, // or ImageFormat.PRIVATE
  private val callback: CameraSession.Callback
) : VideoOutput,
  Closeable {
  companion object {
    private const val MAX_IMAGES = 3
    private const val TAG = "VideoPipeline"
  }

  // Output
  // TODO: `Recording`/`Recorder` output from CameraX?

  init {
    Log.i(TAG, "Initializing VideoPipeline with format $format")
  }

  @SuppressLint("RestrictedApi")
  override fun getMediaSpec(): Observable<MediaSpec> {
    val mediaSpec = MediaSpec.builder().setOutputFormat(MediaSpec.OUTPUT_FORMAT_MPEG_4).configureVideo { video ->
      // TODO: Instead of hardcoding that, can I dynamically get those values from the Camera?
      video.setFrameRate(Range(0, 60))
      video.setQualitySelector(QualitySelector.from(Quality.HD))
    }
    return ConstantObservable.withValue(mediaSpec.build())
  }

  @SuppressLint("RestrictedApi")
  override fun onSurfaceRequested(request: SurfaceRequest) {
    val size = request.resolution
    val surfaceSpec = request.deferrableSurface
    Log.i(TAG, "Creating $size Surface... (${request.expectedFrameRate.upper} FPS, expected format: ${surfaceSpec.prescribedStreamFormat}, ${request.dynamicRange})")

    Log.i(TAG, "Using ImageReader round-trip (format: #$format)")

    // Create ImageReader
    val imageReader = ImageReader.newInstance(size.width, size.height, format, MAX_IMAGES)

    imageReader.setOnImageAvailableListener({ reader ->
      Log.i(TAG, "ImageReader::onImageAvailable!")

      try {
        val image = reader.acquireNextImage() ?: return@setOnImageAvailableListener

        // TODO: Get correct orientation and isMirrored
        val frame = Frame(image, image.timestamp, Orientation.PORTRAIT, isMirrored)
        frame.incrementRefCount()

        try {
          callback.onFrame(frame)

          if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) {
            // This just renders the HardwareBuffer in the OpenGL pipeline.
            // I _could_ use ImageWriter there, but ImageWriter often crashed and was buggy.
            val hardwareBuffer = image.hardwareBuffer ?: throw HardwareBuffersNotAvailableError()
            renderHardwareBuffer(hardwareBuffer)
          }
        } finally {
          frame.decrementRefCount()
        }
      } catch (e: Throwable) {
        Log.e(TAG, "FrameProcessor/ImageReader pipeline threw an error!", e)
        callback.onError(e)
      }
    }, CameraQueues.videoQueue.handler)

    request.provideSurface(imageReader.surface, CameraQueues.videoQueue.executor) { result ->
      synchronized(this) {
        imageReader.close()
      }

      onSurfaceClosed(result.resultCode)
    }
  }
}

It will successfully create the Surfaces, but once the first Frame arrives in the ImageReader, calling acquireNextImage() will immediately crash with the following error:

java.lang.UnsupportedOperationException: The producer output buffer format 0x1 doesn't match the ImageReader's configured buffer format 0x23.
  at android.media.ImageReader.nativeImageSetup(Native Method)
  at android.media.ImageReader.acquireNextSurfaceImage(ImageReader.java:439)
  at android.media.ImageReader.acquireNextImage(ImageReader.java:493)
  at com.mrousavy.camera.core.VideoPipeline.onSurfaceRequested$lambda$1(VideoPipeline.kt:99)
  at com.mrousavy.camera.core.VideoPipeline.$r8$lambda$5heGRS9RL_Z-x-BiU-ziCBMS68U(Unknown Source:0)
  at com.mrousavy.camera.core.VideoPipeline$$ExternalSyntheticLambda1.onImageAvailable(Unknown Source:2)
  at android.media.ImageReader$ListenerHandler.handleMessage(ImageReader.java:800)
  at android.os.Handler.dispatchMessage(Handler.java:112)
  at android.os.Looper.loop(Looper.java:216)
  at android.os.HandlerThread.run(HandlerThread.java:65)

I think there are multiple questions I have in this bug report:

1. Why is the output producer (the Camera) configured to format 0x1 (PixelFormat.RGBA_8888) when my ImageReader is configured with format 0x23 (ImageFormat.YUV_420_888)?
2. Is extending VideoOutput the right approach here so I can then use this as a VideoCapture use-case for the Camera? Or should I extend UseCase?
3. Is there any way I could use a Recorder/Recording instance myself to render into it's Surface with the OpenGL pipeline? I want to avoid rebuilding the entire MediaMuxer part.

LIST ANY EXPERIMENTAL FEATURES:

STEPS TO REPRODUCE:

1. Create custom video pipeline that extends VideoOutput
2. In onSurfaceRequested, create an ImageReader with YUV format and return it's surface

OBSERVED RESULTS: When the ImageReader's frame available callback is called, acquiring an Image will immediately crash because the Surface formats don't match

EXPECTED RESULTS: The Camera should be configured to the Surface format of the ImageReader (here, YUV_420_888 or PRIVATE) to avoid that crash

REPRODUCIBILITY: 100