Fixed
Status Update
No update yet.
Comments
we...@google.com
Hi Ed, Thank you so much for these suggestions. I've been reviewing them and merging them in. Hopefully it should be live. I've included a thank you note too in the article.
we...@google.com
Great! Thanks a lot, I'll look for the live updates soon!
ma...@gmail.com
It is actually fairly noticeable if you record one of the "audio sync test 60fps" type videos on YouTube with a device. The video on my pixel 1 is about 120ms behind the audio.
Looking at the VideoCapture code I strongly suspect that this is due to the presentation time of the video frames being stamped and provided to the muxer as the frames come out of the codec (rather than going in to the codec which is when they are actually in sync with the audio). Your video frames will always be behind and out of sync by however long the codec takes to chew on each frame.
I'm pretty sure that at least since Android 8 the presentation times are properly provided to the surface by the camera (also from the high precision clock) and may not need to be overwritten later with one that is delayed. That may be helpful.
Good luck!
Looking at the VideoCapture code I strongly suspect that this is due to the presentation time of the video frames being stamped and provided to the muxer as the frames come out of the codec (rather than going in to the codec which is when they are actually in sync with the audio). Your video frames will always be behind and out of sync by however long the codec takes to chew on each frame.
I'm pretty sure that at least since Android 8 the presentation times are properly provided to the surface by the camera (also from the high precision clock) and may not need to be overwritten later with one that is delayed. That may be helpful.
Good luck!
tr...@google.com
Thanks for the tips. I agree this is a good place to start looking at. We will take a look.
le...@google.com
le...@google.com
le...@google.com
The attached video in MediaCodec#setParameters with MediaCodec.PARAMETER_KEY_SUSPEND. It will affect not only the current recording video using pause resume function but also the beginning of the next recording.
ap...@google.com
Project: platform/frameworks/support
Branch: androidx-main
commit 75cbc0b8a928dfc6bf3d75da027ed396ba684c87
Author: leo huang <leohuang@google.com>
Date: Wed Mar 23 18:45:00 2022
Fix audio/video out of sync after pause and resume on some Samsung devices pre-API 29
After calling MediaCodec.setParameters() with MediaCodec.PARAMETER_KEY_SUSPEND, the video codec of the issue device will send a timestamp missing the pause interval after resumed, which results in the video and audio to be out of sync after resumed.
The workaround is to not call MediaCodec.setParameters() when recording is paused. This shouldn't cause too much overhead since we'll still stop the camera from sending frames to the encoder surface when paused.
Relnote: "Fixed audio/video out of sync after pause and resume on some Samsung devices pre-API 29."
Bug:
Test: Manual testing
Change-Id: I646221b9f94ae688592d4be96245f17f74806f0b
A camera/camera-video/src/main/java/androidx/camera/video/internal/compat/quirk/VideoEncoderSuspendDoesNotIncludeSuspendTimeQuirk.java
M camera/camera-video/src/main/java/androidx/camera/video/internal/compat/quirk/DeviceQuirksLoader.java
M camera/camera-video/src/main/java/androidx/camera/video/internal/encoder/EncoderImpl.java
https://android-review.googlesource.com/2038784
Branch: androidx-main
commit 75cbc0b8a928dfc6bf3d75da027ed396ba684c87
Author: leo huang <leohuang@google.com>
Date: Wed Mar 23 18:45:00 2022
Fix audio/video out of sync after pause and resume on some Samsung devices pre-API 29
After calling MediaCodec.setParameters() with MediaCodec.PARAMETER_KEY_SUSPEND, the video codec of the issue device will send a timestamp missing the pause interval after resumed, which results in the video and audio to be out of sync after resumed.
The workaround is to not call MediaCodec.setParameters() when recording is paused. This shouldn't cause too much overhead since we'll still stop the camera from sending frames to the encoder surface when paused.
Relnote: "Fixed audio/video out of sync after pause and resume on some Samsung devices pre-API 29."
Bug:
Test: Manual testing
Change-Id: I646221b9f94ae688592d4be96245f17f74806f0b
A camera/camera-video/src/main/java/androidx/camera/video/internal/compat/quirk/VideoEncoderSuspendDoesNotIncludeSuspendTimeQuirk.java
M camera/camera-video/src/main/java/androidx/camera/video/internal/compat/quirk/DeviceQuirksLoader.java
M camera/camera-video/src/main/java/androidx/camera/video/internal/encoder/EncoderImpl.java
Description
On Samsung J7 prime (SM-G610M) use core test app to record video,
after video recorded, audio / video is not in sync and it seems that video frames are paused in the beginning of the video.
see attached video