audio/video out of sync in video capture [Samsung J7 prime, A10e, J4] [202798572]

Fixed

Bug

Status Update

No update yet.

Description

wu...@google.com

created issue on behalf of sc...@google.com #1

Oct 12, 2021 05:52AM

On Samsung J7 prime (SM-G610M) use core test app to record video,

after video recorded, audio / video is not in sync and it seems that video frames are paused in the beginning of the video.

see attached video

Comments

we...@google.com <we...@google.com> #2Nov 10, 2021 03:54AM

Project: platform/frameworks/support
Branch: androidx-master-dev

commit c60f33e229e31ab328ef6b59dab63b264954831c
Author: Alexandre Elias <aelias@google.com>
Date: Fri Jul 10 16:23:09 2020

Semantics no-op cleanups

Partly in response to lmr's broad code review, I did a pass of
superficial API/implementation cleanup. The main changes are:

- I changed each Boolean SemanticsProperty where false is equivalent to
not being present to take "Unit" instead. This is conceptually
clearer: it avoids questions like "can I cancel out a semantics from a
merged child by setting it to false?" Because "property = Unit" looks
weird, I also changed the style of these to "property()".

- I moved the Semantics id generator closer to where it's used, in
SemanticsModifierCore. I made it internal and an AtomicInt.
(Note that integer ids are heavily used in the Android
AccessibilityNodeInfo APIs so I can't simply remove them entirely.)

- I deleted dead code. Some examples include SemanticsHintOverrides,
a public API not connected to anything, and SemanticsPropertyKey
merge() open method which is never called. (In both cases I have
a different plan in mind for accessibility.)

Fixes: 145951226
Fixes: 145955412
Test: existing tests
Relnote: "Single-value semantics properties now use a calling style.
For example, 'semantics { hidden = true }' is now written as:
'semantics { hidden() }'."

Change-Id: Ic1afd12ea22c926babc9662f1804d80b33aa0cfc

M ui/integration-tests/benchmark/src/androidTest/java/androidx/ui/benchmark/test/LayoutNodeModifierBenchmark.kt
M ui/ui-core/api/0.1.0-dev15.txt
M ui/ui-core/api/current.txt
M ui/ui-core/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-core/api/public_plus_experimental_current.txt
M ui/ui-core/api/restricted_0.1.0-dev15.txt
M ui/ui-core/api/restricted_current.txt
M ui/ui-core/src/androidAndroidTest/kotlin/androidx/ui/graphics/vector/VectorTest.kt
M ui/ui-core/src/androidAndroidTest/kotlin/androidx/ui/semantics/SemanticsTests.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidActuals.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidComposeView.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidComposeViewAccessibilityDelegateCompat.kt
M ui/ui-core/src/androidMain/kotlin/androidx/ui/core/AndroidPopup.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/Expect.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsConfiguration.kt
D ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsHintOverrides.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsModifier.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsNode.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsOwner.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/core/semantics/SemanticsWrapper.kt
M ui/ui-core/src/commonMain/kotlin/androidx/ui/semantics/SemanticsProperties.kt
M ui/ui-foundation/api/0.1.0-dev15.txt
M ui/ui-foundation/api/current.txt
M ui/ui-foundation/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-foundation/api/public_plus_experimental_current.txt
M ui/ui-foundation/api/restricted_0.1.0-dev15.txt
M ui/ui-foundation/api/restricted_current.txt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Clickable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Dialog.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/Scroller.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/selection/Selectable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/selection/Toggleable.kt
M ui/ui-foundation/src/main/java/androidx/ui/foundation/semantics/FoundationSemanticsProperties.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ButtonTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/CardTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/CheckboxScreenshotTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/RadioButtonScreenshotTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ScaffoldTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/SnackbarTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/SurfaceTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/ripple/RippleIndicationTest.kt
M ui/ui-material/src/androidTest/java/androidx/ui/material/textfield/TextFieldScreenshotTest.kt
M ui/ui-material/src/main/java/androidx/ui/material/AppBar.kt
M ui/ui-material/src/main/java/androidx/ui/material/TextFieldImpl.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/AssertsTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/CallSemanticsActionTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/ErrorMessagesTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/FindersTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/PrintToStringTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/ScrollToTest.kt
M ui/ui-test/src/androidTest/java/androidx/ui/test/TextActionsTest.kt
M ui/ui-test/src/main/java/androidx/ui/test/Actions.kt
M ui/ui-test/src/main/java/androidx/ui/test/Filters.kt
M ui/ui-text/api/0.1.0-dev15.txt
M ui/ui-text/api/current.txt
M ui/ui-text/api/public_plus_experimental_0.1.0-dev15.txt
M ui/ui-text/api/public_plus_experimental_current.txt
M ui/ui-text/api/restricted_0.1.0-dev15.txt
M ui/ui-text/api/restricted_current.txt
M ui/ui-text/src/commonMain/kotlin/androidx/ui/text/CoreTextField.kt
M ui/ui-text/src/commonMain/kotlin/androidx/ui/text/TextSemanticsProperties.kt

https://android-review.googlesource.com/1360099

we...@google.com <we...@google.com> #3Nov 10, 2021 06:12AM

After comparing some Samsung devices, the A10e is the most obvious about avsyncing issue. \ J4+ is also easy to reproduce, then J4.

ma...@gmail.com <ma...@gmail.com> #4Dec 15, 2021 11:02PM

It is actually fairly noticeable if you record one of the "audio sync test 60fps" type videos on YouTube with a device. The video on my pixel 1 is about 120ms behind the audio.

Looking at the VideoCapture code I strongly suspect that this is due to the presentation time of the video frames being stamped and provided to the muxer as the frames come out of the codec (rather than going in to the codec which is when they are actually in sync with the audio). Your video frames will always be behind and out of sync by however long the codec takes to chew on each frame.

I'm pretty sure that at least since Android 8 the presentation times are properly provided to the surface by the camera (also from the high precision clock) and may not need to be overwritten later with one that is delayed. That may be helpful.

Good luck!

tr...@google.com <tr...@google.com> #5Dec 16, 2021 10:25PM

Thanks for the tips. I agree this is a good place to start looking at. We will take a look.

wu...@google.com <wu...@google.com> #6Dec 27, 2021 09:30AM

A developer reported the same issue: https://groups.google.com/a/android.com/g/camerax-developers/c/cBnVyvwSxGw/m/c8WIRtPFDgAJ

le...@google.com <le...@google.com> Mar 15, 2022 05:15AM

Reassigned to le...@google.com.

le...@google.com <le...@google.com> Mar 22, 2022 03:10AM

Accepted by le...@google.com.

le...@google.com <le...@google.com> #7Mar 24, 2022 04:08AM

The attached video in b/202105088 appears to use pause function. There are some Samsung devices have audio/video out of sync issue after invoking MediaCodec#setParameters with MediaCodec.PARAMETER_KEY_SUSPEND. It will affect not only the current recording video using pause resume function but also the beginning of the next recording.

ap...@google.com <ap...@google.com> #8Mar 29, 2022 01:24AM

Project: platform/frameworks/support
Branch: androidx-main

commit 75cbc0b8a928dfc6bf3d75da027ed396ba684c87
Author: leo huang <leohuang@google.com>
Date: Wed Mar 23 18:45:00 2022

Fix audio/video out of sync after pause and resume on some Samsung devices pre-API 29

After calling MediaCodec.setParameters() with MediaCodec.PARAMETER_KEY_SUSPEND, the video codec of the issue device will send a timestamp missing the pause interval after resumed, which results in the video and audio to be out of sync after resumed.

The workaround is to not call MediaCodec.setParameters() when recording is paused. This shouldn't cause too much overhead since we'll still stop the camera from sending frames to the encoder surface when paused.

Relnote: "Fixed audio/video out of sync after pause and resume on some Samsung devices pre-API 29."

Bug:

b/202798609

b/202798572

Test: Manual testing
Change-Id: I646221b9f94ae688592d4be96245f17f74806f0b

A camera/camera-video/src/main/java/androidx/camera/video/internal/compat/quirk/VideoEncoderSuspendDoesNotIncludeSuspendTimeQuirk.java
M camera/camera-video/src/main/java/androidx/camera/video/internal/compat/quirk/DeviceQuirksLoader.java
M camera/camera-video/src/main/java/androidx/camera/video/internal/encoder/EncoderImpl.java

https://android-review.googlesource.com/2038784

le...@google.com <le...@google.com> Mar 29, 2022 01:25AM

Marked as fixed.