- Number
- 20240422380
- Published
- 2024-12-19
- Filed
- 2024-05-24
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Fernandez Labrador; Clara et al.
- CPC
- H04N19/60; H04N21/4307; H04N21/233; H04N21/43072; H04N21/8547
- Verdict
- Set aside AV synchronization, plumbing
- Source
- Google Patents · FreePatentsOnline
Abstract
A system includes a hardware processor and a memory storing a video/audio (V/A) synchronizer including video and audio encoders. The hardware processor executes the V/A synchronizer to receive raw video and audio extracted from media content, partition the raw video into video frame patches, partition the raw audio into audio samples, pre-process the video frame patches and the audio samples for encoding. The hardware processor further executes the V/A synchronizer to encode, using the video encoder, the pre-processed video frame patches to provide pre-processed and encoded video frame patches used to provide a latent representation of the raw video, encode, using the audio encoder, the pre-processed audio samples to provide pre-processed and encoded audio samples used to provide a latent representation of the raw audio, and synchronize, using the latent representations of the raw video and the raw audio, the raw audio with the raw video.
Background
BACKGROUND
Synchronization of the video and audio components of media content (hereinafter “V/A synchronization”) is a basic expectation held by anyone that is consuming that media content, whether through streaming, social media, cable television, theaters or any other media distribution channel. From the lens of the camera to the eye of the consumer, there are many instances where V/A synchronization errors can be introduced, such as during content mastering, third party modifications, content encoding, or client playback, to name a few examples. Studies show that the viewer experience can be negatively affected by as little as a 45 millisecond discrepancy in V/A synchronization, which is equivalent to a delay of a single frame in a 90 minute film at 25 frames per second (fps).
Although commercial solutions for performing V/A synchronization exist, their scale and capabilities are insufficient for production. As a result, detecting V/A synchronization problems and identifying their origin remain significant burdens for media production quality control teams, as these have remained largely manual processes. Thus, there is a need in the art for an automated V/A synchronization solution that can accurately identify and resolve V/A synchronization errors before they reach the viewer.
Claims
1. A system comprising: a hardware processor; and a memory storing a video/audio (V/A) synchronizer including a video encoder and an audio encoder; the hardware processor configured to execute the V/A synchronizer to: receive raw video and raw audio extracted from media content; partition the raw video into a plurality of video frame patches; partition the raw audio into a plurality of audio samples; pre-process the plurality of video frame patches for encoding to provide a plurality of pre-processed video frame patches; pre-process the plurality of audio samples for encoding to provide a plurality of pre-processed audio samples; encode, using the video encoder, the plurality of pre-processed video frame patches to provide a plurality of pre-processed and encoded video frame patches; encode, using the audio encoder, the plurality of pre-processed audio samples to provide a plurality of pre-processed and encoded audio samples; provide, using one or more of the plurality of pre-processed and encoded video frame patches, a latent representation of the raw video; provide, using the plurality of pre-processed and encoded audio samples, a latent representation of the raw audio; and synchronize, using the latent representation of the raw video and the latent representation of the raw audio, the raw audio with the raw video. ||
13. A method for use by a system including a hardware processor and a memory storing a video/audio (V/A) synchronizer including a video encoder and an audio encoder, the method comprising: receiving, by the V/A synchronizer executed by the hardware processor, raw video and raw audio extracted from media content; partitioning, by the V/A synchronizer executed by the hardware processor, the raw video into a plurality of video frame patches; partitioning, by the V/A synchronizer executed by the hardware processor, the raw audio into a plurality of audio samples; pre-processing, by the V/A synchronizer executed by the hardware processor, the plurality of video frame patches for encoding to provide a plurality of pre-processed video frame patches; pre-processing, by the V/A synchronizer executed by the hardware processor, the plurality of audio samples for encoding to provide a plurality of pre-processed audio samples; encoding, by the V/A synchronizer executed by the hardware processor and using the video encoder, the plurality of pre-processed video frame patches to provide a plurality of pre-processed and encoded video frames; encoding, by the V/A synchronizer executed by the hardware processor and using the audio encoder, the plurality of pre-processed audio samples to provide a plurality of pre-processed and encoded audio samples; providing, by the V/A synchronizer executed by the hardware processor and using one or more of the plurality of pre-processed and encoded video frame patches, a latent representation of the raw video; providing, by the V/A synchronizer executed by the hardware processor and using the plurality of pre-processed and encoded audio samples, a latent representation of the raw audio; and synchronizing, by the V/A synchronizer executed by the hardware processor and using the latent representation of the raw video and the latent representation of the raw audio, the raw audio with the raw video.