AUDIO SYNCHRONIZATION THROUGH LARGE LANGUAGE MODELS AND DIFFERENTIAL EQUATION-BASED MEL SPECTOGRAM INVERSION

Number: 20260310723
Published: 2026-10-08
Filed: 2025-04-03
Assignee: Disney Enterprises, Inc.
Inventors: RUTA; Dan Sebastian, MOLNAR; Logan, GARCIA; Oscar, JAVED; Muhammad, CAMPORESE; Guglielmo, HATTORI; Shinobu
CPC: G06F16/7844; G06F16/7834; G06F40/58; G10L13/08; G10L15/26
Verdict: Set aside dropped in weekly review
In edition: 2026-W41
Source: Google Patents · FreePatentsOnline

The keeper's note

Embodiments provide techniques for audio synchronization with improved timing alignment.

Abstract

Embodiments provide techniques for audio synchronization with improved timing alignment. The audio extracted from a video file is transcribed to generate an original script. Pause and timing information is extracted from the audio, where the pause and timing information comprises at least one of pauses, natural gaps, or a duration of a speech segment in the audio. A new script is generated from the original script using a language model, where the new script is structured to align with the pause and timing information of the audio. The synthesized audio is generated using the new script with a text-to-speech model.

Background

BACKGROUND

When changing the audio content of a video, efforts often focus on achieving lip synchronization, which involves adjusting the visible lip movements of the speaker to match the audio content. While these approaches address visual alignment, they neglect audio synchronization. Audio synchronization typically involves adjusting the timing, pauses, and flow of the new content audio to match the original video's structure and rhythm. Without proper audio synchronization, the new content may overrun or finish too early relative to the speaker's visible lip movements or result in unnatural-sounding speech that lacks proper cadence and natural breaks. These problems can undermine the overall quality of the final video product, giving users a disjointed experience where the audio and visible lip movements are poorly synchronized. SUMMARY

One embodiment presented in this disclosure provides a method, including transcribing audio extracted from a video file to generate an original script, extracting pause and timing information from the audio, where the pause and timing information comprises at least one of pauses, natural gaps, or a duration of a speech segment in the audio, generating a new script from the original script using a language model, where the new script is structured to align with the pause and timing information of the audio, and generating synthesized audio using the new script with a text-to-speech model.

Other embodiments in this discl

Claims

1. A method, comprising: transcribing audio extracted from a video file to generate an original script; extracting pause and timing information from the audio, wherein the pause and timing information comprises at least one of pauses, natural gaps, or a duration of a speech segment in the audio; generating a new script from the original script using a language model, wherein the new script is structured to align with the pause and timing information of the audio; and generating synthesized audio using the new script with a text-to-speech (TTS) model. || 10. A non-transitory computer-readable medium containing computer program code that, when executed by operation of one or more computer processors, performs an operation comprising: transcribing audio extracted from a video file to generate an original script; extracting pause and timing information from the audio, wherein the pause and timing information comprises at least one of pauses, natural gaps, or a duration of a speech segment in the audio; generating a new script from the original script using a language model, wherein the new script is structured to align with the pause and timing information of the audio; and generating synthesized audio using the new script with a text-to-speech (TTS) model. || 18. A system, comprising: one or more memories collectively containing one or more programs; and one or more processors, wherein the one or more processors are configured to, individually or collectively, perform an operation comprising: transcribing audio extracted from a video file to generate an original script; extracting pause and timing information from the audio, wherein the pause and timing information comprises at least one of pauses, natural gaps, or a duration of a speech segment in the audio; generating a new script from the original script using a language model, wherein the new script is structured to align with the pause and timing information of the audio; and generating synthesized audio using the new script with a text-to-speech (TTS) model.