Generating a visually consistent alternative audio for redubbing visual speech

Number: 9922665
Published: 2018-03-20
Filed: 2015-08-06
Assignee: Disney Enterprises, Inc.
Inventors: Matthews; Iain; Taylor; Sarah; Theobald; Barry John
CPC: G10L21/10; G10L25/57
Verdict: Medium Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Visually consistent audio redubbing (lip-sync dubbing).

Abstract

There are provided systems and methods for generating a visually consistent alternative audio for redubbing visual speech using a processor configured to sample a dynamic viseme sequence corresponding to a given utterance by a speaker in a video, identify a plurality of phonemes corresponding to the dynamic viseme sequence, construct a graph of the plurality of phonemes that synchronize with a sequence of lip movements of a mouth of the speaker in the dynamic viseme sequence, use the graph to generate an alternative phrase that substantially matches the sequence of lip movements of the mouth of the speaker in the video.

Background

BACKGROUND(1) Redubbing is the process of replacing the audio track in a video, and has traditionally been used in translating movies and television shows, and in video games for audiences that speak a different language than the original audio recording. Redubbing may also used to replace speech with different audio of the same language, such as redubbing a movie for television broadcast. Conventionally, a replacement audio is meticulously scripted in an attempt to select words that approximate the lip-shapes of actors or animation characters in a video, and a skilled voice actor ensures that the new recording synchronizes well with the original video. The overdubbing process can be time consuming, expensive, and discrepancies between the lip movements of the speaker in the video and the replacement audio may be distracting and appear awkward to viewers.SUMMARY(2) The present disclosure is directed to generating a visually consistent alternative audio for redubbing visual speech, substantially as shown in and/or described in connection with at least one of the figures, as set forth more completely in the claims.

Claims

1. A system for redubbing of a video, the system comprising: a display; an audio speaker; a memory for storing a redubbing application; and a processor configured to execute the reducing application to: sample a dynamic viseme sequence corresponding to an original phrase uttered by a speaking character having a sequence of original lip movements of a mouth in the video; identify, using the sampled dynamic viseme sequence, a plurality of phonemes corresponding to the sampled dynamic viseme sequence; construct a graph of the plurality of phonemes corresponding to the sampled dynamic viseme sequence; generate, using the graph of the plurality of phonemes, a first set of words including al least one word that substantially matches the sequence of the original lip movements of the mouth of the speaking character in the video; construct a second set of phrases, using the first set of words, each of the second set of phrases being an alternative phrase to the original phrase; score each of the second set of phrases based on how closely each of the second set of phrases matches the sequence of lip movements of the mouth of the speaking character in the video; select, based on the score, one of the second set of phrases as the alternative phrase to the original phrase, the alternative phrase formed by the at least one word of the first set of words substantially matching the sequence of the original lip movements of the mouth of the speaking character in the video; and display the sequence of the original lip movements of the mouth in the video on the display in synchronization with playing the at least one alternative phrase via the audio speaker. 7. A system for redubbing of a video, the system comprising: a display; an audio speaker; a memory for storing a redubbing application; and a processor configured to execute the reducing application to: sample a dynamic viseme sequence corresponding to a given utterance by a speaking character in the video; identify a plurality of phonemes corresponding to the dynamic viseme sequence; construct a graph of the plurality of phonemes corresponding to the dynamic viseme sequence; generate, using the graph of the plurality of phonemes, a plurality of words that substantially match a sequence of lip movements of a mouth of the speaking character in the video; construct a plurality of alternative phrases, each of the plurality of alternative phrases is formed by one or more of the plurality of words substantially matching the sequence of lip movements of the mouth of the speaking character in the video; score each alternative phrase of the plurality of alternative phrases based on how closely each alternative phrase matches the sequence of lip movements of the mouth of the speaking character in the video; rank the plurality of alternative phrases based on the score; and display the sequence of lip movements of the mouth in the video on the display in synchronization with playing one of the plurality of alternative phrases via the audio speaker based on ranking. 8. A system for redubbing of a video, the system comprising: a user interface; a display; an audio speaker; a memory for storing a redubbing application; and a processor configured to execute the reducing application to: sample a dynamic viseme sequence corresponding to a given utterance by a speaking character in the video; identify a plurality of phonemes corresponding to the dynamic viseme sequence; construct a graph of the plurality of phonemes corresponding to the dynamic viseme sequence; receive, from a user via the user interface, a suggested alternative phrase; transcribe the suggested alternative phrase into an ordered phoneme list; compare, using the graph, the ordered phoneme list to the dynamic viseme sequence; score how well the suggested alternative phrase matches the lip movements of the mouth of the speaking character in the video corresponding to the dynamic viseme sequence; and display the sequence of lip movements of the mouth in the video on the display in synchronization with playing the suggested alternative phrase via the audio speaker based on scoring. 10. A method for use by a system having a display, an audio speaker, a memory and a processor for redubbing of a video, the method comprising: sampling, using the processor, a dynamic viseme sequence corresponding to an original phrase uttered by a speaking character having a sequence of original lip movements of a mouth in the video; identifying, using the processor and the sampled dynamic viseme sequence, a plurality of phonemes corresponding to the sampled dynamic viseme sequence; constructing, using the processor, a graph of the plurality of phonemes corresponding to the sampled dynamic viseme sequence; generating, using the processor and the graph of the plurality of phonemes, a first set of words including at least one word that substantially matches the sequence of the original lip movements of the mouth of the speaking character in the video; constructing, using the processor, a second set of phrases, using the first set of words, each of the second set of phrases being an alternative phrase to the original phrase; scoring, using the processor, each of the second set of phrases based on how closely each of the second set of phrases matches the sequence of lip movements of the mouth of the speaking character in the video; selecting, using the processor and based on the score, one of the second set of phrases as the alternative phrase to the original phrase, the alternative phrase formed by the at least one word of the first set of words substantially matching the sequence of the original lip movements of the mouth of the speaking character in the video; and displaying, using the processor, the sequence of the original lip movements of the mouth in the video on the display in synchronization with playing the at least one alternative phrase via the audio speaker. 16. A method for use by a system having a display, an audio speaker, a memory and a processor for redubbing of a video, the method comprising: sampling, using the processor, a dynamic viseme sequence corresponding to a given utterance by a speaking character in the video; identifying, using the processor, a plurality of phonemes corresponding to the dynamic viseme sequence; constructing, using the processor, a graph of the plurality of phonemes corresponding to the dynamic viseme sequence; generating, using the processor and the graph of the plurality of phonemes, a plurality of words that substantially match a sequence of lip movements of a mouth of the speaking character in the video; constructing, using the processor, a plurality of alternative phrases, each of the plurality of alternative phrases is formed by one or more of the plurality of words substantially matching the sequence of lip movements of the mouth of the speaking character in the video; scoring, using the processor, each alternative phrase of the plurality of alternative phrases based on how closely each alternative phrase matches the sequence of lip movements of the mouth of the speaking character in the video; and ranking, using the processor, the plurality of alternative phrases based on the score; displaying, using the processor, the sequence of lip movements of the mouth in the video on the display in synchronization with playing one of the plurality of alternative phrases via the audio speaker based on ranking. 17. A method for use by a system having a display, an audio speaker, a memory and a processor for redubbing of a video, the method comprising: sampling, using the processor, a dynamic viseme sequence corresponding to a given utterance by a speaking character in the video; identifying, using the processor, a plurality of phonemes corresponding to the dynamic viseme sequence; constructing, using the processor, a graph of the plurality of phonemes corresponding to the dynamic viseme sequence; receiving, from a user via the user interface, a suggested alternative phrase; transcribing, using the processor, the suggested alternative phrase into an ordered phoneme list; comparing, using the processor and the graph, the ordered phoneme list to the dynamic viseme sequence; scoring, using the processor, how well the suggested alternative phrase matches the lip movements of the mouth of the speaking character in the video corresponding to the dynamic viseme sequence; displaying, using the processor, the sequence of lip movements of the mouth in the video on the display in synchronization with playing the suggested alternative phrase via the audio speaker based on scoring.