Generating a Visually Consistent Alternative Audio for Redubbing Visual Speech

Number: 20170040017
Published: 2017-02-09
Filed: 2015-08-06
Assignee: Disney Enterprises, Inc.
Inventors: Matthews; Iain et al.
CPC: G10L25/57; G10L21/10
Verdict: Medium Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Generates visually consistent alternative audio for redubbing visual speech — animation/dubbing production technique.

Abstract

There are provided systems and methods for generating a visually consistent alternative audio for redubbing visual speech using a processor configured to sample a dynamic viseme sequence corresponding to a given utterance by a speaker in a video, identify a plurality of phonemes corresponding to the dynamic viseme sequence, construct a graph of the plurality of phonemes that synchronize with a sequence of lipmovements of a mouth of the speaker in the dynamic viseme sequence, use the graph to generate an alternative phrase that substantially matches the sequence of lip movements of themouth of the speaker in the video.

Background

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary system for generating visually consistent alternative audio for visual speech redubbing, according to one implementation of the present disclosure;

FIG. 2a illustrates an exemplary diagram showing a sampling of phoneme string distributions for three dynamic viseme classes and depicting the complex many-to-many mapping between phoneme sequences and dynamic visemes, according to one implementation of the present disclosure;

FIG. 2b illustrates an exemplary diagram showing phonemes and dynamic visemes corresponding to the phrase “a helpful leaflet,” according to one implementation of the present disclosure;

FIG. 3 illustrates a diagram displaying examples of visually consistent speech redubbing, according to one implementation of the present disclosure;

FIG. 4 illustrates an exemplary flowchart of a method of visually consistent speech redubbing, according to one implementation of the present disclosure; and

FIG. 5 illustrates an exemplary flowchart of a method of visually consistent speech redubbing, according to one implementation of the present disclosure.DETAILED DESCRIPTION

The following description contains specific information pertaining to implementations in the present disclosure. The drawings in the present application and their accompanying detailed description are directed to merely exemplary implementations. Unless noted otherwise, like or corresponding elemen

Claims

1. A system for redubbing of a video, the system comprising: a memory for storing a redubbing application; a processor configured to execute the reducing application to: sample a dynamic viseme sequence corresponding to a given utterance by a speaker in the video; identify a plurality of phonemes corresponding to the dynamic viseme sequence; construct a graph of the plurality of phonemes corresponding to the dynamic viseme sequence; generate, using the graph of the plurality of phonemes, a first set including at least one word that substantially matches a sequence of lip movements of a mouth of the speaker in the video; and construct a second set including at leastone alternative phrase, the at least one alternative phrase formed by the at least one word of the first set that substantially matches the sequence of lip movements of the mouth of the speaker in the video. 11. A method for use by a system having a memory and a processor for redubbing of a video, the method comprising: sampling, using the processor, a dynamic viseme sequence corresponding to a given utterance by a speaker in the video; identifying, using the processor, a plurality of phonemes corresponding to the dynamic viseme sequence; constructing, using the processor, a graph of the plurality of phonemes corresponding to the dynamic viseme sequence; generating, using the processor, a first set including at least one word that substantially matches a sequence of lip movements of a mouth of the speaker in the video using the graph of the plurality of phonemes; and constructing, usingthe processor, a second set including at least one alternative phrase, the at least one alternative phrase formed by the at least one word of the first set that substantially matches the sequence of lip movements of the mouth of the speaker in the video.