VISUAL DUBBING OF AN AUDIOVISUAL SEQUENCE

Visual/face dubbing VFX technique.

The present invention sets forth a technique for performing visual dubbing on an audiovisual sequence. The technique includes identifying, based on an actor frame included in the audiovisual sequence, one or more regions of an actor's face included in the actor frame, identifying, based on a dubber frame included in a visual recording of a dubber's performance, one or more regions of a dubber's face included in the dubber frame, generating a plurality of latent vectors based on at least one identified region of the actor's face and at least one identified region of the dubber's face, and generating, via the machine learning model, an output image based on the plurality of latent vectors.

BACKGROUND Field of the Various Embodiments

Embodiments of the present disclosure relate generally to machine learning and video effects processing and, more specifically, to techniques for dubbing an audiovisual sequence. Description of the Related Art

During the production of a live action or animated audiovisual sequence, producers, creators, dubbing directors, or distributors may wish to dub or replace one or more lines of dialogue in the audiovisual sequence with an alternate audio recording. For example, producers, creators, dubbing directors, or distributors may wish to generate a localized version of the audiovisual sequence, where dialogue included in the audiovisual sequence is replaced with a translation of the dialogue into a different language. Producers, creators, dubbing directors, or distributors may also wish to replace dialogue with an alternate version, with or without translation, to correct errors in the spoken dialogue, to achieve a different artistic goal, or to comply with ratings guidelines or societal standards.

Existing techniques for dubbing audiovisual sequences may simply replace a section of the original audio included in the audiovisual sequence with an alternate audio recording. These techniques require only that the duration of the alternate audio recording approximately matches the duration of the original audio included in the audiovisual sequence. One drawback of these techniques is that the techniques do not perform a