MULTI-CAMERA FACE SWAPPING

Number: 20240078726
Published: 2024-03-07
Filed: 2022-09-07
Assignee: DISNEY ENTERPRISES, INC.
Inventors: Weber; Romann Matthew et al.
CPC: G06T11/60; G06V10/82; G06V20/653; G06V40/172
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Multi-camera face-swapping VFX technique.

Abstract

One embodiment of the present invention sets forth a technique for performing face swapping. The technique includes converting a first input image that depicts a first facial identity from a first viewpoint at a first time into a first latent representation and converting a second input image that depicts the first facial identity from a second viewpoint at the first time into a second latent representation. The technique also includes generating, via a first machine learning model, a first output image that depicts a second facial identity from the first viewpoint based on the first latent representation. The technique further includes generating, via the first machine learning model, a second output image that depicts the second facial identity from the second viewpoint based on the second latent representation.

Background

BACKGROUND Field of the Various Embodiments

Embodiments of the present disclosure relate generally to machine learning and computer vision and, more specifically, to techniques for performing multi-camera face swapping. Description of the Related Art

Face swapping refers to the changing of the facial identity of an individual in standalone images or video frames while maintaining the performance of the individual within the standalone images or video frames. This facial identity includes aspects of a facial appearance that arise from differences in personal identities, ages, lighting conditions, and/or other factors. For example, two different facial identities may be attributed to two different individuals, the same individual under different lighting conditions, and/or the same individual at different ages. Further, the performance of an individual, which also is referred to as the dynamic behavior of an individual, includes the facial expressions and poses of the individual, as depicted in the video frames or in standalone images.

Face swapping can be conducted under various types of scenarios. For example, the facial identity of an actor within a given scene of video content (e.g., a film, a show, etc.) could be changed to a different facial identify of the same actor at a younger age or at an older age. In another example, a first actor could be unavailable for a video shoot because of scheduling conflicts, because the first actor is deceased, and/or

Claims

1. A computer-implemented method for performing face swapping, the computer-implemented method comprising: converting a first input image that depicts a first facial identity from a first viewpoint at a first time into a first latent representation; converting a second input image that depicts the first facial identity from a second viewpoint at the first time into a second latent representation; generating, via a first machine learning model, a first output image that depicts a second facial identity from the first viewpoint based on the first latent representation; and generating, via the first machine learning model, a second output image that depicts the second facial identity from the second viewpoint based on the second latent representation. || 11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: converting a first input image that depicts a first facial identity from a first viewpoint at a first time into a first latent representation; converting a second input image that depicts the first facial identity from a second viewpoint at the first time into a second latent representation; generating, via a first machine learning model, a first output image that depicts a second facial identity from the first viewpoint based on the first latent representation; and generating, via the first machine learning model, a second output image that depicts the second facial identity from the second viewpoint based on the second latent representation. || 20. A system, comprising: one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform the steps of: converting a first input image that depicts a first facial identity from a first viewpoint at a first time into a first latent representation; converting a second input image that depicts the first facial identity from a second viewpoint at the first time into a second latent representation; generating, via a first machine learning model, a first output image that depicts a second facial identity from the first viewpoint based on the first latent representation; and generating, via the first machine learning model, a second output image that depicts the second facial identity from the second viewpoint based on the second latent representation.