Three-dimensional geometry-based models for changing facial identities in video frames and images

Number: 11222466
Published: 2022-01-11
Filed: 2020-09-30
Assignee: Disney Enterprises, Inc.
Inventors: Naruniec; Jacek Krzysztof, Bradley; Derek Edward, Etterlin; Thomas, Urnau Gotardo; Paulo Fabiano, Helminger; Leonhard Markus, Schroers; Christopher Richard, Weber; Romann Matthew
CPC: G06N3/045; G06N3/0455; G06N3/0464; G06N3/048; G06N3/088; G06N3/09; G06T15/04; G06T17/10; G06T19/20; G06V10/82; G06V20/52; G06V40/167; G06V40/171
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

3D-geometry facial-identity VFX technique (face swap).

Abstract

Techniques are disclosed for changing the identities of faces in video frames and images. In embodiments, three-dimensional (3D) geometry of a face is used to inform the facial identity change produced by an image-to-image translation model, such as a comb network model. In some embodiments, the model can take a two-dimensional (2D) texture map and/or a 3D displacement map associated with one facial identity as inputs and output another 2D texture map and/or 3D displacement map associated with a different facial identity. The other 2D texture map and/or 3D displacement map can then be used to render an image that includes the different facial identity.

Background

BACKGROUND Technical Field (1) Embodiments of the present disclosure relate generally to computer science and computer graphics and, more specifically, to three-dimensional geometry-based models for changing facial identities in video frames and images. Description of the Related Art (2) Oftentimes, the facial identity of an individual needs to be changed in the frames of a video or in a standalone image while maintaining the way the individual is performing within the video or image. As used herein, a “facial identity” refers to aspects of a facial appearance that are considered distinct from other aspects of a facial appearance that arise from differences in personal identities, ages, lighting conditions, and the like. Thus, two different facial identities may be attributed to different individuals or to the same individual under different conditions, such as the same individual at different ages or under different lighting conditions. As used herein, the “performance” of an individual, which also is referred to as the dynamic “behavior” of an individual, includes the facial expressions and poses with which the individual appears in the frames of a video or in a standalone image. (3) One example scenario that requires the facial identity of an individual to be changed while maintaining the way the individual is performing is when the individual needs to be portrayed at a younger age in a particular scene within a film. As another example, an individual may be unavailable fo

Claims

1. A computer-implemented method for changing an object within a video frame or image, the method comprising: generating at least one of a first texture map or a first displacement map associated with a first image that includes a first object; generating, via a machine learning model, at least one of a second texture map or a second displacement map based on the at least one of the first texture map or the first displacement map; and rendering a second image based on the at least one of the second texture map or the second displacement map and three-dimensional (3D) geometry associated with the first object or a second object. || 10. One or more non-transitory computer-readable storage media including instructions that, when executed by at least one processor, cause the at least one processor to perform steps for changing an object within a video frame or image, the steps comprising: generating at least one of a first texture map or a first displacement map associated with a first image that includes a first object; generating, via a machine learning model, at least one of a second texture map or a second displacement map based on the at least one of the first texture map or the first displacement map; and rendering a second image based on the at least one of the second texture map or the second displacement map and three-dimensional (3D) geometry associated with the first object or a second object. || 19. A system, comprising: one or more memories storing instructions; and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to: generate at least one of a first texture map or a first displacement map associated with a first image that includes a first object, generate, via a machine learning model, at least one of a second texture map or a second displacement map based on the at least one of the first texture map or the first displacement map, and render a second image based on the at least one of the second texture map or the second displacement map and three-dimensional (3D) geometry associated with the first object or a second object.