Semantic deep face models

Number: 11276231
Published: 2022-03-15
Filed: 2020-03-04
Assignee: DISNEY ENTERPRISES, INC.
Inventors: Chandran; Prashanth, Beeler; Dominik Thabo, Bradley; Derek Edward
CPC: G06V10/754; G06T17/20; G06V40/168; G06V40/175; G06T13/40; G06T19/20
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Semantic facial modeling technique for animation/VFX.

Abstract

Techniques are disclosed for training and applying nonlinear face models. In embodiments, a nonlinear face model includes an identity encoder, an expression encoder, and a decoder. The identity encoder takes as input a representation of a facial identity, such as a neutral face mesh minus a reference mesh, and outputs a code associated with the facial identity. The expression encoder takes as input a representation of a target expression, such as a set of blendweight values, and outputs a code associated with the target expression. The codes associated with the facial identity and the facial expression can be concatenated and input into the decoder, which outputs a representation of a face having the facial identity and expression. The representation of the face can include vertex displacements for deforming the reference mesh.

Background

BACKGROUND Technical Field (1) Embodiments of the present disclosure relate generally to computer vision and computer graphics and, more specifically, to semantic deep face models. Description of the Related Art (2) Multi-linear morphable models that are built from three-dimensional (3D) face databases are commonly used to generate virtual 3D geometry representing human faces, which are also referred to herein as “faces.” Such models typically generate a tensor of different dimensions that a user is permitted to control, such as the identity and expressions of faces that are being generated. User control over the identity and expressions of faces is oftentimes referred to as having “semantic control” of those facial dimensions. One drawback of multi-linear morphable models is that these models rely on linear combinations of different dataset shapes to generate faces, which can limit the quality and expressiveness of the generated faces. For example, the linear blending of facial shapes can result in an unrealistic-looking facial expression or unwanted artifacts, because human faces are highly nonlinear in their deformations. (3) Non-linear face models, including those based on deep neural networks, have been used to generate more realistic-looking facial images. However, typical non-linear face models do not produce 3D geometry and do not have any notion of semantic control. In particular, such models cannot be used to generate faces having user-controllable identities and ex

Claims

1. A computer-implemented method for generating a representation of a face, the computer-implemented method comprising: receiving a representation of a facial identity and a representation of a facial expression; processing the representation of the facial identity using a first variational autoencoder (VAE) encoder to generate an encoded representation of the facial identity; processing the representation of the facial expression using a second VAE encoder to generate an encoded representation of the facial expression; and performing one or more decoding operations based on the encoded representation of the facial identity and the encoded representation of the facial expression to generate the representation of the face having the facial identity and the facial expression. || 11. A non-transitory computer-readable storage medium including instructions that, when executed by a processing unit, cause the processing unit to perform steps for generating a representation of a face, the steps comprising: receiving a representation of a facial identity and a representation of a facial expression; processing the representation of the facial identity using a first variational autoencoder (VAE) encoder to generate an encoded representation of the facial identity; and processing the representation of the facial expression using a second VAE encoder to generate an encoded representation of the facial expression; and processing the encoded representation of the facial identity and the encoded representation of the facial expression using a decoder to generate the representation of the face having the facial identity and the facial expression. || 20. A computer-implemented method of training a model for generating a representation of a face, the computer-implemented method comprising: receiving a data set that includes meshes of neutral faces that respectively represent a face having a neutral expression and meshes of faces that respectively represent a face having a facial expression; and training a model based on the data set, wherein the model comprises: a first variational autoencoder (VAE) encoder configured to map representations of neutral faces to encoded representations of facial identities, a second VAE encoder configured to map representations of faces having facial expressions to encoded representations of the facial expressions, and a decoder configured to map encoded representations of facial identities and encoded representations of facial expressions to representations of faces.