SEMANTIC DEEP FACE MODELS

Number: 20210279956
Published: 2021-09-09
Filed: 2020-03-04
Assignee: DISNEY ENTERPRISES, INC.
Inventors: CHANDRAN; Prashanth, BEELER; Dominik Thabo, BRADLEY; Derek Edward
CPC: G06T17/20; G06V40/175; G06T13/40; G06T19/20; G06V40/168; G06V10/754
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Deep-learning semantic face modeling technique.

Abstract

Techniques are disclosed for training and applying nonlinear face models. In embodiments, a nonlinear face model includes an identity encoder, an expression encoder, and a decoder. The identity encoder takes as input a representation of a facial identity, such as a neutral face mesh minus a reference mesh, and outputs a code associated with the facial identity. The expression encoder takes as input a representation of a target expression, such as a set of blendweight values, and outputs a code associated with the target expression. The codes associated with the facial identity and the facial expression can be concatenated and input into the decoder, which outputs a representation of a face having the facial identity and expression. The representation of the face can include vertex displacements for deforming the reference mesh.

Background

BACKGROUND Technical Field

Embodiments of the present disclosure relate generally to computer vision and computer graphics and, more specifically, to semantic deep face models. Description of the Related Art

Multi-linear morphable models that are built from three-dimensional (3D) face databases are commonly used to generate virtual 3D geometry representing human faces, which are also referred to herein as “faces.” Such models typically generate a tensor of different dimensions that a user is permitted to control, such as the identity and expressions of faces that are being generated. User control over the identity and expressions of faces is oftentimes referred to as having “semantic control” of those facial dimensions. One drawback of multi-linear morphable models is that these models rely on linear combinations of different dataset shapes to generate faces, which can limit the quality and expressiveness of the generated faces. For example, the linear blending of facial shapes can result in an unrealistic-looking facial expression or unwanted artifacts, because human faces are highly nonlinear in their deformations.

Non-linear face models, including those based on deep neural networks, have been used to generate more realistic-looking facial images. However, typical non-linear face models do not produce 3D geometry and do not have any notion of semantic control. In particular, such models cannot be used to generate faces having user-controllable identiti

Claims

1. A computer-implemented method for generating a representation of a face, the method comprising: receiving a representation of a facial identity and a representation of a facial expression; and performing one or more decoding operations based on an encoded representation of the facial identity and an encoded representation of the facial expression to generate the representation of the face having the facial identity and the facial expression. || 11. A non-transitory computer-readable storage medium including instructions that, when executed by a processing unit, cause the processing unit to perform steps for generating a representation of a face, the steps comprising: receiving a representation of a facial identity and a representation of a facial expression; and processing an encoded representation of the facial identity and an encoded representation of the facial expression using a decoder to generate the representation of the face having the facial identity and the facial expression. || 20. A computer-implemented method of training a model for generating a representation of a face, the method comprising: receiving a data set that includes meshes of neutral faces and meshes of faces having expressions; and training a model based on the data set, wherein the model comprises: a first encoder configured to map representations of neutral faces to encoded representations of facial identities, a second encoder configured to map representations of facial expressions to encoded representations of the facial expressions, and a decoder configured to map encoded representations of facial identities and encoded representations of facial expressions to representations of faces.