ON-SET FACIAL PERFORMANCE CAPTURE AND TRANSFER TO A THREE-DIMENSIONAL COMPUTER-GENERATED MODEL

Number: 20200286284
Published: 2020-09-10
Filed: 2019-11-12
Assignee: LUCASFILM ENTERTAINMENT COMPANY LTD.
Inventors: Grabli; Stéphane, Bao; Michael, Karefelt; Per, Ferrall-Nunge; Adam, Yost; Jeffery, Fedkiw; Ronald, Phillips; Cary, Helman; Pablo, Estebecorena; Leandro
CPC: G06T13/40; H04N5/2224; G06T17/20; G06V20/653; G06V40/174; G06T15/50; G06T19/20
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

On-set facial performance capture pipeline for CG models.

Abstract

A method of transferring a facial expression from a subject to a computer generated character that includes receiving a plate with an image of the subject's facial expression, a three-dimensional parameterized deformable model of the subject's face where different facial expressions of the subject can be obtained by varying values of the model parameters, a model of a camera rig used to capture the plate, and a virtual lighting model that estimates lighting conditions when the image on the plate was captured. The method can solve for the facial expression in the plate by executing a deformation solver to solve for at least some parameters of the deformable model with a differentiable renderer and shape from shading techniques, using, as inputs, the three-dimensional parameterized deformable model, the model of the camera rig and the virtual lighting model over a series of iterations to infer geometry of the facial expression and generate a final facial mesh using the set of parameter values of the deformable model which result in a facial expression that closely matches the expression of the subject in the plate.

Background

FIELD

The present disclosure relates generally to performance capture, and more specifically to methods, techniques and systems for capturing facial expressions from a subject during a performance and transferring the captured expressions to a three-dimensional model of the subject.BACKGROUND OF THE INVENTION

Facial expression transfer is the act of adapting the facial expressions of a subject, such as an actor, to a three-dimensional computer-generated (CG) model that can be used to create visual effects that can then be incorporated into animations, movies, video games and the like. Mastering facial expression transfer and other aspects of facial animation is a long-standing challenge in computer graphics. The face can describe the emotions of a character, convey their state of mind, and hint at their future actions. Audiences are particularly trained to look at faces and identify these subtle characteristics. Accurately capturing the shape and motion of real human faces in the expression transfer process can play an important role in transferring subtle facial expressions of the subject to a computer-generated character giving the computer-generated character natural, life-like expressions.

In order to transfer facial expressions from a subject to a computer-generated model, the subject's facial expressions first have to be captured, for example, on digital film or another appropriate medium. Some traditional techniques that are used to capture facial exp

Claims

1. A method of transferring a facial expression from a subject to a computer generated character, the method comprising: receiving a plate with an image of the subject's facial expression, a three-dimensional parameterized deformable model of the subject's face where different facial expressions of the subject can be obtained by varying values of the model parameters, a model of a camera rig used to capture the plate, and a virtual lighting model that estimates lighting conditions when the image on the plate was captured; solving for the facial expression in the plate by executing a deformation solver to solve for at least some parameters of the deformable model with a differentiable renderer and shape from shading techniques, using, as inputs, the three-dimensional parameterized deformable model, the model of the camera rig and the virtual lighting model over a series of iterations to infer geometry of the facial expression and generate a final facial mesh using the set of parameter values of the deformable model which result in a facial expression that closely matches the expression of the subject in the plate. 17. A method of transferring a facial expression from a subject during a performance to a computer generated character, the method comprising: obtaining digital video footage of the performance including a plurality of sequentially ordered plates each of which includes an image of the subject's facial expression during the performance; obtaining a three-dimensional parameterized deformable model of the subject's face where different facial expressions of the subject can be obtained by varying the values of the model parameters, a model of a camera rig used to capture the performance, and a virtual lighting model that estimates lighting conditions used during the performance; generating a computer model of the performance by, for each individual plate in the plurality of sequentially ordered plates, processing the individual plate independently of other plates in the plurality to solve for the facial expression in the plate being processed using a differential renderer with shape from shading techniques over a series of iterations to infer geometry of the facial expression and generate a final facial mesh using the set of parameter values for the deformable model which result in a facial expression that closely matches the expression of the subject in the plate being processed, wherein the solving uses the three-dimensional deformable model, the camera rig and the virtual lighting model as inputs.