- Number
- 11069135
- Published
- 2021-07-20
- Filed
- 2019-11-12
- Assignee
- LUCASFILM ENTERTAINMENT COMPANY LTD.
- Inventors
- Grabli; Stéphane, Bao; Michael, Karefelt; Per, Ferrall-Nunge; Adam, Yost; Jeffery, Fedkiw; Ronald, Phillips; Cary, Helman; Pablo, Estebecorena; Leandro
- CPC
- G06T13/40; G06T15/50; G06V40/174; H04N5/2224; G06T19/20; G06V20/653; G06T17/20
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
On-set facial performance capture pipeline for CG models (granted).
Abstract
A method of transferring a facial expression from a subject to a computer generated character that includes receiving a plate with an image of the subject's facial expression, a three-dimensional parameterized deformable model of the subject's face where different facial expressions of the subject can be obtained by varying values of the model parameters, a model of a camera rig used to capture the plate, and a virtual lighting model that estimates lighting conditions when the image on the plate was captured. The method can solve for the facial expression in the plate by executing a deformation solver to solve for at least some parameters of the deformable model with a differentiable renderer and shape from shading techniques, using, as inputs, the three-dimensional parameterized deformable model, the model of the camera rig and the virtual lighting model over a series of iterations to infer geometry of the facial expression and generate a final facial mesh using the set of parameter values of the deformable model which result in a facial expression that closely matches the expression of the subject in the plate.
Background
FIELD (1) The present disclosure relates generally to performance capture, and more specifically to methods, techniques and systems for capturing facial expressions from a subject during a performance and transferring the captured expressions to a three-dimensional model of the subject. BACKGROUND OF THE INVENTION (2) Facial expression transfer is the act of adapting the facial expressions of a subject, such as an actor, to a three-dimensional computer-generated (CG) model that can be used to create visual effects that can then be incorporated into animations, movies, video games and the like. Mastering facial expression transfer and other aspects of facial animation is a long-standing challenge in computer graphics. The face can describe the emotions of a character, convey their state of mind, and hint at their future actions. Audiences are particularly trained to look at faces and identify these subtle characteristics. Accurately capturing the shape and motion of real human faces in the expression transfer process can play an important role in transferring subtle facial expressions of the subject to a computer-generated character giving the computer-generated character natural, life-like expressions. (3) In order to transfer facial expressions from a subject to a computer-generated model, the subject's facial expressions first have to be captured, for example, on digital film or another appropriate medium. Some traditional techniques that are used to capture facial expressi
Claims
1. A method of transferring a facial expression from a subject to a computer generated character, the method comprising: receiving a plate with an image of the facial expression of the subject where the plate comprises more than a million pixels with each pixel having a particular RGB value, a three-dimensional parameterized deformable model of a face of the subject where different facial expressions of the subject can be obtained by varying values of model parameters, a model of a camera rig used to capture the plate, and a virtual lighting model that estimates lighting conditions when the image of the plate was captured; solving for the facial expression in the plate by executing a deformation solver to solve for at least some parameters of the three-dimensional parameterized deformable model with a differentiable renderer and shape from shading techniques, using, as inputs, the three-dimensional parameterized deformable model, the model of the camera rig and the virtual lighting model over a series of iterations to infer geometry of the facial expression in the plate and generate a final facial mesh using a set of parameter values of the three-dimensional parameterized deformable model which result in a final facial expression that closely matches the facial expression of the subject in the plate; and wherein, during each iteration of the series of iterations, the differentiable renderer generates a rendering of the three-dimensional parameterized deformable model and the deformation solver minimizes differences between RGB values of the plate and RGB values of corresponding pixels in the rendered version of the three-dimensional parameterized deformable model. ||
16. A method of transferring a facial expression from a subject during a performance to a computer generated character, the method comprising: obtaining digital video footage of the performance including a plurality of sequentially ordered plates, each of which includes an image of the facial expression of the subject during the performance and comprises more than a million pixels with each pixel having a particular RGB value; obtaining a three-dimensional parameterized deformable model of a face of the subject where different facial expressions of the subject can be obtained by varying the values of model parameters, a model of a camera rig used to capture the performance, and a virtual lighting model that estimates lighting conditions used during the performance; generating a computer model of the performance by, for each individual plate in the plurality of sequentially ordered plates, processing a respective individual plate independently of other plates in the plurality of sequentially ordered plates to solve for the facial expression in the respective plate with a differential renderer and shape from shading techniques, using, as inputs, the three-dimensional parameterized deformable model, the model of the camera rig and the virtual lighting model over a series of iterations to infer geometry of the facial expression in the respective plate and generate a final facial mesh using a set of parameter values for the three-dimensional parameterized deformable model which result in a final facial expression that closely matches the facial expression of the subject in the respective plate being processed; and wherein, during each iteration of the series of iterations, the differentiable renderer generates a rendering of the three-dimensional parameterized deformable model and the deformation solver minimizes differences between RGB values of the respective plate and RGB values of corresponding pixels in the rendered version of the three-dimensional parameterized deformable model.