- Number
- 11049332
- Published
- 2021-06-29
- Filed
- 2020-03-03
- Assignee
- LUCASFILM ENTERTAINMENT COMPANY LTD.
- Inventors
- Loper; Matthew, Grabli; Stéphane, Bhat; Kiran
- CPC
- G06T19/20; G06T17/00; G06T7/80; G06T13/40
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Facial performance capture technique for uncontrolled environments (granted).
Abstract
A method of transferring a facial expression from a subject to a computer generated character that includes: receiving a plate with an image of the subject's facial expression and an estimate of intrinsic parameters of a camera used to film the plate; generating a three-dimensional parameterized deformable model of the subject's face where different facial expressions of the subject can be obtained by varying values of the model parameters; solving for the facial expression in the plate by executing a deformation solver to solve for at least some parameters of the deformable model with a differentiable renderer and shape-from-shading techniques, using as inputs, the three-dimensional parameterized deformable model, estimated intrinsic camera parameters, estimated lighting conditions and albedo estimates over a series of iterations to infer geometry of the facial expression and generate an intermediate facial; generating, from the intermediate facial mesh, refined albedo estimates for the deformable model; and solving for the facial expression in the plate by executing the deformation solver using the intermediate facial mesh, the estimated intrinsic camera parameters, the estimated
Background
FIELD (1) The present disclosure relates generally to performance capture, and more specifically to methods, techniques and systems for capturing facial expressions from a subject in previously filmed footage and transferring the captured expressions to a three-dimensional rendering of the subject. BACKGROUND OF THE INVENTION (2) Facial expression transfer is the act of adapting the facial expressions of a subject, such as an actor, to a three-dimensional computer-generated (CG) model that can be used to create visual effects that can then be incorporated into animations, movies, video games and the like. Mastering facial expression transfer and other aspects of facial animation is a long-standing challenge in computer graphics. The face can describe the emotions of a character, convey their state of mind, and hint at their future actions. Audiences are particularly trained to look at faces and identify these subtle characteristics. Accurately capturing the shape and motion of real human faces in the expression transfer process can play an important role in transferring subtle facial expressions of the subject to a computer-generated character giving the computer-generated character natural, life-like expressions. (3) In order to transfer facial expressions from a subject to a computer-generated model, the subject's facial expressions first have to be captured, for example, on digital film or another appropriate medium. Some traditional techniques that are used to capture fac
Claims
1. A method of transferring a facial expression from a subject to a computer generated character where the subject was filmed in an uncontrolled environment with low-frequency, static illumination, the method comprising: receiving a plate with an image of the facial expression of the subject and an estimate of intrinsic parameters of a camera used to film the plate; generating a three-dimensional parameterized deformable model of a face of the subject, the three-dimensional parameterized deformable model comprising a plurality of model parameters where different facial expressions of the subject can be obtained by varying values of the plurality of model parameters; generating, from the plate, an estimate of lighting conditions that illuminate the subject in the plate and albedo estimates for the three-dimensional parameterized deformable model; initially solving for the facial expression in the plate by executing a deformation solver to solve for at least some parameters of the three-dimensional parameterized deformable model with a differentiable renderer and shape-from-shading techniques, using as inputs, the three-dimensional parameterized deformable model, the estimated intrinsic camera parameters, the estimated lighting conditions and the albedo estimates over a first series of iterations to infer geometry of the facial expression of the subject in the plate and generate an intermediate facial mesh using a set of parameter values of the three-dimensional parameterized deformable model which result in an intermediate facial expression that approximates the facial expression of the subject in the plate; generating, from the intermediate facial mesh, refined albedo estimates for the three-dimensional parameterized deformable model; and subsequently solving for the facial expression in the plate by executing the deformation solver using the intermediate facial mesh, the estimated intrinsic camera parameters, the estimated lighting conditions and the refined albedo estimates as inputs over a second series of iterations to infer geometry of the facial expression of the subject in the plate and generate a final facial mesh using the set of parameter values of the three-dimensional parameterized deformable model which result in a final facial expression that more closely matches the facial expression of the subject in the plate than the intermediate facial mesh. ||
12. A method of transferring a facial expression from a subject in a previously filmed performance to a computer generated character where the subject was filmed in an uncontrolled environment with low-frequency, static illumination, the method comprising: receiving digital footage of the previously filmed performance including a plurality of sequentially ordered plates each of which includes an image of the facial expression of the subject and receiving an estimate of intrinsic parameters of a camera used to film the plurality of plates; generating a three-dimensional parameterized deformable model of a face of the subject, the three-dimensional parameterized model comprising a plurality of model parameters where different facial expressions of the subject can be obtained by varying values of the plurality of model parameters; generating, from one or more plates in the plurality of plates, an estimate of lighting conditions that illuminate the subject in each plate and albedo estimates at different vertices of the three-dimensional parameterized deformable model; generating a computer model of the previously filmed performance by, for each individual plate in the plurality of sequentially ordered plates, processing the individual plate independently of other plates in the plurality of plates to solve for the facial expression in the plate being processed by: executing a deformation solver to initially solve for at least some parameters of the three-dimensional parameterized deformable model with a differentiable renderer and shape-from-shading techniques, using as inputs, the three-dimensional parameterized deformable model, the estimated intrinsic camera parameters, the estimated lighting conditions and the albedo estimates over a first series of iterations to infer geometry of the facial expression of the subject in the plate and generate an intermediate facial mesh using a set of parameter values of the three-dimensional parameterized deformable model which result in an intermediate facial expression that approximates the facial expression of the subject in the plate; generating, from the intermediate facial mesh, refined albedo estimates at the different positions of the three-dimensional parameterized deformable model; and subsequently solving for the facial expression in the plate by executing the deformation solver using the intermediate facial mesh, the estimated intrinsic camera parameters, the estimated lighting conditions and the refined albedo estimates as inputs over a second series of iterations to infer geometry of the facial expression of the subject in the plate and generate a final facial mesh using the set of parameter values of the three-dimensional parameterized deformable model which result in a final facial expression that more closely matches the facial expression of the subject in the plate than the intermediate facial mesh.