Real-time facial performance capture.
A method of transferring a facial expression from a subject to a computer-generated character and a system and non-transitory computer-readable medium for the same. The method can include receiving an input image depicting a face of a subject; matching a first facial model to the input image; generating a displacement map representing of finer-scale details not present in the first facial model using a regression function that estimates the shape of the finer-scale details. The displacement map can be combined with the first facial model to create a second facial model that includes the finer-scale details, and the second facial model can be rendered, if desired, to create a computer-generated image of the face of the subject that includes the finer-scale details.
FIELD(1) The present disclosure relates generally to performance capture, and more specifically to methods, techniques and systems for transferring facial expressions from a subject to a computer-generated animation of the subject.BACKGROUND(2) Facial expression transfer is the act of adapting the facial expressions of a subject, such as an actor or home computer user, to a computer-generated (CG) target character. Mastering facial expression transfer and other aspects of facial animation is a long-standing challenge in computer graphics. The face can describe the emotions of a character, convey their state of mind, and hint at their future actions. Audiences are particularly trained to look at faces and identify these subtle characteristics. Accurately capturing the shape and motion of real human faces in the expression transfer process plays an important role in transferring subtle facial expressions of the subject to the CG character giving the CG character natural, life-like expressions.(3) Facial motion capture, a fundamental aspect of the facial expression transfer process, has come a long way from the original marker-based tracking approaches. Modern performance capture techniques can deliver extremely high-resolution facial geometry with very high fidelity motion information. In recent years the growing trend has been to capture faces in real-time, opening up new applications in immersive computer games, social media and real-time preview for visual effects. These met
1. A method for transferring a facial expression from a subject to a computer-generated character, the method comprising: receiving an input image capturing a facial expression of a subject; matching a first facial model to the facial expression in the input image; generating a displacement map representing finer-scale details not present in the first facial model using a regression function, wherein the regression function estimates the finer-scale details with one or more shapes using shading information determined from the input image, and wherein the one or more shapes correspond to the facial expression in the input image; combining the displacement map with the first facial model to create a second facial model that includes the finer-scale details; and rendering the second facial model to create a computer-generated image of the facial expression of the subject that includes the finer-scale details, wherein the computer-generated image is used for a computer-generated animation.
14. A system comprising: one or more processors; and a computer-readable memory coupled to the one or more processors, the computer readable memory storing instructions that cause the one or more processors to: receive an input image capturing a facial expression of a subject; match a first facial model to the facial expression in the input image; generate a displacement map representing finer-scale details not present in the first facial model using a regression function, wherein the regression function estimates the finer-scale details with one or more shapes using shading information determined from the input image, and wherein the one or more shapes correspond to the facial expression in the input image; combine the displacement map with the first facial model to create a second facial model that includes the finer-scale details; and render the second facial model to create a computer-generated image of the facial expression of the subject that includes the finer-scale details, wherein the computer-generated image is used for a computer-generated animation.
16. A non-transitory computer-readable medium storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising: instructions that receive an input image capturing a facial expression of a subject; instructions that match a first facial model to the facial expression in the input image; instructions that generate a displacement map representing finer-scale details not present in the first facial model using a regression function, wherein the regression function estimates the finer-scale details with one or more shapes using shading information determined from the input image, and wherein the one or more shapes correspond to the facial expression in the input image; instructions that combine the displacement map with the first facial model to create a second facial model that includes the finer-scale details; and instructions that render the second facial model to create a computer-generated image of the facial expression of the subject that includes the finer-scale details, wherein the computer-generated image is used for a computer-generated animation.
17. A method of transferring captured information from a subject to a computer-generated version of the subject, the method comprising: receiving an input image capturing a facial expression of a subject; matching a first model comprising a plurality of first features to the facial expression in the input image, the first model being a statistical global model created from a plurality of samples each of which includes the plurality of first features; generating a displacement map representing one or more second features not present in the first model using a regression function that estimates the one or more second features with one or more shapes using shading information determined from the input image, wherein the one or more shapes correspond to the facial expression in the input image, and wherein the one or more second features are one or more features present in one or more of the samples but not all the samples or where variation of the one or more second features between the plurality of samples is too complex to be subsumed by the first model; combining the displacement map with the first model to create a second model; and rendering the second model to create a computer-generated image of the facial expression of the subject, wherein the computer-generated image is used for a computer-generated animation.