- Number
- 20180096511
- Published
- 2018-04-05
- Filed
- 2017-12-06
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Beeler; Thabo; Bradley; Derek; Cao; Chen
- CPC
- G06V40/16; G06V40/176; G06T15/04; G06T13/40
- Verdict
- High Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Real-time facial performance capture (PGPUB dup).
Abstract
A method of transferring a facial expression from a subject to a computer-generated character and a system and non-transitory computer-readable medium for the same. The method can include receiving an input image depicting a face of a subject; matching a first facial model to the input image; generating a displacement map representing of finer-scale details not present in the first facial model using a regression function that estimates the shape of the finer-scale details. The displacement map can be combined with the first facial model to create a second facial model that includes the finer-scale details, and the second facial model can be rendered, if desired, to create a computer-generated image of the face of the subject that includes the finer-scale details.
Background
FIELD
The present disclosure relates generally to performance capture, and more specifically to methods, techniques and systems for transferring facial expressions from a subject to a computer-generated animation of the subject.BACKGROUND
Facial expression transfer is the act of adapting the facial expressions of a subject, such as an actor or home computer user, to a computer-generated (CG) target character. Mastering facial expression transfer and other aspects of facial animation is a long-standing challenge in computer graphics. The face can describe the emotions of a character, convey their state of mind, and hint at their future actions. Audiences are particularly trained to look at faces and identify these subtle characteristics. Accurately capturing the shape and motion of real human faces in the expression transfer process plays an important role in transferring subtle facial expressions of the subject to the CG character giving the CG character natural, life-like expressions.
Facial motion capture, a fundamental aspect of the facial expression transfer process, has come a long way from the original marker-based tracking approaches. Modern performance capture techniques can deliver extremely high-resolution facial geometry with very high fidelity motion information. In recent years the growing trend has been to capture faces in real-time, opening up new applications in immersive computer games, social media and real-time preview for visual effects.
Claims
1. A method comprising: receiving an enhancement function trained to infer a shape of a detail from a two-dimensional image; receiving a plurality of two-dimensional images of a subject, wherein the plurality of two-dimensional images include a first set of one or more images and a second image, wherein the first set of one or more images are used for customizing use of the enhancement function to the subject, and wherein the second image is used with the enhancement function to render a three-dimensional representation of the subject; receiving a generic model, the generic model including one or more features common among a plurality of subjects; fitting the generic model to the subject in the first image; identifying a location in the generic model that corresponds to a location in the first image, wherein the location in the first image includes a particular detail that is not present in the location in the generic model; fitting the generic model to the subject in the second image; determining, using the enhancement function on the location, the particular detail; creating an enhanced model for the second image, wherein the enhanced model is created by adding the particular detail to the generic model fit to the subject in the second image; and rendering, using the enhanced model, the three-dimensional representation of the subject, wherein the three-dimensional representation corresponds to the second image.
9. A non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more processors, the plurality of instructions when executed by the one or more processors cause the one or more processors to: receive an enhancement function trained to infer a shape of a detail from a two-dimensional image; receive a plurality of two-dimensional images of a subject, wherein the plurality of two-dimensional images include a first set of one or more images and a second image, wherein the first set of one or more images are used for customizing use of the enhancement function to the subject, and wherein the second image is used with the enhancement function to render a three-dimensional representation of the subject; receive a generic model, the generic model including one or more features common among a plurality of subjects; fit the generic model to the subject in the first image; identify a location in the generic model that corresponds to a location in the first image, wherein the location in the first image includes a particular detail that is not present in the location in the generic model; fit the generic model to the subject in the second image; determine, using the enhancement function on the location, the particular detail; create an enhanced model for the second image, wherein the enhanced model is created by adding the particular detail to the generic model fit to the subject in the second image; and render, using the enhanced model, the three-dimensional representation of the subject, wherein the three-dimensional representation corresponds to the second image.
16. A method comprising: receiving an enhancement function trained to infer a shape of a detail from a two-dimensional image; receiving a plurality of two-dimensional images of a facial expression of a subject, wherein the plurality of two-dimensional images include a first set of one or more images and a second image, wherein the first set of one or more images are used for customizing use of the enhancement function to the subject, and wherein the second image is used with the enhancement function to render a three-dimensional representation of the facial expression of the subject; receiving a generic facial model, the generic facial model including one or more facial features common among a plurality of subjects; fitting the generic facial model to the subject in the first image; identifying a location in the generic facial model that corresponds to a location in the first image, wherein the location in the first image includes a particular detail that is not present in the location in the generic facial model; fitting the generic facial model to the subject in the second image; determining, using the enhancement function on the location, the particular detail; creating an enhanced facial model for the second image, wherein the enhanced facial model is created by adding the particular detail to the generic facial model fit to the subject in the second image; and rendering, using the enhanced facial model, the three-dimensional representation of the facial expression of the subject, wherein the three-dimensional representation corresponds to the second image.