REAL-TIME HIGH-QUALITY FACIAL PERFORMANCE CAPTURE

Number: 20170024921
Published: 2017-01-26
Filed: 2015-09-30
Assignee: Disney Enterprises, Inc.
Inventors: Beeler; Thabo et al.
CPC: G06T13/40; G06T15/04
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Real-time high-quality facial performance-capture technique.

Abstract

A method of transferring a facial expression from a subject to a computer-generated character and a system and non-transitory computer-readable medium for the same. The method can include receiving an input image depicting a face of a subject; matching a first facial model to the input image; generating a displacement map representing of finer-scale details not present in the first facial model using a regression function that estimates the shape of the finer-scale details. The displacement map can be combined with the first facial model to create a second facial model that includes the finer-scale details, and thesecond facial model can be rendered, if desired, to create a computer-generated image of the face of the subject that includes the finer-scale details.

Background

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a flowchart illustrating a method according to some embodiments of the present disclosure;

FIG. 2 is a block diagram of an animation system according to embodiments of the disclosure that can implement the method shown in FIG.1;

FIG. 3 is a flowchart illustrating a method of implementing training stage 110 shown in FIG. 1 according to some embodiments of the present disclosure;

FIG. 4 is a simplified block diagram of a training engine 400 that can execute the method illustrated in FIG. 3 according to some embodiments of the disclosure;

FIG. 5 illustrates a highresolution training mesh tracked to a neutral expression and an extreme expression and further illustrates the extraction of medium-frequency details from the mesh according to some embodiments of the disclosure;

FIG. 6 illustrates an expression and aligned mesh along with images that depict displacement information and a displacement mesh in accordance with some embodiments of the disclosure;

FIG. 7 illustrates the selection of wrinkle patches from a texture image according to some embodiments of the disclosure;

FIG. 8 illustrates a set of 16 eigenvectors of displacement patches from a training set thatcan encode the predominant variations in wrinkle shape according to some embodiments of the disclosure;

FIGS. 9A and 9B are flowcharts illustrating a method of implementing performance capture and enhancement stage 120 shown

Claims

1. A method of transferring a facial expression from a subject to a computer-generated character, the method comprising: receiving an input image depicting a face of a subject; matching a first facial model to the input image; generating a displacement map representing the finer-scale details not present in the first facial model using a regression function that estimates the shape of the finer-scale details. 16. A system comprising: one or more processors; a computer-readable memory coupled to the one or more processors, the computer readable memorystoring instructions that cause the one or more processors to: receive an input image depicting a face of a subject; match a first facial model to the input image; generate a displacement map representing the finer-scale details not present in the first facial model using a regression function that estimates the shape of the finer-scale details. 19. A non-transistory computer-readable medium storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising: instructions that receive aninput image depicting a face of a subject; instructions that match a first facial model to the input image; instructions that generate a displacement map representing the finer-scale details not present in the first facial model using a regression function that estimates the shape of the finer-scale details; and instructions that combine the displacement map with the first facial model to create a second facial model that includes the finer-scale details. 20. A method of transferring captured information from a subject to a computer-generated version of the subject, the method comprising: receiving an input image depicting the subject; matching a first model comprising a plurality of first features to the input image, the first model being a statistical global model created from a plurality of samples each of which includes the plurality of first features; generating a displacement map representing second features not present in the first model using a regression function that estimates the shape of the second features, wherein the second features are features present in one or more of the samples but not all the samples or where variation of the second features between the plurality of samples is too complex to be subsumed by the first model; and combining the displacement map with the first model to create a second model thatincludes the plurality of first features and the second features.