JAW TRACKING WITHOUT MARKERS FOR FACIAL PERFORMANCE CAPTURE

Number: 20210012512
Published: 2021-01-14
Filed: 2019-07-12
Assignee: Disney Enterprises, Inc.
Inventors: Beeler; Dominik Thabo, Bradley; Derek Edward, Zoss; Gaspard
CPC: G06T7/75; G06T17/00; G06T7/251; G06T7/33; G06T7/246; G06T13/40
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Markerless jaw-tracking facial performance capture technique.

Abstract

Some implementations of the disclosure are directed to capturing facial training data for one or more subjects, the captured facial training data including each of the one or more subject's facial skin geometry tracked over a plurality of times and the subject's corresponding jaw poses for each of those plurality of times; and using the captured facial training data to create a model that provides a mapping from skin motion to jaw motion. Additional implementations of the disclosure are directed to determining a facial skin geometry of a subject; using a model that provides a mapping from skin motion to jaw motion to predict a motion of the subject's jaw from a rest pose given the facial skin geometry; and determining a jaw pose of the subject using the predicted motion of the subject's jaw.

Background

BRIEF SUMMARY OF THE DISCLOSURE

Implementations of the disclosure describe systems and methods for training and using a model to accurately track the jaw of a subject during facial performance capture based on the facial skin motion of the subject.

In one embodiment, a method comprises: capturing facial training data for one or more subjects, the captured facial training data including each of the one or more subject's facial skin geometry tracked over a plurality of times and the subject's corresponding jaw poses for each of those plurality of times; and using the captured facial training data to create a model that provides a mapping from skin motion to jaw motion. The mapping may be from a set of skin features that define the skin motion to a set of jaw features that define the jaw motion. In particular implementations, the jaw features are displacements of jaw points.

In some implementations, the facial training data is captured for a plurality of subjects over a plurality of facial expressions. In some implementations, using the captured facial training data to create a model that provides a mapping from the set of skin features that define the skin motion to the set of jaw features that define the jaw motion, comprises: for a first of the plurality of subjects, using the first subject's facial skin geometry captured over a plurality of times and the first subject's corresponding jaw poses for each of those plurality of times to learn a first mapping

Claims

1. A non-transitory computer-readable medium having executable instructions stored thereon that, when executed by a processor, causes a system to perform operations of: capturing facial training data for one or more subjects, the captured facial training data including each of the one or more subject's facial skin geometry tracked over a plurality of times and the subject's corresponding jaw poses for each of those plurality of times; and using the captured facial training data to create a model that provides a mapping from skin motion to jaw motion. || 9. The non-transitory computer-readable medium of 1, wherein the instructions, when executed by the processor, further causes the system to perform an operation of: using the created model to predict a jaw pose of a subject for a time using as an input the subject's corresponding facial skin geometry for the time. || 12. The non-transitory computer-readable medium of 11, wherein the instructions, when executed by the processor, further causes the system to perform operations of: capturing a plurality of calibration poses for the new subject; extracting calibration skin features and corresponding calibration bone features from each of the plurality of captured calibration poses; and using at least the extracted calibration skin features and corresponding calibration bone features, determining a transformation of skin features of the new subject to align with a feature space the mapping. || 14. A method, comprising: capturing facial training data for one or more subjects, the captured facial training data including each of the one or more subject's facial skin geometry over a plurality of times and the subject's corresponding jaw poses for each of those plurality of times; and using the captured facial training data to create a model that provides a mapping from a set of skin features to a set of jaw features. || 19. A non-transitory computer-readable medium having executable instructions stored thereon that, when executed by a processor, causes a system to perform operations of: determining a facial skin geometry of a subject; using a model that provides a mapping from skin motion to jaw motion to predict a motion of the subject's jaw from a rest pose given the facial skin geometry; and determining a jaw pose of the subject using the predicted motion of the subject's jaw.