JAW TRACKING WITHOUT MARKERS FOR FACIAL PERFORMANCE CAPTURE

Number: 20220327717
Published: 2022-10-13
Filed: 2022-06-28
Assignee: Disney Enterprises, Inc.
Inventors: Beeler; Dominik Thabo, Bradley; Derek Edward, Zoss; Gaspard
CPC: G06T17/00; G06T7/251; G06T7/33; G06T7/75; G06T7/246; G06T13/40
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Markerless jaw-tracking facial performance capture technique (pre-grant).

Abstract

Some implementations of the disclosure are directed to capturing facial training data for one or more subjects, the captured facial training data including each of the one or more subject's facial skin geometry tracked over a plurality of times and the subject's corresponding jaw poses for each of those plurality of times; and using the captured facial training data to create a model that provides a mapping from skin motion to jaw motion. Additional implementations of the disclosure are directed to determining a facial skin geometry of a subject; using a model that provides a mapping from skin motion to jaw motion to predict a motion of the subject's jaw from a rest pose given the facial skin geometry; and determining a jaw pose of the subject using the predicted motion of the subject's jaw.

Background

BRIEF SUMMARY OF THE DISCLOSURE

Implementations of the disclosure describe systems and methods for training and using a model to accurately track the jaw of a subject during facial performance capture based on the facial skin motion of the subject.

In one embodiment, a method comprises: capturing facial training data for one or more subjects, the captured facial training data including each of the one or more subject's facial skin geometry tracked over a plurality of times and the subject's corresponding jaw poses for each of those plurality of times; and using the captured facial training data to create a model that provides a mapping from skin motion to jaw motion. The mapping may be from a set of skin features that define the skin motion to a set of jaw features that define the jaw motion. In particular implementations, the jaw features are displacements of jaw points.

In some implementations, the facial training data is captured for a plurality of subjects over a plurality of facial expressions. In some implementations, using the captured facial training data to create a model that provides a mapping from the set of skin features that define the skin motion to the set of jaw features that define the jaw motion, comprises: for a first of the plurality of subjects, using the first subject's facial skin geometry captured over a plurality of times and the first subject's corresponding jaw poses for each of those plurality of times to learn a first mapping

Claims

1. A non-transitory computer-readable medium having executable instructions stored thereon that, when executed by a processor, cause a system to perform operations comprising: obtaining a model that provides a mapping from skin motion to jaw motion, the model created using facial training data captured for one or more subjects; determining a first skin motion of a first subject that is different from the one or more subjects; predicting, using the model, based at least on the first skin motion, a first jaw motion of a jaw of the first subject. || 12. A method, comprising: obtaining, at a computing device, a model that provides a mapping from skin motion to jaw motion, the model created using facial training data captured for one or more subjects; determining, at the computing device, a first skin motion of a first subject that is different from the one or more subjects; and predicting, at the computing device, using the model, based at least on the first skin motion, a first jaw motion of a jaw of the first subject.