- Number
- 12118734
- Published
- 2024-10-15
- Filed
- 2022-06-28
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Beeler; Dominik Thabo et al.
- CPC
- G06T7/75; G06T7/33; G06T7/251; G06T17/00; G06T13/40; G06T7/246
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Markerless jaw-tracking facial performance capture technique (granted).
Abstract
Some implementations of the disclosure are directed to capturing facial training data for one or more subjects, the captured facial training data including each of the one or more subject's facial skin geometry tracked over a plurality of times and the subject's corresponding jaw poses for each of those plurality of times; and using the captured facial training data to create a model that provides a mapping from skin motion to jaw motion. Additional implementations of the disclosure are directed to determining a facial skin geometry of a subject; using a model that provides a mapping from skin motion to jaw motion to predict a motion of the subject's jaw from a rest pose given the facial skin geometry; and determining a jaw pose of the subject using the predicted motion of the subject's jaw.
Background
BRIEF SUMMARY OF THE DISCLOSURE (1) Implementations of the disclosure describe systems and methods for training and using a model to accurately track the jaw of a subject during facial performance capture based on the facial skin motion of the subject. (2) In one embodiment, a method comprises: capturing facial training data for one or more subjects, the captured facial training data including each of the one or more subject's facial skin geometry tracked over a plurality of times and the subject's corresponding jaw poses for each of those plurality of times; and using the captured facial training data to create a model that provides a mapping from skin motion to jaw motion. The mapping may be from a set of skin features that define the skin motion to a set of jaw features that define the jaw motion. In particular implementations, the jaw features are displacements of jaw points. (3) In some implementations, the facial training data is captured for a plurality of subjects over a plurality of facial expressions. In some implementations, using the captured facial training data to create a model that provides a mapping from the set of skin features that define the skin motion to the set of jaw features that define the jaw motion, comprises: for a first of the plurality of subjects, using the first subject's facial skin geometry captured over a plurality of times and the first subject's corresponding jaw poses for each of those plurality of times to learn a first mapping from a s
Claims
1. A non-transitory computer-readable medium having executable instructions stored thereon that, when executed by a processor, cause a system to perform operations comprising: obtaining a trained model that provides a mapping from facial skin motion to jaw motion, the trained model created using facial training data that includes, for each of one or more subjects, a facial skin geometry and corresponding jaw pose tracked over a plurality of times; determining a first facial skin motion of a first subject corresponding to a facial performance capture, the first subject being different from the one or more subjects; predicting, using the trained model, based at least on the first facial skin motion, a first jaw motion of a jaw of the first subject, the first jaw motion corresponding to the facial performance capture; and generating, using the first jaw motion predicted using the trained model, a facial animation of a digital character. ||
10. A method, comprising: obtaining, at a computing device, a trained model that provides a mapping from facial skin motion to jaw motion, the trained model created using facial training data that includes, for each of one or more subjects, a facial skin geometry and corresponding jaw pose tracked over a plurality of times; determining, at the computing device, a first facial skin motion of a first subject corresponding to a facial performance capture, the first subject being different from the one or more subjects; predicting, at the computing device, using the trained model, based at least on the first facial skin motion, a first jaw motion of a jaw of the first subject, the first jaw motion corresponding to the facial performance capture; and generating, at the computing device, using the first jaw motion predicted using the trained model, a facial animation of a digital character. ||
18. A non-transitory computer-readable medium having executable instructions stored thereon that, when executed by a processor, cause a system to perform operations comprising: obtaining a trained model that provides a mapping from facial skin motion to jaw motion, the trained model created using facial training data captured for one or more subjects; determining a first facial skin motion of a first subject corresponding to a facial performance capture, the first subject being different from the one or more subjects; predicting, using the trained model, based at least on the first facial skin motion, a first jaw motion of a jaw of the first subject, the first jaw motion corresponding to the facial performance capture; and generating, using the first jaw motion predicted using the trained model, a facial animation of a digital character, wherein determining the first facial skin motion comprises determining multiple skin features corresponding to the first facial skin motion, and determining the multiple skin features comprises determining a position of multiple skin feature vertices relative to a skull of the first subject. ||
19. A method, comprising: obtaining, at a computing device, a trained model that provides a mapping from facial skin motion to jaw motion, the trained model created using facial training data captured for one or more subjects; determining, at the computing device, a first facial skin motion of a first subject corresponding to a facial performance capture, the first subject being different from the one or more subjects; predicting, at the computing device, using the trained model, based at least on the first facial skin motion, a first jaw motion of a jaw of the first subject, the first jaw motion corresponding to the facial performance capture; and generating, at the computing device, using the first jaw motion predicted using the trained model, a facial animation of a digital character, wherein determining the first facial skin motion comprises determining multiple skin features corresponding to the first facial skin motion, and determining the multiple skin features comprises determining a position of multiple skin feature vertices relative to a skull of the first subject.