- Number
- 9639737
- Published
- 2017-05-02
- Filed
- 2015-09-29
- Assignee
- ETH ZÜRICH (EIDGENÖESSISCHE TECHNISCHE HOCHSCHULE ZÜRICH)
- Inventors
- Beeler; Thabo, Bradley; Derek, Wu; Chenglei
- CPC
- G06V20/653; G06T17/20; G06T19/20; G06V40/165; G06T7/223; G06T7/246; G06V40/176
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Generates an anatomically-constrained local model with shape and anatomical subspaces to improve performance capture accuracy (granted, ETH Zurich co-assigned).
Abstract
Techniques and systems are described for generating an anatomically-constrained local model and for performing performance capture using the model. The local model includes a local shape subspace and an anatomical subspace. In one example, the local shape subspace constrains local deformation of various patches that represent the geometry of a subject's face. In the same example, the anatomical subspace includes an anatomical bone structure, and can be used to constrain movement and deformation of the patches globally on the subject's face. The anatomically-constrained local face model and performance capture technique can be used to track three-dimensional faces or other parts of a subject from motion data in a high-quality manner. Local model parameters that best describe the observed motion of the subject's physical deformations (e.g., facial expressions) under the given constraints are estimated through optimization. The optimization can solve for rigid local patch motion, local patch deformation, and the rigid motion of the anatomical bones.The solution can be formulated as an energy minimization problem for each frame that is obtained for performance capture.
Background
BRIEF DESCRIPTION OF DRAWINGS(1) The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.(2) Illustrative embodimentsof the present invention are described in detail below with reference to the following drawing figures:(3) FIG. 1 illustrates an example of a system for generating an anatomically-constrained local face model, in accordance with an embodiment of the present invention.(4) FIG. 2A-FIG. 2B illustrate an example of an anatomically-constrained local face model, in accordance with an embodiment of the present invention.(5) FIG. 3A-FIG. 3C illustrate an example computation of an anatomical subspace of an anatomically-constrained local face model, in accordance with an embodiment of the present invention.(6) FIG. 4 illustrates anexample of an environment including a system for capturing images of a subject and performing facial performance tracking using motion data, in accordance with an embodiment of the present invention.(7) FIG. 5 illustrates an example of a system for performing facial performance tracking using an anatomically-constrained model, in accordance with an embodiment of the present invention.(8) FIG. 6A-FIG. 6C illustrate examples of results of different patch blending techniques.(9) FIG. 7A illustrates an example of a process of generating an anatomically-constrained model
Claims
1. A computer-implemented method of performingfacial performance tracking of a subject using an anatomically-constrained model of a face of the subject, the method comprising: obtaining the anatomically-constrained model, theanatomically-constrained model including a combination of a local shape subspace and an anatomical subspace, the local shape subspace including deformation shapes for each patch of a plurality of patches representing a geometry of the face, wherein a deformation shape of a patch defines a deformation of the patch for an observed facial expression, and wherein the anatomical subspace includes an anatomical bone structure constraining each of the plurality of patches; obtaining motion data of the face of the subject as the subject conducts a performance; determining, for each patch using the motion data, parameters of the anatomically-constrained model that match a facial expression in the performance, the parameters including rigid local patch motion defining one or more positions of each patch on the face, local patch deformation of each patch defined by a combination of deformation components for each patch, and rigid motion of one or more underlying bones of the anatomicalbone structure relative to each patch; modifying the plurality of patches using the determined parameters to match the plurality of patches to the facial expression in the performance; and combining the deformed plurality of patches into a global face mesh for the face.
10. A system for performing facial performance tracking of a subject using an anatomically-constrained model of a face of the subject, comprising: a memory storing a plurality ofinstructions; and one or more processors configurable to: obtain the anatomically-constrained model, the anatomically-constrained model including a combination of a local shape subspace and an anatomical subspace, the local shape subspace including deformation shapes for each patch of a plurality of patches representing a geometry of the face, wherein a deformation shape of a patch defines a deformation of the patch for an observed facial expression, and wherein the anatomical subspace includes an anatomical bone structure constraining each of the plurality of patches; obtain motion data of the face of the subject as the subject conducts a performance; determine, for each patch using the motion data, parameters of the anatomically-constrained model that match a facial expression in the performance,the parameters including rigid local patch motion defining one or more positions of each patch on the face, local patch deformation of each patch defined by a combination of deformation components for each patch, and rigid motion of one or more underlying bones of the anatomical bone structure relative to each patch; modify the plurality of patches using the determined parameters to match the plurality of patches to the facial expression in the performance; and combine the deformed plurality of patches into a global face mesh for theface.
17. A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising: instructions that cause the one or more processors to obtain an anatomically-constrained model of a face of a subject, the anatomically-constrained model including a combination of a local shape subspace and an anatomical subspace, the local shape subspace including deformation shapes for each patch of a plurality of patches representing a geometry of the face, whereina deformation shape of a patch defines a deformation of the patch for an observed facial expression, and wherein the anatomical subspace includes an anatomical bone structure constraining each of the plurality of patches; instructions that cause the one or more processors to obtain motion data of the face of the subject as the subject conducts a performance; instructions that cause the one or more processors to determine, for each patch using the motion data, parameters of the anatomically-constrained model that match a facial expression in the performance, the parameters including rigid local patch motion defining one ormore positions of each patch on the face, local patch deformation of each patch defined by a combination of deformation components for each patch, and rigid motion of one or more underlying bones of the anatomical bone structure relative to each patch; instructions that cause the one or more processors to modify the plurality of patches using the determinedparameters to match the plurality of patches to the facial expression in the performance;and instructions that cause the one or more processors to combine the deformed plurality of patches into a global face mesh for the face.