- Number
- 9652890
- Published
- 2017-05-16
- Filed
- 2015-09-29
- Assignee
- DISNEY ENTERPRISES, INC.
- Inventors
- Beeler; Thabo, Bradley; Derek, Wu; Chenglei
- CPC
- G06T17/20; G06T7/11
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Generates an anatomically-constrained local model with shape and anatomical subspaces to improve performance capture accuracy.
Abstract
Techniques and systems are described for generating an anatomically-constrained local model and for performing performance capture using the model. The local model includes a local shape subspace and an anatomical subspace. In one example, the local shape subspace constrains local deformation of various patches that represent the geometry of a subject's face. In the same example, the anatomical subspace includes an anatomical bone structure, and can be used to constrain movement and deformation of thepatches globally on the subject's face. The anatomically-constrained local face model andperformance capture technique can be used to track three-dimensional faces or other partsof a subject from motion data in a high-quality manner. Local model parameters that best describe the observed motion of the subject's physical deformations (e.g., facial expressions) under the given constraints are estimated through optimization. The optimization can solve for rigid local patch motion, local patch deformation, and the rigid motion of the anatomical bones. The solution can be formulated as an energy minimization problem for eachframe that is obtained for performance capture.
Background
BRIEF DESCRIPTION OF DRAWINGS(1) The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.(2) Illustrativeembodiments of the present invention are described in detail below with reference to the following drawing figures:(3) FIG. 1 illustrates an example of a system for generating an anatomically-constrained local face model, in accordance with an embodiment of the presentinvention.(4) FIG. 2A-FIG. 2B illustrate an example of an anatomically-constrained local face model, in accordance with an embodiment of the present invention.(5) FIG. 3A-FIG. 3C illustrate an example computation of an anatomical subspace of an anatomically-constrainedlocal face model, in accordance with an embodiment of the present invention.(6) FIG. 4 illustrates an example of an environment including a system for capturing images of a subject and performing facial performance tracking using motion data, in accordance with an embodiment of the present invention.(7) FIG. 5 illustrates an example of a system for performing facial performance tracking using an anatomically-constrained model, in accordance withan embodiment of the present invention.(8) FIG. 6A-FIG. 6C illustrate examples of resultsof different patch blending techniques.(9) FIG. 7A illustrates an example of a process ofgenerating an anatomically-constrained model
Claims
1. A computer-implemented method of generating an anatomically-constrained model of a face of a subject, the method comprising: obtaining one or more three-dimensional shapes of the face of the subject; segmenting the face into a plurality of patches; determining a plurality of local shape subspacesfor the plurality of patches, each local shape subspace including local shape deformationof a corresponding patch, wherein a local shape subspace for a patch includes a pluralityof deformation shapes for the patch, and wherein a deformation shape of the patch definesa deformation of the patch for a facial expression; determining an anatomical subspace, the anatomical subspace including tissue thickness constraints for the plurality of patchesand an anatomical bone structure, the anatomical subspace globally constraining local shape deformation and tissue thickness for the plurality of patches, wherein the patch is constrained by the anatomical bone structure using one or more tissue thickness constraints determined for the plurality of deformation shapes of the patch; and generating the anatomically-constrained model of the face by combining the plurality of local shape subspaces and the anatomical subspace.
8. A system for generating an anatomically-constrained model ofa face of a subject, comprising: a memory storing a plurality of instructions; and one ormore processors configurable to: obtain one or more three-dimensional shapes of the face of the subject; segment the face into a plurality of patches; determine a plurality of local shape subspaces for the plurality of patches, each local shape subspace including localshape deformation of a corresponding patch, wherein a local shape subspace for a patch includes a plurality of deformation shapes for the patch, and wherein a deformation shape ofthe patch defines a deformation of the patch for a facial expression; determine an anatomical subspace, the anatomical subspace including tissue thickness constraints for the plurality of patches and an anatomical bone structure, the anatomical subspace globally constraining local shape deformation and tissue thickness for the plurality of patches, wherein the patch is constrained by the anatomical bone structure using one or more tissue thickness constraints determined for the plurality of deformation shapes of the patch; and generate the anatomically-constrained model of the face by combining the plurality of local shape subspaces and the anatomical subspace.
15. A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising: instructions that cause the one or more processors to obtain one ormore three-dimensional shapes of a face of a subject; instructions that cause the one or more processors to segment the face into a plurality of patches; instructions that cause the one or more processors to determine a plurality of local shape subspaces for the plurality of patches, each local shape subspace including local shape deformation of a corresponding patch, wherein a local shape subspace for a patch includes a plurality of deformationshapes for the patch, and wherein a deformation shape of the patch defines a deformation of the patch for a facial expression; instructions that cause the one or more processors to determine an anatomical subspace, the anatomical subspace including tissue thickness constraints for the plurality of patches and an anatomical bone structure, the anatomical subspace globally constraining local shape deformation and tissue thickness for the pluralityof patches, wherein the patch is constrained by the anatomical bone structure using one or more tissue thickness constraints determined for the plurality of deformation shapes of the patch; and instructions that cause the one or more processors to generate the anatomically-constrained model of the face by combining the plurality of local shape subspaces andthe anatomical subspace.
20. A computer-implemented method of generating an anatomically-constrained model of a subject, the method comprising: obtaining one or more three-dimensional shapes of a part of the subject; segmenting the part into a plurality of patches; determining a plurality of local shape subspaces for the plurality of patches, each local subspace including local shape deformation of a corresponding patch, wherein a local shape subspace for a patch includes a plurality of deformation shapes for the patch, and wherein adeformation shape of the patch defines a deformation of the patch for an observed shape of the part; determining an anatomical subspace, the anatomical subspace including materialthickness constraints for the plurality of patches and a rigid structure, the anatomical subspace globally constraining local shape deformation and material thickness for the plurality of patches, wherein the patch is constrained by the rigid structure using one or more material thickness constraints determined for the plurality of deformation shapes of thepatch; and generating the anatomically-constrained model of the part by combining the plurality of local shape subspaces and the anatomical subspace.