Obtaining high resolution and dense reconstruction of face from sparse facial markers

Number: 11783493
Published: 2023-10-10
Filed: 2021-11-05
Assignee: Lucasfilm Entertainment Company Ltd. LLC
Inventors: Cong; Matthew et al.
CPC: G06F18/22; G06T17/20; G06T5/50; G06T7/246; G06T7/55; G06T7/75; G06V10/761; G06V10/774; G06V20/647; G06V40/171; G06V40/172
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Dense facial reconstruction from sparse facial markers (VFX, continuation).

Abstract

Some implementations of the disclosure are directed to techniques for facial reconstruction from a sparse set of facial markers. In one implementation, a method comprises: obtaining data comprising a captured facial performance of a subject with a plurality of facial markers; determining a three-dimensional (3D) bundle corresponding to each of the plurality of facial markers of the captured facial performance; using at least the determined 3D bundles to retrieve, from a facial dataset comprising a plurality of facial shapes of the subject, a local geometric shape corresponding to each of the plurality of the facial markers; and merging the retrieved local geometric shapes to create a facial reconstruction of the subject for the captured facial performance.

Background

BRIEF SUMMARY OF THE DISCLOSURE (1) Implementations of the disclosure describe improved techniques for facial reconstruction from a sparse set of facial markers. In one embodiment, a method comprises: obtaining data comprising a captured facial performance of a subject with a plurality of facial markers; determining a three-dimensional (3D) bundle corresponding to each of the plurality of facial markers of the captured facial performance; using at least the determined 3D bundles to retrieve, from a facial dataset comprising a plurality of facial shapes of the subject, a local geometric shape corresponding to each of the plurality of the facial markers; and merging the retrieved local geometric shapes to create a facial reconstruction of the subject for the captured facial performance. (2) In some implementations, retrieving the local geometric shape corresponding to each of the plurality of the facial markers, comprises: evaluating a surface position of each of the 3D bundles on each of the plurality of facial shapes in the facial dataset to derive a point cloud corresponding to each of the 3D bundles; creating a tetrahedral mesh from each of the point clouds; and using each of the created tetrahedral meshes to retrieve, from the facial dataset, the local geometric shape corresponding to each of the facial markers. (3) In some implementations, the created tetrahedral mesh is a non-manifold tetrahedral mesh. (4) In some implementations, creating the tetrahedral mesh from each

Claims

1. A method, comprising: obtaining data comprising a captured facial performance of a subject with a plurality of facial markers; determining a three-dimensional (3D) bundle corresponding to each of the plurality of facial markers of the captured facial performance; retrieving, for each 3D bundle, a local surface geometry corresponding to the 3D bundle by: deriving a point cloud corresponding to the 3D bundle by evaluating a surface position of the 3D bundle on multiple facial shapes of the subject in a facial dataset; creating a tetrahedral mesh from the point cloud; and using the created tetrahedral mesh to retrieve, using the facial dataset, the local surface geometry corresponding to the 3D bundle; and merging the retrieved local surface geometries to create a facial reconstruction of the subject for the captured facial performance. || 12. A non-transitory computer-readable medium having executable instructions stored thereon that, when executed by a processor, cause a system to perform operations comprising: obtaining data comprising a captured facial performance of a subject with a plurality of facial markers; determining a three-dimensional (3D) bundle corresponding to each of the plurality of facial markers of the captured facial performance; retrieving, for each 3D bundle, a local surface geometry corresponding to the 3D bundle by: deriving a point cloud corresponding to the 3D bundle by evaluating a surface position of the 3D bundle on multiple facial shapes of the subject in a facial dataset; creating a tetrahedral mesh from the point cloud; and using the created tetrahedral mesh to retrieve, using the facial dataset, the local surface geometry corresponding to the 3D bundle; and merging the retrieved local surface geometries to create a facial reconstruction of the subject for the captured facial performance.