- Number
- 20230237753
- Published
- 2023-07-27
- Filed
- 2023-01-27
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Bradley; Derek Edward et al.
- CPC
- G06T13/40; G06T17/20; G06T19/20; G06T7/251; G06T7/344; G06T7/55; G06V20/653; G06V40/165
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Dynamic facial-hair capture technique (VFX).
Abstract
Embodiments of the present disclosure are directed to methods and systems for generating three-dimensional (3D) models and facial hair models representative of subjects (e.g., actors or actresses) using facial scanning technology. Methods accord to embodiments may be useful for performing facial capture on subjects with dense facial hair. Initial subject facial data, including facial frames and facial performance frames (e.g., images of the subject collected from a capture system) can be used to accurately predict the structure of the subject's face underneath their facial hair to produce a reference 3D facial shape of the subject. Likewise, image processing techniques can be used to identify facial hairs and generate a reference facial hair model. The reference 3D facial shape and reference facial hair mode can subsequently be used to generate performance 3D facial shapes and a performance facial hair model corresponding to a performance by the subject (e.g., reciting dialog).
Background
BACKGROUND
Many contemporary feature films involve both a mix of live action acting and computer generated imagery. Some films entirely comprise computer generated imagery. Such films may including digital characters rendered from three-dimensional (3D) models (sometimes referred to as “3D shapes”). For decades, motion capture of real actors and actresses (or “subjects”) have been used to produce realistic digital character performances. Motion capture and computer generated imagery can be useful for producing scenes that may be difficult or impossible to convincingly film using live actors or practical effects, such as complex action or fantasy sequences.
Motion capture (also referred to as “capture” or “performance capture”) can be performed using images and videos collected from cameras, particularly multi-camera reconstruction systems (more generically, “capture systems”). Such systems can accurately recover a subject's movements digitally, which is particularly popular among filmmakers for “facial performances” (e.g., an actor or actress delivering lines of dialog). Videos or images collected from a capture system can be used to generate 3D models of the subject, which can then be manipulated, edited, and rendered in order to accurately portray the subject and their performance in a film.
There are a variety of useful applications for motion capture and particularly facial performance capture. For example, digitally reconstructed performances can be
Claims
1. A computer-implemented method of generating a reference facial hair model that represents facial hair of a subject, the computer-implemented method comprising performing, by a computer system: retrieving initial subject facial data comprising a plurality of facial frames of the subject, each facial frame comprising one or more facial images of the subject, wherein the plurality of facial frames comprise a reference facial frame and a plurality of non-reference facial frames; for each facial frame of the plurality of facial frames, performing a facial hair identification process, thereby determining a plurality of initial reference facial hair data elements and a plurality of sets of non-reference facial hair data elements, wherein the plurality of initial reference facial hair data elements and the plurality of sets of non-reference facial hair data elements represent facial hair of the subject; for each set of non-reference facial hair data elements, determining a set of projected non-reference facial hair data elements, thereby determining a plurality of sets of projected non-reference facial hair data elements; generating using an optimization solver, for each set of projected non-reference facial hair data elements, a set of alignment transformations, thereby determining a plurality of sets of alignment transformations, wherein the optimization solver is constrained by a facial hair alignment error function relating the set of alignment transformations to the set of projected non-reference facial hair data elements; applying the plurality of sets of alignment transformations to the plurality of sets of non-reference facial hair data elements, thereby determining a plurality of sets of aligned non-reference facial hair data elements; and combining the plurality of sets of aligned non-reference facial hair data elements and the plurality of initial reference facial hair data elements, thereby determining a plurality of reference facial hair data elements that represent facial hair of the subject, wherein the reference facial hair model comprises the plurality of reference facial hair data elements. ||
10. A computer-implemented method of generating a performance facial hair model corresponding to a facial performance by a subject, the computer-implemented method comprising performing, by a computer system: retrieving or generating a reference facial hair model comprising a plurality of reference facial hair data elements; retrieving a plurality of facial performance frames corresponding to the facial performance by the subject, wherein each facial performance frame comprises one or more facial images of the subject; for each facial performance frame of the plurality of facial performance frames, performing an optical flow projection process on the reference facial hair model, thereby determining a set of projected reference facial hair data elements, thereby determining a plurality of sets of projected reference facial hair data elements corresponding to the plurality of facial performance frames; for each facial performance frame of the plurality of facial performance frames, generating using an optimization solver, a set of reference alignment transformations, wherein the optimization solver is constrained by a facial hair performance error function relating the set of reference alignment transformations to a corresponding set of projected reference facial hair data elements, thereby determining a plurality of sets of reference alignment transformations; and applying the plurality of sets of reference alignment transformations to the plurality of reference facial hair data elements, thereby determining a plurality of sets of aligned reference facial hair data elements, wherein the performance facial hair model comprises a plurality of sets of performance facial hair data elements comprising the plurality of sets of aligned reference facial hair data elements.