Dynamic facial hair capture of a subject

Number: 12361663
Published: 2025-07-15
Filed: 2023-01-27
Assignee: Disney Enterprises, Inc.
Inventors: Bradley; Derek Edward et al.
CPC: G06T13/40; G06T17/20; G06T19/20; G06T7/251; G06T7/344; G06T7/55; G06V20/653; G06V40/165
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Dynamic facial-hair capture technique for character rendering.

Abstract

Embodiments of the present disclosure are directed to methods and systems for generating three-dimensional (3D) models and facial hair models representative of subjects (e.g., actors or actresses) using facial scanning technology. Methods accord to embodiments may be useful for performing facial capture on subjects with dense facial hair. Initial subject facial data, including facial frames and facial performance frames (e.g., images of the subject collected from a capture system) can be used to accurately predict the structure of the subject's face underneath their facial hair to produce a reference 3D facial shape of the subject. Likewise, image processing techniques can be used to identify facial hairs and generate a reference facial hair model. The reference 3D facial shape and reference facial hair mode can subsequently be used to generate performance 3D facial shapes and a performance facial hair model corresponding to a performance by the subject (e.g., reciting dialog).

Background

(1) U.S. patent application Ser. No. 18/102,480 entitled “Generating a Facial-Hair-Free Mesh of a Subject,” is being filed concurrently, and the entire disclosure of which is hereby incorporated by reference into this application in its entirety for all purposes. BACKGROUND (2) Many contemporary feature films involve both a mix of live action acting and computer generated imagery. Some films entirely comprise computer generated imagery. Such films may including digital characters rendered from three-dimensional (3D) models (sometimes referred to as “3D shapes”). For decades, motion capture of real actors and actresses (or “subjects”) have been used to produce realistic digital character performances. Motion capture and computer generated imagery can be useful for producing scenes that may be difficult or impossible to convincingly film using live actors or practical effects, such as complex action or fantasy sequences. (3) Motion capture (also referred to as “capture” or “performance capture”) can be performed using images and videos collected from cameras, particularly multi-camera reconstruction systems (more generically, “capture systems”). Such systems can accurately recover a subject's movements digitally, which is particularly popular among filmmakers for “facial performances” (e.g., an actor or actress delivering lines of dialog). Videos or images collected from a capture system can be used to generate 3D models of the subject, which can then be manipulated, edited, an

Claims

1. A computer-implemented method of generating a reference facial hair model that represents facial hair of a subject, the computer-implemented method comprising performing, by a computer system: retrieving initial subject facial data comprising a plurality of facial frames of the subject, each facial frame comprising one or more facial images of the subject, wherein the plurality of facial frames comprise a reference facial frame and a plurality of non-reference facial frames; for each facial frame of the plurality of facial frames, performing a facial hair identification process, thereby determining a plurality of initial reference facial hair data elements and a plurality of sets of non-reference facial hair data elements, wherein the plurality of initial reference facial hair data elements and the plurality of sets of non-reference facial hair data elements represent facial hair of the subject; for each set of non-reference facial hair data elements, determining a set of projected non-reference facial hair data elements, thereby determining a plurality of sets of projected non-reference facial hair data elements; generating using an optimization solver, for each set of projected non-reference facial hair data elements, a set of alignment transformations, thereby determining a plurality of sets of alignment transformations, wherein the optimization solver is constrained by a facial hair alignment error function relating the set of alignment transformations to the set of projected non-reference facial hair data elements; applying the plurality of sets of alignment transformations to the plurality of sets of non-reference facial hair data elements, thereby determining a plurality of sets of aligned non-reference facial hair data elements; combining the plurality of sets of aligned non-reference facial hair data elements and the plurality of initial reference facial hair data elements, thereby determining a plurality of reference facial hair data elements, wherein the reference facial hair model comprises the plurality of reference facial hair data elements, wherein the plurality of reference facial hair data elements comprises a plurality of facial hair sets of reference facial hair data elements, each facial hair set comprising a three-dimensional representation of a different facial hair of the subject; retrieving a plurality of facial performance frames corresponding to a facial performance by the subject, wherein each facial performance frame comprises one or more facial images of the subject; for each facial performance frame of the plurality of facial performance frames, performing an optical flow projection process on the reference facial hair model, thereby determining a set of projected reference facial hair data elements, thereby determining a plurality of sets of performance facial hair data elements corresponding to the plurality of facial frames; for each facial performance frame of the plurality of facial performance frames, determining a set of projected reference facial hair data elements, thereby determining a plurality of sets of projected reference facial hair data elements corresponding to the plurality of facial performance frames; for each facial performance frame of the plurality of facial performance frames, generating using an optimization solver, a set of reference alignment transformations, wherein the optimization solver is constrained by a facial hair performance error function relating the set of reference alignment transformations to a corresponding set of projected reference facial hair data elements, thereby determining a plurality of sets of reference alignment transformations; and applying the plurality of sets of reference alignment transformations to the plurality of reference facial hair data elements, thereby determining a plurality of sets of aligned reference facial hair data elements, wherein a performance facial hair model comprises a plurality of sets of performance facial hair data elements comprising the plurality of sets of aligned reference facial hair data elements; retrieving or generating a reference 3D facial shape, wherein the reference 3D facial shape represents a face of the subject, wherein the reference 3D facial shape comprises a plurality of reference geometric elements; retrieving or determining a facial hair mask comprising a plurality of probabilities corresponding to a plurality of facial regions on a face of the subject, each probability indicating the probability that facial hair is located within a corresponding facial region; retrieving or generating a plurality of performance 3D facial shapes that represent the face of the subject during the facial performance, each performance 3D facial shape comprising a plurality of performance geometric elements; determining a set of reference regions corresponding to the reference 3D facial shape, each reference region of the set of reference regions corresponding to a plurality of reference region geometric elements from the plurality of reference geometric elements; determining a plurality of sets of performance regions corresponding to the plurality of performance 3D facial shapes, each performance region of the plurality of sets of performance regions corresponding to a plurality of performance region geometric elements from the plurality of performance geometric elements; generating a plurality of performance refinement weights using the facial hair mask; determining, based on the set of reference regions, a set of subsets of reference facial hair data elements from the plurality of reference facial hair data elements from the reference facial hair model, wherein each subset of reference facial hair data elements corresponds to a reference region of the set of reference regions; determining, based on the plurality of sets of performance regions, a plurality of sets of subsets of performance facial hair data elements from the plurality of sets of performance facial hair data elements from the performance facial hair model, wherein each subset of performance facial hair data elements corresponds to a performance region of the plurality of sets of performance regions; for each subset of performance facial hair data elements of the plurality of sets of subsets of performance facial hair data elements, determining a facial hair transformation between that subset of performance facial hair data elements and a corresponding subset of reference facial hair data elements, thereby determining a plurality of facial hair transformations; applying the plurality of facial hair transformations to the plurality of reference regional geometric elements, thereby determining a plurality of transformed reference regional geometric elements; and for each facial performance frame of the plurality of facial performance frames, generating a refined performance 3D facial shape using an optimization solver, wherein the optimization solver is constrained based on a refined performance error function relating the plurality of transformed reference regional geometric elements, the plurality of performance geometric elements corresponding to that facial frame, and the plurality of performance refinement weights, the computer system thereby generating a plurality of refined performance 3D facial shapes. || 8. A computer system comprising: a processor; and a non-transitory computer readable medium coupled to the process, the non-transitory computer readable medium comprising code, executable by the processor for implementing the computer-implemented method of claim || 1. || 9. A computer-implemented method of generating a performance facial hair model corresponding to a facial performance by a subject, the computer-implemented method comprising performing, by a computer system: retrieving or generating a reference facial hair model comprising a plurality of reference facial hair data elements, wherein the plurality of reference facial hair data elements comprises a plurality of facial hair sets of reference facial hair data elements, each facial hair set comprising a three-dimensional representation of a different facial hair of the subject; retrieving a plurality of facial performance frames corresponding to the facial performance by the subject, wherein each facial performance frame comprises one or more facial images of the subject; for each facial performance frame of the plurality of facial performance frames, performing an optical flow projection process on the reference facial hair model, thereby determining a set of projected reference facial hair data elements, thereby determining a plurality of sets of projected reference facial hair data elements corresponding to the plurality of facial performance frames; for each facial performance frame of the plurality of facial performance frames, generating using an optimization solver, a set of reference alignment transformations, wherein the optimization solver is constrained by a facial hair performance error function relating the set of reference alignment transformations to a corresponding set of projected reference facial hair data elements, thereby determining a plurality of sets of reference alignment transformations; applying the plurality of sets of reference alignment transformations to the plurality of reference facial hair data elements, thereby determining a plurality of sets of aligned reference facial hair data elements, wherein the performance facial hair model comprises a plurality of sets of performance facial hair data elements comprising the plurality of sets of aligned reference facial hair data elements; retrieving or generating a reference 3D facial shape, wherein the reference 3D facial shape represents a face of the subject, wherein the reference 3D facial shape comprises a plurality of reference geometric elements; retrieving or determining a facial hair mask comprising a plurality of probabilities corresponding to a plurality of facial regions on a face of the subject, each probability indicating the probability that facial hair is located with a corresponding facial region; retrieving or generating a plurality of performance 3D facial shapes that represent a face of the subject during a facial performance, each performance 3D facial shape comprising a plurality of performance geometric elements; determining a set of reference regions corresponding to the reference 3D facial shape, each reference region of the set of reference regions corresponding to a plurality of reference region geometric elements from the plurality of reference geometric elements; determining a plurality of sets of performance regions corresponding to the plurality of performance 3D facial shapes, each performance region of the plurality of sets of performance regions corresponding to a plurality of performance region geometric elements from the plurality of performance geometric elements; generating a plurality of performance refinement weights using the facial hair mask; determining, based on the set of reference regions, a set of subsets of reference facial hair data elements from the plurality of reference facial hair data elements from the reference facial hair model, wherein each subset of reference facial hair data elements corresponds to a reference region of the set of reference regions; determining, based on the plurality of sets of performance regions, a plurality of sets of subsets of performance facial hair data elements from the plurality of sets of performance facial hair data elements from the performance facial hair model, wherein each subset of performance facial hair data elements corresponds to a performance region of the plurality of sets of performance regions; for each subset of performance facial hair data elements of the plurality of sets of subsets of performance facial hair data elements, determining a facial hair transformation between that subset of performance facial hair data elements and a corresponding subset of reference facial hair data elements, thereby determining a plurality of facial hair transformations; applying the plurality of facial hair transformations to the plurality of reference regional geometric elements, thereby determining a plurality of transformed reference regional geometric elements; and for each facial performance frame of the plurality of facial performance frames, generating a refined performance 3D facial shape using an optimization solver, wherein the optimization solver is constrained based on a refined performance error function relating the plurality of transformed reference regional geometric elements, the plurality of performance geometric elements corresponding to that facial frame, and the plurality of performance refinement weights, the computer system thereby generating a plurality of refined performance 3D facial shapes.