- Number
- 12367649
- Published
- 2025-07-22
- Filed
- 2023-01-27
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Winberg; Sebastian et al.
- CPC
- G06T13/40; G06T17/20; G06T19/20; G06T7/251; G06T7/344; G06T7/55; G06V20/653; G06V40/165
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Facial-hair-free mesh generation technique (companion to facial-hair-capture patent).
Abstract
Methods and systems for generating three-dimensional (3D) models and facial hair models representative of subjects (e.g., actors or actresses) using facial scanning technology. Initial subject facial data, including facial frames and facial performance frames (e.g., images of the subject collected from a capture system) can be used to accurately predict the structure of the subject's face underneath their facial hair to produce a reference 3D facial shape of the subject. Likewise, image processing techniques can be used to identify facial hairs and generate a reference facial hair model. The reference 3D facial shape and reference facial hair mode can subsequently be used to generate performance 3D facial shapes and a performance facial hair model corresponding to a performance by the subject (e.g., reciting dialog).
Background
BACKGROUND (1) Many contemporary feature films involve both a mix of live action acting and computer generated imagery. Some films entirely comprise computer generated imagery. Such films may including digital characters rendered from three-dimensional (3D) models (sometimes referred to as “3D shapes”). For decades, motion capture of real actors and actresses (or “subjects”) have been used to produce realistic digital character performances. Motion capture and computer generated imagery can be useful for producing scenes that may be difficult or impossible to convincingly film using live actors or practical effects, such as complex action or fantasy sequences. (2) Motion capture (also referred to as “capture” or “performance capture”) can be performed using images and videos collected from cameras, particularly multi-camera reconstruction systems (more generically, “capture systems”). Such systems can accurately recover a subject's movements digitally, which is particularly popular among filmmakers for “facial performances” (e.g., an actor or actress delivering lines of dialog). Videos or images collected from a capture system can be used to generate 3D models of the subject, which can then be manipulated, edited, and rendered in order to accurately portray the subject and their performance in a film. (3) There are a variety of useful applications for motion capture and particularly facial performance capture. For example, digitally reconstructed performances can be edited by
Claims
1. A computer-implemented method of generating a reference three-dimensional (3D) facial shape corresponding to a subject, the method comprising performing, by a computer system: retrieving initial subject facial data comprising an initial reference 3D facial shape, wherein the initial reference 3D facial shape represents the subject and comprising a plurality of initial reference geometric elements; determining a facial hair mask based on the initial subject facial data, the facial hair mask defining a plurality of probabilities corresponding to the plurality of initial reference geometric elements, wherein each probability of the plurality of probabilities indicates a probability that a corresponding initial geometric element represents facial hair; determining a plurality of reference facial shape weights corresponding to the plurality of initial reference geometric elements using the facial hair mask; generating or retrieving an estimate 3D facial shape; and generating the reference 3D facial shape by combining the estimate 3D facial shape and the initial reference 3D facial shape, wherein the reference 3D facial shape comprises a plurality of reference geometric elements, wherein the reference 3D facial shape represents the subject without facial hair. ||
15. A computer-implemented method of tracking a facial performance by a subject, the computer-implemented method comprising performing, by a computer system: retrieving or generating a reference three-dimensional (3D) facial shape that represents the subject without facial hair, wherein the reference 3D facial shape comprises a plurality of reference geometric elements; retrieving initial subject facial data including a reference facial frame comprising one or more facial images of the subject, each comprising a plurality of reference pixels; retrieving or determining a facial hair mask comprising a plurality of probabilities corresponding to a plurality of facial regions on a face of the subject, each probability indicating the probability that facial hair is located within a corresponding facial region; retrieving a plurality of facial performance frames corresponding to a facial performance by the subject, wherein each facial performance frame comprises one or more facial performance images of the subject each comprising a plurality of performance pixels; determining a set of facial hair free reference pixels from the reference facial frame using the facial hair mask; for each facial performance frame of the plurality of facial performance frames, determining a set of facial hair free performance pixels using the facial hair mask, thereby determining a plurality of sets of facial hair free performance pixels; performing a pixel motion estimate process between the set of facial hair free reference pixels and each set of facial hair free performance pixels, thereby determining a plurality of pixel motion estimates corresponding to the plurality of facial performance frames; determining a plurality of facial shape transformations corresponding to the plurality of facial performance frames, wherein each facial shape transformation comprises a facial hair free transformation component and a facial hair transformation component, wherein the facial hair free transformation component is derived from a corresponding pixel motion estimate of the plurality of pixel motion estimates, wherein the facial hair transformation component comprises a semi-rigid transformation based on the facial hair free transform component; and generating a plurality of performance 3D facial shapes by applying the plurality of facial shape transformations to the reference 3D facial shape, each performance 3D facial shape of the plurality of performance 3D facial shapes corresponding to a facial performance frame of the plurality of facial performance frames, each performance 3D facial shape comprising a plurality of performance geometric elements. ||
20. A computer system comprising: a processor; and a non-transitory computer readable medium coupled to the processor, the non-transitory computer readable medium comprising code, executable by the processor for: retrieving initial subject facial data comprising an initial reference 3D facial shape, wherein the initial reference 3D facial shape represents the subject and comprising a plurality of initial reference geometric elements; determining a facial hair mask based on the initial subject facial data, the facial hair mask defining a plurality of probabilities corresponding to the plurality of initial reference geometric elements, wherein each probability of the plurality of probabilities indicates a probability that a corresponding initial geometric element represents facial hair; determining a plurality of reference facial shape weights corresponding to the plurality of initial reference geometric elements using the facial hair mask; generating or retrieving an estimate 3D facial shape; and generating the reference 3D facial shape by combining the estimate 3D facial shape and the initial reference 3D facial shape, wherein the reference 3D facial shape comprises a plurality of reference geometric elements, wherein the reference 3D facial shape represents the subject without facial hair.