- Number
- 11836860
- Published
- 2023-12-05
- Filed
- 2022-01-27
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Chandran; Prashanth et al.
- CPC
- G06T13/40; G06T17/20; G06T7/10
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Facial-animation retargeting technique via blend-shape solver (granted).
Abstract
Methods and systems for performing facial retargeting using a patch-based technique are disclosed. One or more three-dimensional (3D) representations of a source character's (e.g., a human actor's) face can be transferred onto one or more corresponding representations of a target character's (e.g., a cartoon character's) face, enabling filmmakers to transfer a performance by a source character to a target character. The source character's 3D facial shape can separated into patches. For each patch, a patch combination (representing that patch as a combination of source reference patches) can be determined. The patch combinations and target reference patches can then be used to create target patches corresponding to the target character. The target patches can be combined using an anatomical local model solver to produce a 3D facial shape corresponding to the target character, effectively transferring a facial performance by the source character to the target character.
Background
BACKGROUND OF THE INVENTION (1) In modern filmmaking, performances by actors and actresses (human, animal, or otherwise) are sometimes used to generate animated performances by computer-generated characters. Often, filmmakers want the computer-generated character's performance to closely match the performance of the human actor. For example, if the human actor expresses a particular emotion, such as joy, the filmmaker may want the computer-generated character to express that same emotion. The process of mapping the performance of a source character (e.g., the actor) to a target character (e.g., the computer-generated character) may be referred to as “animation retargeting.” The process of specifically mapping a facial performance (e.g., the formation of expressions) from a source character to a target character may be referred to as “facial animation retargeting.” Although uncommon, in some cases the source character may not be a living actor or actress, and may also be a computer-generated character. (2) It can sometimes be difficult to perform facial animation retargeting, particularly when there are large differences between source characters and target characters. A source character can have physical features (such as wrinkles or jowls) that a target character does not possess, and vice versa. These physical features and their movement can inadvertently be transferred during animation retargeting, which can result in a visually-displeasing animated performance by the targ
Claims
1. A computer-implemented method of performing facial animation retargeting from a source character to a target character, the method comprising: generating a plurality of source patches corresponding to a three-dimensional (3D) source facial shape and the source character, each source patch of the plurality of source patches comprising geometric elements corresponding to the 3D source facial shape; generating a patch combination for each source patch of the plurality of source patches, thereby generating a plurality of patch combinations corresponding to the plurality of source patches, wherein each patch combination is generated using a patch solver and a plurality of source reference patches, the plurality of source reference patches corresponding to a plurality of 3D source facial reference shapes; generating a plurality of target patches using the plurality of patch combinations corresponding to the plurality of source patches and a plurality of target reference patches, the plurality of target reference patches corresponding to a plurality of 3D target facial reference shapes and the target character, wherein each target reference patch of the plurality of target reference patches corresponds to a corresponding source reference patch of the plurality of source reference patches; and generating a 3D target facial shape using an anatomical model solver and the plurality of target patches, wherein the anatomical model solver generates the 3D target facial shape by determining anatomical parameters corresponding to an anatomically-constrained model of the target character, and wherein the anatomically-constrained model comprises a combination of a plurality of local shape subspaces and an anatomical subspace. ||
20. A computer-implemented method of transferring a facial performance from a subject to a computer-generated character, the method comprising: receiving one or more frames representing at least a portion of the facial performance by the subject, wherein each frame of the one or more frames includes a three-dimensional (3D) subject facial shape corresponding to a facial expression of the subject during the facial performance; and (a) for each frame of the one or more frames: removing rigid motion of the 3D subject facial shape; segmenting the 3D subject facial shape into a plurality of subject performance patches with overlapping boundaries using a subject patch layout comprising a plurality of subject patches having overlapping boundaries; for each subject performance patch of the plurality of subject performance patches: representing a deformation of the subject performance patch using a corresponding subject patch model and a non-linear patch solver, wherein the corresponding subject patch model is a subject patch model of a plurality of subject patch models, wherein each subject patch model defines a subject patch as a linear combination of corresponding patches in a plurality of 3D subject facial reference shapes; generating a corresponding character performance patch using the deformation of the subject performance patch and a corresponding character patch model, thereby generating a plurality of character performance patches, wherein the corresponding character patch model is a character patch model of a plurality of character patch models, wherein each character patch model defines a character patch as a linear combination of corresponding patches in a plurality of 3D character facial reference shapes, wherein the plurality of 3D character facial reference shapes are defined by a character patch layout, wherein the character patch layout comprises a plurality of character patches having overlapping boundaries, and wherein each character patch of the plurality of character patches is mapped to a corresponding subject patch of the plurality of subject patches; and generating a 3D character facial shape using the plurality of character performance patches and an anatomically-constrained model of the computer-generated character, thereby generating one or more 3D character facial shapes corresponding to the one or more frames. ||
22. A system for performing facial animation retargeting, the system comprising: one or more processors; and a non-transitory computer readable medium coupled to the one or more processors, the non-transitory computer readable medium comprising instructions executable by the processor to cause the system to: generate a plurality of source patches corresponding to a three-dimensional (3D) source facial shape corresponding to a source character, each source patch of the plurality of source patches comprising geometric elements corresponding to the 3D source facial shape; generate a patch combination for each source patch of the plurality of source patches, thereby generating a plurality of patch combinations corresponding to the plurality of source patches, wherein each patch combination is generated using a patch solver and a plurality of source reference patches, the plurality of source reference patches corresponding to a plurality of 3D source facial reference shapes; generate a plurality of target patches using the plurality of patch combinations corresponding to the plurality of source patches and a plurality of target reference patches, the plurality of target reference patches corresponding to a plurality of 3D target facial reference shapes and a target character, wherein each target reference patch of the plurality of target reference patches corresponds to a corresponding source reference patch of the plurality of source reference patches; and generate a 3D target facial shape using an anatomical model solver and the plurality of target patches, wherein the anatomical model solver generates the 3D target facial shape by determining anatomical parameters corresponding to an anatomically-constrained model of the target character, and wherein the anatomically-constrained model comprises a combination of a plurality of local shape subspaces and an anatomical subspace.