Facial animation models

Number: 9747716
Published: 2017-08-29
Filed: 2013-03-15
Assignee: LUCASFILM ENTERTAINMENT COMPANY LTD.
Inventors: Mallet; Ronald, Ye; Yuting, Koperwas; Michael, Goldenthal; Adrian R., Bhat; Kiran S.
CPC: G06T19/20; G06T13/40
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Builds facial animation models from multiple positional representations of a captured face.

Abstract

A system includes a computing device that includes a memory configured to store instructions. The system also includes a processor configured to execute the instructions to perform a method that includes receiving multiple representations of an object. Each of the representations includes position information of the object and corresponds to an instance in time. For at least one of the representations, the method includes defining a contour that represents a movable silhouette of a surface feature of the object. The method also includes producing a deformable model of the surface of the object from the defined contour and from the at least one representation of the object.

Background

DESCRIPTION OF DRAWINGS(1) FIG. 1 illustrates a system for using a modelto animate a character from facial motion information collected from an actor's face.(2) FIGS. 2(a)-(d) illustrate collected information used for producing a model capable of representing facial expressions of an actor.(3) FIGS. 3(a)-(d) illustrate contours of an actor's mouth.(4) FIG. 4 illustrates editing expressions provided by a model of an actor's face.(5) FIGS. 5(a)-(c) illustrates mapping an actor's facial expression to a representation of the actor and a representation of a character.(6) FIG. 6(a)-(c) illustrates adjustments to a characters mouth for tracking an actor's performance.(7) FIG. 7 is a flow chart of operations for producing an animation model.(8) FIG. 8 is a flow chart of operation for adjusting the mouth represented by an animation model.(9) FIG. 9 shows an example of a computing device that can be used in connection with computer-implemented methods and systems described in this document.DETAILED DESCRIPTION(10) Digital characters, both humans and creatures, are becoming prevalent in feature films and video games, often times performing alongside human actors on screen. With realistically rendered images, modelers, animators, editors, etc. strive to deliver believable facial animations. In general, human viewers rely heavily on facial expressions for nonverbal communications. For example, subtle expressions are helpful to decipher emotions; and lip reading often assists viewers in u

Claims

1. A computer-implemented method comprising: receiving, at a processor, multiple images capturing an actor's facial performance at different instances in time; from each of the images, estimating, by the processor, position information indicating positions of a set of features on the actor's face, wherein the set of features include a first curve in a mouth region of the actor; for each of the images, defining,by the processor, based on the position information estimated from the image, a silhouette contour that represents a movable boundary between a visual portion and a hidden portionof the actor's mouth, wherein the silhouette contour is defined such that the silhouette contour corresponds to the first curve; producing, at the processor, one or more deformable models representing a surface of the actor's face at the different instances in time based on the position information estimated from the images; generating, by the processor, one or more corrective shapes based on the silhouette contour and correspondence between thesilhouette contour and the first curve defined in the images; defining separation thresholds for closing the portion of the deformable model; and applying, by the processor, the corrective shapes to the one or more deformable models to adjust the mouth region of the one or more deformable models, wherein applying the corrective shapes to the one or more deformable models to adjust the mouth region of the one or more deformable models includes: adjusting an inner lip shape or a corner of the mouth region of the one or more deformable models of the animated character including closing the portion of the deformable model based on the separation thresholds. 6. A computer-implemented method comprising: receiving multiple images of an actor's mouth for generating a deformable model representing an animated character, the multiple images being captured while a dialogue is being delivered by the actor; from each of the images, estimating, by the processor, position information indicating positions of a set of features on the actor's face, wherein the set of features include a first curve in a mouth region of the actor; for each of the images, defining, based on the position information estimated from the image, a silhouette contour that representsa movable boundary of the actor's mouth, wherein the silhouette contour is defined such that the silhouette contour corresponds to the first curve; defining separation thresholds for closing the portion of the deformable model; based on the silhouette contour, and correspondence between the silhouette contour and the first curve defined in the images, adjusting the geometry of a portion of a deformable model of the animated character to reduce the size of a portion of a mouth of the animated character including closing the portion ofthe deformable model based on the separation thresholds, wherein adjusting the geometry of the portion of the deformable model of the animated character to reduce the size of the portion of the mouth of the animated character includes adjusting an inner lip shape or a corner of the mouth of the deformable model of the animated character; and generating an image of the animated character that presents the adjusted portions of the animated character's mouth. 7. A system comprising: a computing device comprising: a memory configured to store instructions; and a processor configured to execute the instructions to perform a method comprising: receiving, at a processor, multiple images capturing an actor's facial performance at different instances in time; from each of the images, estimating, by the processor, position information indicating positions of a set of features on the actor's face,wherein the set of features include a first curve in a mouth region of the actor; for each of the images, defining, by the processor, based on the position information estimated from the image, a silhouette contour that represents a movable boundary between a visual portion and a hidden portion of the actor's mouth, wherein the silhouette contour is definedsuch that the silhouette contour corresponds to the first curve; producing one or more deformable models representing a surface of the actor's face at the different instances in time based on the position information estimated from the images; generating one or more corrective shapes based on the silhouette contour and correspondence between the silhouette contour and the first curve defined in the images; defining separation thresholds for closing the portion of the deformable model; and applying the corrective shapes to the one or more deformable models to adjust the mouth region of the one or more deformable models based on thresholds, wherein applying the corrective shapes to the one or more deformable models to adjust the mouth region of the one or more deformable models includes: adjusting aninner lip shape or a corner of the mouth region of the one or more deformable models of the animated character including closing the portion of the deformable model based on the separation thresholds. 12. A system comprising: a computing device comprising: a memory configured to store instructions; and a processor configured to execute the instructions to perform a method comprising: receiving multiple images an actor's mouth for generating a deformable model representing an animated character, the multiple images being captured while a dialogue is being delivered by the actor; from each of the images, estimating, by the processor, position information indicating positions of a set of features on the actor's face, wherein the set of features include a first curve in a mouth region of the actor; foreach of the images, defining, based on the position information estimated from the image,a silhouette contour that represents a movable boundary of the actor's mouth, wherein thesilhouette contour is defined such that the silhouette contour corresponds to the first curve; defining separation thresholds for closing the portion of the deformable model; based on the silhouette contour, and correspondence between the silhouette contour and the first curve defined in the images, adjusting the geometry of a portion of a deformable model of the animated character to reduce the size of a portion of a mouth of the animated character including closing the portion of the deformable model based on the separation thresholds, wherein adjusting the geometry of the portion of the deformable model of the animatedcharacter to reduce the size of the portion of the mouth of the animated character includes adjusting an inner lip shape or a corner of the mouth of the deformable model of the animated character; and generating an image of the animated character that presents the adjusted portions of the animated character's mouth. 13. A non-transitory computer program product tangibly embodied in an information carrier and comprising instructions that when executed by a processor perform a method comprising: receiving multiple images capturing an actor's facial performance at different instances in time; from each of the images, estimating position information indicating positions of a set of features on the actor's face, wherein the set of features include a first curve in a mouth region of the actor; for each of the images, defining, based on the position information estimated from the image, a silhouette contour that represents a movable boundary between a visual portion and a hidden portion of the actor's mouth, wherein the silhouette contour is defined such that the silhouette contour corresponds to the first curve; producing one or more deformable models representing a surface of the actor's face at the different instances in time based on the position information estimated from the images; generating one or more corrective shapes basedon the silhouette contour and correspondence between the silhouette contour and the firstcurve defined in the images; defining separation thresholds for closing the portion of the deformable model; and applying the corrective shapes to the one or more deformable models to adjust the mouth region of the one or more deformable models, wherein applying the corrective shapes to the one or more deformable models to adjust the mouth region of the oneor more deformable models includes: adjusting an inner lip shape or a corner of the mouthregion of the one or more deformable models of the animated character including closing the portion of the deformable model based on the separation thresholds. 18. A non-transitory computer program product tangibly embodied in an information carrier and comprising instructions that when executed by a processor perform a method comprising: receiving multipleimages an actor's mouth for generating a deformable model representing an animated character, the multiple images being captured while a dialogue is being delivered by the actor; from each of the images, estimating, by the processor, position information indicating positions of a set of features on the actor's face, wherein the set of features include a first curve in a mouth region of the actor; for each of the images, defining, based on the position information estimated from the image, a silhouette contour that represents a movable boundary of the actor's mouth, wherein the silhouette contour is defined such that the silhouette contour corresponds to the first curve; defining separation thresholds for closing the portion of the deformable model; based on the silhouette contour, and correspondence between the silhouette contour and the first curve defined in the images, adjusting the geometry of a portion of a deformable model of the animated character to reduce the size of a portion of a mouth of the animated character including closing the portion of the deformable model based on the separation thresholds, wherein adjusting the geometry of the portion of the deformable model of the animated character to reduce the size of the portion of the mouth of the animated character includes adjusting an inner lip shape or a corner ofthe mouth of the deformable model of the animated character; and generating an image of the animated character that presents the adjusted portions of the animated character's mouth.