Projecting augmentation images onto moving objects

Number: 10380802
Published: 2019-08-13
Filed: 2018-02-27
Assignee: Disney Enterprises, Inc.
Inventors: Grundhöfer; Anselm, Bermano; Amit
CPC: G06T11/00; G06T19/006; G06T7/33; G06T7/50; G06T7/73; G06V40/174; H04N23/20
Verdict: Medium Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Projecting augmentation images onto moving objects (projection mapping+tracking).

Abstract

A method of augmenting a human face with projected light is disclosed. The method includes determining a blend of component attributes to define visual characteristics of the human face, modifying an input image based, at least in part, on an image of the human face, wherein the modified input image defines an augmented visual characteristic of the human face, determining a present location of one or more landmarks on the human face based, at least in part, on the image of the human face, predicting a future location of the one or more landmarks, deforming a model of the human face based on the future location of the one or more landmarks, generating an augmentation image based on the deformed model and the modified input image, and transmitting for projection the augmentation image.

Background

BACKGROUND(1) Manipulating the appearance of surfaces with high frequency motion or rapid deformation is desirable in many fields such as the visual and performing arts. For example, manipulating the appearance of human faces is a common practice in many cultures and industries. Traditionally, appearances are altered through the application of paint, die, makeup, prosthetics, etc. Such alterations are static, and manual labor is typically required to apply and change the alteration. Projected light has also been used to alter the appearance of static, stationary surfaces, such as movie screens. However, realistic real-time projection onto moving or changing surfaces has proved difficult. Slow processing speeds and inefficient methods of calculation and projection have historically prevented realistic real-time projection onto moving or changing surfaces. To allow performers to move at the their natural pace, more efficient methods of computing projectable images to augment a moving surface in real time without perceptible lag are needed.SUMMARY(2) An example of the present disclosure relates to a method of augmenting a target object with projected light. The method includes determining, by a processor, a blend of component attributes to define visual characteristics of the target object, modifying, by the processor, an input image based, at least in part, on an image of the target object, wherein the modified input image defines an augmented visual characteristic of the targe

Claims

1. A method to project images onto a human face comprising: capturing by a camera a first orientation image of infrared light reflecting off of at least a portion of the human face in an initial position; detecting by a processing element a plurality of landmarks of the human face in the initial position in the first orientation image; predicting by the processing element a future location of the plurality of landmarks of the human face at a future instance in time; deforming by the processing element a mesh corresponding to the human face based on the predicted future location of the plurality of landmarks; generating by the processing element an output image based on the deformed mesh and an input image; and projecting by a projector in communication with the processing element the output image onto the human face in a future position. 8. A projection system for augmenting an appearance of a moving actor comprising: a lighting source that emits non-visible wavelengths of light onto the moving actor; a camera that captures an orientation image corresponding to reflection of the non-visible wavelengths off of the moving actor; a computer in electrical communication with the camera, wherein the computer performs the following operations: determine a blend of component attributes to define visual characteristics of the moving actor; modify an input image based on the orientation image to generate a modified input image; determine a present location of a plurality of facial landmarks on the actor based on the orientation image; predict a future location of the plurality of facial landmarks; deform a mesh model corresponding to the actor based on the predicted future location of the landmarks; overlay the modified input image over the deformed mesh model to generate an output image; and a projector in electrical communication with the computer, wherein the projector emits visible wavelengths of light corresponding to the output image onto the moving actor. 15. A visual augmentation system comprising: a camera; a projector; and a processor electrically coupled to the camera and the projector, the processor configured to: receive, from the camera, an image of a human face based on reflected infrared light; determine a weighted blend of component attributes to approximate the human face based on the received image; modify a texture based on the weighted blend of component attributes; detect a location of at least one facial feature the image of the human face; predict a future location of the at least one facial feature; generate a deformed model of the human face based on the future location of the at least one facial feature; apply the texture to the deformed model to generate a projectable image; and transmit for projection by the projector, the projectable image.