Motion tracking and image recognition of hand gestures to animate a digital puppet, synchronized with recorded audio

Number: 11049309
Published: 2021-06-29
Filed: 2013-12-06
Assignee: Disney Enterprises, Inc.
Inventors: Murdock; Malcolm, Hy; Michael, Mkrtchyan; Armen
CPC: G06F3/011; G06F3/0304; G06T13/40
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Hand-gesture motion tracking driving digital puppet/character animation.

Abstract

There is provided a system and method for creating a digital puppet show by animating a digital puppet using gestures and sound. The method includes presenting the digital puppet to a user on a display. The method further includes receiving motion data corresponding to a gesture, using a camera, from the user, translating the motion data into digital data using a motion tracking algorithm, and animating the digital puppet, on the display, using the digital data. The method can further include receiving audio data from the user using a microphone and playing the audio data, using a speaker, while animating the digital puppet on the display to create the digital puppet show.

Background

BACKGROUND (1) Real life puppet shows are a popular source of entertainment for children and can be performed by placing a sock over a person's hand to impersonate a character. Puppet shows give children the ability to dress socks up to look like their favorite characters, and using their fingers and thumb, children are able to move the mouth of the sock character while giving a narrative in order to create the puppet show. However, this form of entertainment has started to become less popular for children, who are now looking to the digital world for new forms of entertainment. (2) Interactive digital storybook applications for phones and computers are also very popular with children and allow children to create puppet shows using their favorite characters. Children are able to create and record puppet shows by moving 2-D cutouts of their favorite characters around a scene, while at the same time narrating the characters to give the characters a voice. When a child is finished recording the puppet show, the child can playback the puppet show to entertain friends and family. This gives children the experience of creating their very own animated story. However, these interactive digital storybook applications can be very limited by their simplicity in that they only allow children to move 2-D characters around a scene while narrating the puppet show. As such, any gestures made by the children, like sock puppets in the physical space, have no impact on the animation of the char

Claims

1. A system for animating a three-dimensional (3-D) digital puppet, the system comprising: a display; a microphone; a speaker; a camera; and a processor configured to: present the 3-D digital puppet to a user on the display; receive, using the camera, motion data corresponding to a gesture made by the user; receive, using the microphone, audio data corresponding to voice of the user for use in a 3-D digital puppet show; translate the motion data into digital data using a motion tracking algorithm, the digital data including at least (X,Y,Z) movement data; create the 3-D digital puppet show using the 3-D digital puppet by: animating the 3-D digital puppet, on the display, using the digital data; recording the audio data in real time and along with the animating of the 3-D digital puppet; searching the audio data for a command; playing, using the speaker, the audio data at the same time as the animating of the 3-D digital puppet on the display; and animating facial expressions of the 3-D digital puppet using the command at the same time as the animating the 3-D digital puppet on the display using the digital data translated from the motion data; wherein the audio data includes a plurality of time marks, and wherein playing the audio data includes synchronizing, using the plurality of time marks, the audio data with the animating of the 3-D digital puppet. || 10. A method for animating a 3-D digital puppet on a device including a processor, a display, a microphone, a speaker and a camera, the method comprising: presenting the 3-D digital puppet to a user on the display; receiving, using the camera, motion data corresponding to a gesture made by the user; receiving, using the microphone, audio data corresponding to voice of the user for use in a 3-D digital puppet show; translating the motion data into digital data using a motion tracking algorithm, the digital data including at least (X,Y,Z) movement data; creating a 3-D digital puppet show using the 3-D digital puppet by: animating the 3-D digital puppet, on the display, using the digital data; recording the audio data in real time and along with the animating of the 3-D digital puppet; searching the audio data for a command; playing, using the speaker, the audio data at the same time as the animating of the 3-D digital puppet on the display; and animating facial expressions of the 3-D digital puppet using the command at the same time as the animating the 3-D digital puppet on the display using the digital data translated from the motion data; wherein the audio data includes a plurality of time marks, and wherein playing the audio data includes synchronizing, using the plurality of time marks, the audio data with the animating of the 3-D digital puppet.