Eye contact sensing and control for robotic characters

Number: 11072075
Published: 2021-07-27
Filed: 2019-10-24
Assignee: Disney Enterprises, Inc.
Inventors: Christensen; David Loyal, Wieland; Alexis P., Smoot; Lanny S., Ayala; Alfredo M., Smithwick; Quinn Y., McIntosh; Kyna P., Kennedy; James R., Pan; Matthew K., Niemeyer; Günter D., Campos Zamora; Daniel De Jesus, Choi; Sungjoon
CPC: B25J9/1697; B25J11/0015; G06F3/013; G06V10/147; G06V20/10; G06V40/19; G06V40/166
Verdict: High Hardware
Source: Google Patents · FreePatentsOnline

The keeper's note

Eye-contact sensing/control for a robotic/animatronic character (granted).

Abstract

A system for sensing and controlling eye contact for a robot. The system includes a robotic figure with a movable eye. The system includes a light source positioned in the robotic figure to output light through a light outlet of the eye. A light sensor is included that senses light striking surfaces in a physical space in which the robotic figure is positioned including the output light from the light source. The system includes an image processor processing output of the light sensor to identify a location of a target formed by the output light striking surfaces in the physical space and to identify a location of a face of a human. Further, the system includes a robot controller generating eye movement control signals based on the location of the target and the location of the face to position the eye to provide eye contact with the human observer.

Background

BACKGROUND 1. Field of the Description (1) The present description relates, in general, to design and control of robots, robotic figures or characters, and/or animatronics that include eyes such as, but not limited to, human-like robotic characters. More particularly, the description relates to a new assembly and method for sensing and for controlling a robotic figure or character (i.e., any robot or robotic system/device with one or more eyes) to achieve desirable eye contact between the robotic figure or character and a nearby person who may be interacting with (e.g., speaking with) the robotic figure or character. The description also relates to new robotic figure/character systems that make use of the new eye contact sensing and control taught herein. 2. Relevant Background (2) Robots that interact with humans are becoming increasingly popular in numerous and varied applications including the service and entertainment industries. As an example, many entertainment settings utilize robotic devices or animatronics to provide physical embodiments of characters including characters known to the people viewing and interacting with the robotic devices or animatronics. These characters may be from comic books, movies, or television shows such that people may have a deep connection with the characters and have specific expectations of how a robotic device implementing that character should move and behave in the physical world. (3) In this regard, it has proven difficult to design

Claims

1. A system for sensing and controlling eye contact with a robot, comprising: a robotic figure with a movable eye with a front light outlet; a light source positioned in the robotic figure to output light through the front light outlet of the movable eye; a light sensor sensing light striking surfaces in a physical space in which the robotic figure is positioned including the output light from the light source; an image processor processing output of the light sensor to identify a location of a target formed by the output light striking one or more surfaces in the physical space and to identify a location of a face of a human in the physical space; and a robot controller generating eye movement control signals for the robotic figure based on the location of the target and the location of the face to position the movable eye to provide eye contact with the human, wherein the light source is an infrared (IR) light source and the light sensor comprises an IR camera, and wherein the IR light source is mounted within an inner space of the movable eye to move with the movable eye or is mounted external to the movable eye and directs light onto a mirror positioned in the inner space. || 9. A system for sensing and controlling eye contact with a robot, comprising: a robot with a head including first and second eyes positionable with eye movement drivers; a light source positioned in the first eye of the robot and mounted for moving with the first eye and for outputting light from the first eye along a center axis of the first eye; a first camera capturing an image of a space in field of view of the robot based on the light from the first eye; a second camera capturing an image of the space based on visible light from a source external to the robot, wherein a human observer is positioned in the space; an image processor processing the images from the first and second cameras to locate a targeted area in the space illuminated by the light from the first eye and a face of the human observer; and a controller running an eye movement module generating control signals for the eye movement drivers to cause the targeted area illuminated by the light from the first eye to move at least partially onto the face of the human observer, wherein the light source is an infrared (IR) light source and the first camera comprises an IR camera or the light source is a laser and the first camera comprises a camera configured to detect light from the laser. || 13. An eye contact method for controlling a robot, comprising: operating a light source in a robotic figure to output light from a movable eye, wherein the light source is mounted within an inner space of the movable eye to move with the moveable eye, and is an infrared (IR) source or a laser; sensing illumination of a target area by the output light in a physical scene; detecting a location of an object in the physical scene, wherein the detecting is performed by an IR camera when the light source is an IR source and by a camera configured to detect light from the laser when the light source is a laser; determining the robot is to attend to the object; and moving the movable eye to move the target area to the location of the object.