MULTISENSORY AUGMENTED REALITY

Number: 20180315243
Published: 2018-11-01
Filed: 2017-04-26
Assignee: Disney Enterprises, Inc.
Inventors: Mahler; Moshe B.; Mars; John A.; McCann; James L.; Israr; Ali; Lawson; Shawn; McIntosh; Kyna P.; Bedford; Job T.
CPC: G06T19/006; G06F3/011; G06F3/016; G06F3/017; G06T11/60; G06T19/20
Verdict: Medium Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Multisensory AR experience.

Abstract

According to one aspect of the present disclosure, a method for providing a multisensory augmented reality environment is disclosed. The method includes receiving a real-time video stream of a physical environment, receiving depth information of one or more objects within the physical environment, generating real-time content depicting the physical environment based on the real-time video stream and the depth information, rendering augmented content, wherein the augmented content integrates original content and the real-time content, generating sensory feedback instructions to provide sensory feedback based on the augmented content, wherein the sensory feedback instructions are dynamically adjusted to changes in the augmented content, transmitting the augmented content for display and instructions to provide sensory feedback correlated to the augmented content.

Background

BACKGROUND

Mixed reality (MR) and augmented reality (AR) create exciting opportunities to engage users in immersive experiences, resulting in natural human-computer interaction. Many MR interactions are generated around a first-person point of view (POV). In these cases, the user directs to the environment, which is digitally displayed either through a head-mounted display or a handheld computing device. However, such conventional AR/MR platforms provide an experience that is limited to a single participant's point of view for each piece of equipment, precluding other participants from also interacting without also having their own piece of expensive equipment. In other words, only one person can participate in the experience at a time. In such systems, a user wears an expensive head-mounted display that provides the user with a first-person perspective of a virtual environment. The device uses expensive components, such as accelerometers, to detect the movement of the user's head and adjust the displayed scene based on the detected movement. Some systems also include additional handheld components that can be cumbersome to the user.SUMMARY

According to one aspect of the present disclosure, a multisensory augmented reality system is disclosed. The system includes a visual augmentation system that displays augmented content, wherein the augmented content is a blend of original content integrated with real-time content, a sensory feedback system that provides sensor

Claims

1. A multisensory augmented reality system comprising: a visual augmentation system that displays augmented content, wherein the augmented content is a blend of original content integrated with real-time content; a sensory feedback system that provides sensory feedback to a user based on the augmented content, wherein the sensory feedback dynamically adjusts based on changes in the augmented content; and a processing element in communication with the visual augmentation system and the sensory feedback system, wherein the processing element performs the following operations: receive the real-time content of a physical environment from the visual augmentation system; construct using the real time content a visual representation of the physical environment; integrate the original content with the visual representation to generate the augmented content; and generate and transmit sensory feedback instructions to the sensory feedback system based on the augmented content and the augmented content to the visual augmentation system for display 9. A method for providing a multisensory augmented reality environment comprising: receiving, by a processing element, a real-time video stream of a physical environment; receiving, by the processing element, depth information of one or more objects within the physical environment; generating, by the processing element, real-time content depicting the physical environment based on the real-time video stream and the depth information; rendering, by the processing element, augmented content, wherein the augmented content integrates original content and the real-time content; generating, by the processing element, sensory feedback instructions to provide sensory feedback based on the augmented content, wherein the sensory feedback instructions are dynamically adjusted to changes in the augmented content; transmitting, by the processing element, the augmented content for display and instructions to provide sensory feedback correlated to the augmented content. 16. A method for providing an augmented reality environment comprising: determining, by a processing element, a correspondence between a camera and a depth sensor; receiving, by the processing element, a calibration image and depth information corresponding to a physical environment; generating, by the processing element, a real-time content depicting the physical environment based on the image, the depth information, and the correspondence; detecting, by the processing element, a physical landmark in the real-time content; aligning, by the processing element, a virtual representation of the landmark in a virtual environment with the physical landmark in the real-time content; generating, by the processing element, original content in the virtual environment; and transmitting, by the processing element, the original content and the real-time content for display.