Mixed-reality environment streaming between multiple devices (granted).
An immersive content presentation system can capture the motion or position of a performer in a real-world environment. A game engine can be modified to receive the position or motion of the performer and identify predetermined gestures or positions that can be used to trigger actions in a 3-D virtual environment, such as generating a digital effect, transitioning virtual assets through an animation graph, adding new objects, and so forth. The use of the 3-D environment can be rendered and composited views can be generated. Information for constructing the composited views can be streamed to numerous display devices in many different physical locations using a customized communication protocol. Multiple real-world performers can interact with virtual objects through the game engine in a shared mixed-reality experience.
TECHNICAL FIELD (1) This application discloses technology related to the fields of computer animation, virtual reality environments, and digital content generation. Specifically, this application discloses technology for using a real-time gestures to drive computer-generated assets in a multi-device veiwing environment. BACKGROUND (2) Augmented reality includes a live view of a real-world environment that is augmented by computer generated sensory input(s), such as GPS graphics, video, sound, data statistics, and so forth. In contrast to virtual reality, which replaces the real-world environment with a simulated one, augmented reality elements are often displayed in real time in semantic context with elements of the real-world environment. For example, sports scores can be displayed on a television during a basketball game on a same screen. Headmounted displays can also be used to place the virtual images over a view of the physical world such that both are in the user's field of view. (3) Virtual reality is very similar to augmented reality in that it entails an interactive computer generated experience. However, virtual reality takes place within a simulated, immersive environment. Current virtual reality technology commonly uses virtual-reality headset or multi-projected environments to create realistic sensations, sales, and images that simulate the user having a physical presence in the virtual or imaginary environment. This effect is usually implemented using a virtual-
1. A method comprising: receiving a motion or position of a first real-world performer in a first real-world environment; moving or positioning a virtual representation of the first real-world performer in a 3-D virtual environment based on the motion or position of the first real-world performer; rendering a 2-D video stream of the virtual representation of the first real-world performer in the 3-D virtual environment; adding or altering a virtual asset in a 3-D virtual environment in response to the motion or position of the first real-world performer; determining a mathematical 3-D transform to be applied to the virtual asset based on identifying the motion or position; preparing a stream of real-time communication packets, wherein each communication packet in the stream of real-time communication packets comprises a field configured to represent the mathematical 3-D transform to be applied to the virtual asset in the 3-D virtual environment; and transmitting the stream of real-time communication packets to a plurality of display devices distributed within a second real-world environment; and causing a real-time video stream to be visible on a display device of a handheld or wearable device in the plurality of display devices within the second real-world environment, wherein: the handheld or wearable device captures the real-time view of a second real-world environment comprising view of a second real-world performer; the real-time video stream comprises the 2-D video stream of the virtual representation of the first real-world performer composited with the real-time view of the second real-world environment comprising a view of the second real-world performer; and the motion or position of the first real-world performer in the first real-world environment moves or positions the virtual representation of the first real-world performer in the 3-D virtual environment in real time as the 2-D video stream is composited with the real-time view of the second real-world environment such that the second real-world performer and the virtual representation of the first real-world performer appear to interact in real time with each other when viewed on the display device of the handheld or wearable device. ||
7. An immersive content presentation system comprising: one or more processors; and one or more memory devices comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving a motion or position of a first real-world performer in a first real-world environment; moving or positioning a virtual representation of the first real-world performer in a 3-D virtual environment based on the motion or position of the first real-world performer, wherein the 3-D virtual environment includes models or volumes that correspond to physical objects in a second real-world environment; rendering a 2-D video stream of the virtual representation of the first real-world performer in the 3-D virtual environment, including: rendering images in the 2-D video stream without images of the models or volumes that correspond to the physical objects in the second real-world environment; and generating cutouts in images of the virtual representation of the first real-world performer in the 2-D video stream when the virtual representation of the first real-world performer is behind the models or volumes that correspond to the physical objects in the second real-world environment; and causing a real-time video stream to be visible on a display device of a handheld or wearable device, wherein: the handheld or wearable device captures the real-time view of a second real-world environment comprising view of a second real-world performer; the real-time video stream comprises the 2-D video stream of the virtual representation of the first real-world performer composited with the real-time view of the second real-world environment comprising a view of the second real-world performer; and the motion or position of the first real-world performer in the first real-world environment moves or positions the virtual representation of the first real-world performer in the 3-D virtual environment in real time as the 2-D video stream is composited with the real-time view of the second real-world environment such that the second real-world performer and the virtual representation of the first real-world performer appear to interact in real time with each other when viewed on the display device of the handheld or wearable device. ||
14. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving a motion or position of a first real-world performer in a first real-world environment; identifying the motion or position as a predefined motion or position; and moving or positioning a virtual representation of the first real-world performer in a 3-D virtual environment based on the motion or position of the first real-world performer; adding or altering a virtual asset in the 3-D virtual environment in response to identifying the motion or position as the predefined motion or position; rendering a 2-D video stream of the virtual representation of the first real-world performer in the 3-D virtual environment; and causing a real-time video stream to be visible on a display device of a handheld or wearable device, wherein: the handheld or wearable device captures a real-time view of a second real-world environment comprising view of a second real-world performer; the real-time video stream comprises the 2-D video stream of the virtual representation of the first real-world performer composited with the real-time view of the second real-world environment comprising a view of the second real-world performer; the motion or position of the first real-world performer in the first real-world environment moves or positions the virtual representation of the first real-world performer in the 3-D virtual environment in real time as the 2-D video stream is composited with the real-time view of the second real-world environment such that the second real-world performer and the virtual representation of the first real-world performer appear to interact in real time with each other when viewed on the display device of the handheld or wearable device; and the motion or position of the first real-world performer in the first real-world environment adds or alters the virtual asset in the 3-D virtual environment in real time as the 2-D video stream is composited, such that the second real-world performer and the virtual representation of the first real-world performer appear to interact in real time with the virtual asset when viewed on the display device of the handheld or wearable device.