Real-time projection mapping hardware for mixed reality (granted).
The present disclosure relates generally to systems and methods for creating a mixed reality environment. A mixed reality system includes a performance area for generating a mixed reality environment, a motion determination module that determines and tracks the motion of an object within the performance area, a physical article module that generates a physical article within the performance area, and a mixed reality display that displays a virtual article within the performance area, wherein the physical article and the virtual article are correlated to the motion of the object.
TECHNICAL FIELD (1) The technology described herein generally relates systems and methods to project light onto and around objects in real time in a mixed reality environment. BACKGROUND (2) Traditional mixed reality, virtual reality, and augmented reality environments involve the merging of real and virtual components in a user environment such that virtual and physical objects may coexist and interact. Many mixed reality implementations do not include virtual object to physical object interactions (e.g., virtual light emitted from a virtual object does not wash adjacent physical objects with light), since such interactions are difficult to generate virtually in a realistic manner. In particular, such interactions may require high-resolution three-dimensional scans of the environment that are time and computing resource intensive, as well as modeling of surface reflections, refractions, and other light interactions that are difficult and computationally intensive to determine. (3) For example, with “see through” mixed reality systems, such as head worn displays that pass through light from the environment, these systems can generally perform “additive effects,” such as adding effects or objects to a scene, but subtractive effects, such as creating shadows, are difficult to produce realistically and typically will not generate downstream or supplemental effects that would be expected in object-to-object interactions in the physical world. Similarly, traditional techniques may
1. A mixed reality system comprising: an environment model hosted on a computing device, including data corresponding to a physical environment and a virtual environment, wherein the environmental model replicates a mixed reality environment; a physical article module that generates a physical article within the physical environment; and a user wearable mixed reality display in electronic communication with the computing device that receives image data from the computing device and displays a virtual article within the virtual environment based on the image data, wherein: an actuation of the physical article and an actuation of the virtual article are correlated with one another based on the environment model, light from the physical article passes through the mixed reality display toward a user such that the physical environment and the virtual environment are combined and the physical article and the virtual article are viewable together as a composite effect through the mixed reality display, and the environmental model tracks a dynamic position of the physical article such that the physical article is configured to enhance an authenticity of the virtual article when viewed together as the composite effect through the mixed reality display. ||
23. A method of generating a mixed reality environment comprising: tracking, based on an environment model that replicates a mixed reality environment hosted on a computing device, a dynamic position of a physical article within a performance area; determining a position of a virtual article within a virtual environment, based on the environment model, that overlays the performance area; rendering the virtual article via a user wearable mixed reality display that receives image data from the computing device and displays the virtual article within the virtual environment; rendering the physical article within the performance area to enhance an authenticity of the virtual article; and rendering the virtual article with a mixed reality display within the virtual environment synchronized with the physical article in the performance area to generate a mixed reality experience for a user visualizing the virtual environment and the performance area with the mixed reality display, wherein light from the physical article passes through the mixed reality display toward a user such that the physical environment and virtual environment are combined and the physical article and the virtual article are viewable together as a composite effect through the mixed reality display. ||
28. A method to generate a mixed reality experience comprising: detecting state information from an environmental model corresponding to a physical light source and a virtual object, wherein the environmental model tracks a position of the physical light source; based on the state information, actuating the physical light source to project light on to a location within a physical environment, such that the light source enhances an authenticity of the virtual object; rendering the virtual object within a virtual environment presented by a user wearable mixed reality display, wherein light from the physical light source passes through the mixed reality display toward a user such that the virtual object and the physical light source are viewable together as a composite effect through the mixed reality display; and updating the location of the physical light source based on a free movement of the virtual object within the virtual environment.