Space/content-matching technique for AR/MR experiences.
Techniques for automatically placing and manipulating virtual objects of augmented reality (AR) and mixed reality (MR) simulations are described. One technique includes obtaining an indication of at least one virtual object available for placing within a real-world environment during an AR simulation or a MR simulation. A first representation of the real-world environment is generated, based on a scan of the real-world environment. At least one second representation of the real-world environment is generated from the first representation. A match is determined between the at least one virtual object and at least one available space within the real-world environment, based at least in part on evaluating the at least one virtual object and the at least one second representation with a machine learning model(s). The at least one virtual object is rendered on a computing device, based on the match.
BACKGROUND
Today, augmented reality (AR) technology and mixed reality (MR) technology are increasingly being used in a variety of different fields, including, for example, the medical field, social networking and engagement, communications, shopping, entertainment industry, travel, navigation, education, etc. AR involves superimposing computer generated imagery on a user's view of the real-world environment. An AR system can use a video device to display a series of images (or video feed) of the real-world environment to a user, where the images have various virtual objects inserted into appropriate places in the environment. For example, the AR system can identify a real-world object of a table, so that a virtual object of a cup may be displayed on the video device as appearing on the table (e.g., from the perspective of the video device). MR is generally an extension of AR that allows for interaction between real and virtual objects in an environment. For example, in an MR experience, a user may adapt and manipulate visual content (e.g. graphics for gaming or an interactive story) overlaying a view of the physical environment.
Currently, many AR/MR experiences rely on the user to place and manipulate the size of a virtual object on the user's display device in order for the virtual object to “fit” in the user's real-world environment. Relying on this user manipulation to set up an AR/MR experience can negatively impact the user's immersive experience within th
1. A computer-implemented method comprising: obtaining an indication of at least one virtual object available for placing within a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation; generating a first representation of the real-world environment, based on a scan of the real-world environment with one or more sensors of a computing device; generating at least one second representation of the real-world environment from the first representation of the real-world environment, wherein the at least one second representation comprises a three-dimensional (3D) representation of at least one empty space within the real-world environment; determining a match between the at least one virtual object and the at least one empty space within the real-world environment, based at least in part on evaluating the at least one virtual object and the at least one second representation of the real-world environment with one or more machine learning models; and rendering the at least one virtual object at a position on a display of the computing device, based on the match, the position on the display being associated with the at least one empty space within the real-world environment. ||
10. A computing device comprising: a display; one or more sensors; one or more processors; and a memory storing instructions, which, when executed on the one or more processors perform an operation comprising: obtaining an indication of at least one virtual object available for placing within a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation; generating a first representation of the real-world environment, based on a scan of the real-world environment with the one or more sensors; generating at least one second representation of the real-world environment from the first representation of the real-world environment, wherein the at least one second representation comprises a three-dimensional (3D) representation of at least one empty space within the real-world environment; determining a match between the at least one virtual object and the at least one empty space within the real-world environment, based at least in part on evaluating the at least one virtual object and the at least one second representation of the real-world environment with one or more machine learning models; and rendering the at least one virtual object at a position on the display, based on the match, the position on the display being associated with the at least one empty space within the real-world environment. ||
17. A non-transitory computer-readable medium containing computer program code that, when executed by operation of one or more computer processors, performs an operation comprising: obtaining an indication of at least one virtual object available for placing within a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation; generating a first representation of the real-world environment, based on a scan of the real-world environment with one or more sensors of a computing device; generating at least one second representation of the real-world environment from the first representation of the real-world environment, wherein the at least one second representation comprises a three-dimensional (3D) representation of at least one empty space within the real-world environment; determining a match between the at least one virtual object and the at least one empty space within the real-world environment, based at least in part on evaluating the at least one virtual object and the at least one second representation of the real-world environment with one or more machine learning models; and rendering the at least one virtual object at a position on a display of the computing device, based on the match, the position on the display being associated with the at least one empty space within the real-world environment. ||
21. A computer-implemented method comprising: obtaining an indication of at least one virtual object available for placing within a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation; generating a first representation of the real-world environment, based on a scan of the real-world environment with one or more sensors of a computing device; determining a plurality of paths of the computing device through the real-world environment; generating at least one second representation of the real-world environment from the first representation of the real-world environment and the plurality of paths, wherein the at least one second representation indicates at least one available space along a first path of the plurality of paths; determining a match between the at least one virtual object and the at least one available space within the real-world environment, based at least in part on evaluating the at least one virtual object and the at least one second representation of the real-world environment with one or more machine learning models; and rendering the at least one virtual object at a position on a display of the computing device, based on the match, the position on the display being associated with the at least one available space within the real-world environment.