- Number
- 12475656
- Published
- 2025-11-18
- Filed
- 2022-05-19
- Assignee
- DISNEY ENTERPRISES, INC.
- Inventors
- Varis Doggett; Erika et al.
- CPC
- A63F13/67; G06T19/006; A63F13/65; A63F13/56
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Space/content matching technique for AR/MR environments.
Abstract
Techniques for automatically placing and manipulating virtual objects of augmented reality (AR) and mixed reality (MR) simulations are described. One technique includes obtaining an indication of at least one virtual object available for placing within a real-world environment during an AR simulation or a MR simulation. A first representation of the real-world environment is generated, based on a scan of the real-world environment. At least one second representation of the real-world environment is generated from the first representation. A match is determined between the at least one virtual object and at least one available space within the real-world environment, based at least in part on evaluating the at least one virtual object and the at least one second representation with a machine learning model(s). The at least one virtual object is rendered on a computing device, based on the match.
Background
BACKGROUND (1) Today, augmented reality (AR) technology and mixed reality (MR) technology are increasingly being used in a variety of different fields, including, for example, the medical field, social networking and engagement, communications, shopping, entertainment industry, travel, navigation, education, etc. AR involves superimposing computer generated imagery on a user's view of the real-world environment. An AR system can use a video device to display a series of images (or video feed) of the real-world environment to a user, where the images have various virtual objects inserted into appropriate places in the environment. For example, the AR system can identify a real-world object of a table, so that a virtual object of a cup may be displayed on the video device as appearing on the table (e.g., from the perspective of the video device). MR is generally an extension of AR that allows for interaction between real and virtual objects in an environment. For example, in an MR experience, a user may adapt and manipulate visual content (e.g. graphics for gaming or an interactive story) overlaying a view of the physical environment. (2) Currently, many AR/MR experiences rely on the user to place and manipulate the size of a virtual object on the user's display device in order for the virtual object to “fit” in the user's real-world environment. Relying on this user manipulation to set up an AR/MR experience can negatively impact the user's immersive experience within the phys
Claims
1. A computer-implemented method comprising: obtaining an indication of at least one virtual object available for placing within a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation; generating a first representation of the real-world environment, based on a scan of the real-world environment with one or more sensors of a computing device, wherein the first representation includes a first three-dimensional (3D) mesh representing a geometry of the real-world environment; generating at least one second representation of the real-world environment by computing an inverse of the first 3D mesh corresponding to the first representation of the real-world environment, wherein the at least one second representation comprises a second 3D mesh of at least one empty space in the real-world environment; determining a match between one or more attributes of the at least one virtual object and one or more additional attributes of the at least one empty space within the real-world environment, based at least in part on evaluating the at least one virtual object and the at least one second representation of the real-world environment with one or more machine learning models; and rendering the at least one virtual object at a position on a display of the computing device, based on the match, the position on the display being associated with the at least one empty space within the real-world environment. ||
7. A computer-implemented method comprising: obtaining an indication of at least one virtual object available for placing within a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation; generating a first representation of the real-world environment, based on a scan of the real-world environment with one or more sensors of a computing device, wherein the first representation includes a first three-dimensional (3D) mesh representing a geometry of the real-world environment; generating at least one second representation of the real-world environment by computing an inverse of the first 3D mesh corresponding to the first representation of the real-world environment, wherein the at least one second representation comprises a second 3D mesh of at least one empty space in the real-world environment; determining a match between the at least one virtual object and the at least one empty space within the real-world environment, based at least in part on evaluating the at least one virtual object and the at least one second representation of the real-world environment with one or more machine learning models; rendering the at least one virtual object at a position on a display of the computing device, based on the match, the position on the display being associated with the at least one empty space within the real-world environment; and determining a plurality of paths of the computing device through the real-world environment, wherein: the at least one second representation indicates the at least one empty space along a first path of the plurality of paths, or the at least one virtual object is matched to the at least one empty space along the first path of the plurality of paths. ||
10. A computing device comprising: a display; one or more sensors; one or more processors; and a memory storing instructions, which, when executed on the one or more processors perform an operation comprising: obtaining an indication of at least one virtual object available for placing within a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation; generating a first representation of the real-world environment, based on a scan of the real-world environment with one or more sensors of a computing device, wherein the first representation includes a first three-dimensional (3D) mesh representing a geometry of the real-world environment; generating at least one second representation of the real-world environment by computing an inverse of the first 3D mesh corresponding to the first representation of the real-world environment, wherein the at least one second representation comprises a second 3D mesh of at least one empty space in the real-world environment; determining a match between one or more attributes of the at least one virtual object and one or more additional attributes of the at least one empty space within the real-world environment, based at least in part on evaluating the at least one virtual object and the at least one second representation of the real-world environment with one or more machine learning models; and rendering the at least one virtual object at a position on the display, based on the match, the position on the display being associated with the at least one empty space within the real-world environment. ||
16. A non-transitory computer-readable medium containing computer program code that, when executed by operation of one or more computer processors, performs an operation comprising: obtaining an indication of at least one virtual object available for placing within a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation; generating a first representation of the real-world environment, based on a scan of the real-world environment with one or more sensors of a computing device, wherein the first representation includes a first three-dimensional (3D) mesh representing a geometry of the real-world environment; generating at least one second representation of the real-world environment by computing an inverse of the first 3D mesh corresponding to the first representation of the real-world environment, wherein the at least one second representation comprises a second 3D mesh of at least one empty space in the real-world environment; determining a match between one or more attributes of the at least one virtual object and one or more additional attributes of the at least one empty space within the real-world environment, based at least in part on evaluating the at least one virtual object and the at least one second representation of the real-world environment with one or more machine learning models; and rendering the at least one virtual object at a position on a display of the computing device, based on the match, the position on the display being associated with the at least one empty space within the real-world environment. ||
20. A computer-implemented method comprising: obtaining an indication of at least one virtual object available for placing within a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation; generating a first representation of the real-world environment, based on a scan of the real-world environment with one or more sensors of a computing device, wherein the first representation includes a first three-dimensional (3D) mesh representing a geometry of the real-world environment; determining a plurality of paths of the computing device through the real-world environment, wherein each path included in the plurality of paths includes a 3D volume of an amount of space taken by a user through the real-world environment; generating at least one second representation of the real-world environment by computing an inverse of the first 3D mesh corresponding to the first representation of the real-world environment and the plurality of paths, wherein the at least one second representation indicates at least one available space along a first path of the plurality of paths; determining a match between the at least one virtual object and the at least one available space within the real-world environment, based at least in part on evaluating the at least one virtual object and the at least one second representation of the real-world environment with one or more machine learning models; and rendering the at least one virtual object at a position on a display of the computing device, based on the match, the position on the display being associated with the at least one available space within the real-world environment.