- Number
- 20190325661
- Published
- 2019-10-24
- Filed
- 2018-04-19
- Assignee
- Disney Enterprises, Inc.
- Inventors
- BAUMBACH; Elliott H., HSU; Jonathan RD
- CPC
- G06T15/04; A63F13/211; G06T15/005; G06T19/20; G06T19/006; A63F13/52; G06F3/0346; G06F3/0304; A63F13/65; G06F3/011; A63F13/25
- Verdict
- Medium Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Pre-generated 3D model occlusion for AR (novel rendering).
Abstract
To prevent virtual content from occluding physical objects in an augmented reality (AR) system, the embodiments herein describe generating a pass-through texture using a pre-generated model of a physical object. In one embodiment, the AR system determines the location of the physical object as well as its orientation. With this information, the system rotates and re-sizes the pre-generated model of the object to match its location and orientation in the real-world. The system then generates the pass-through texture from the pre-generated model and inserts the texture into the virtual content. When the virtual content is displayed, the pass-through texture permits light from the real-world view to pass through substantially unaffected by the virtual content. In this manner, the physical object (which aligns with the location of the pass-through texture in the virtual content) can be seen by the user—i.e., is not occluded by the virtual content.
Background
BACKGROUNDField of the Invention
The present disclosure generally relates to the occlusion of a physical object by virtual content in an augmented reality (AR) and/or virtual reality (VR) system.Description of the Related Art
Computer graphics technology has significantly progressed since the first video games were developed. Relatively inexpensive 3D graphics engines now provide nearly photo-realistic interactive gameplay on hand-held video game systems, home video game systems, and personal computer hardware platforms costing only a few hundred dollars. These video game systems typically include a hand-held controller, game controller, or, in the case of a hand-held video game systems, an integrated controller. A user interacts with the controller to send commands or other instructions to the video game system to control a video game or other simulation. For example, the controller may include a joystick and buttons operated by the user.
While video games allow the user to interact directly with the video game system, such interactions primarily influence the graphical depiction shown on the video game device (or on a connected display), and rarely influence any other objects outside of the virtual world. That is, a user may specify an input to the video game system, indicating that the user's avatar should perform a jump action, and in response the video game system could display the user's avatar jumping. However, such interactions are typically limited
Claims
1. A method, comprising: determining a location of a peripheral device of an augmented reality (AR) system in a real-world environment; determining an orientation of the peripheral device in the real-world environment; orienting and sizing a pre-generated model of the peripheral device to match a size and the orientation of the peripheral device relative to a viewpoint of a user; generating a pass-through texture using the pre-generated model; inserting the pass-through texture into the virtual content, wherein the pass-through texture permits light emitted from the real-world environment to be seen by the user without being occluded by virtual content; and outputting for display the pass-through texture and the virtual content.
2. The method of claim 1, further comprising: determining that the peripheral device has a location in the real-world environment that overlaps a location of the virtual content in an AR image.
3. The method of claim 1, wherein the pre-generated model is a 3D model, and wherein generating the pass-through texture comprises: converting the pre-generated model into the pass-through texture, wherein the pass-through texture comprises a 2D displayable texture; and assigning a color to the pass-through texture such that a display in the AR system is black in an area displaying the pass-through texture.
4. The method of claim 1, wherein the peripheral device comprises a game controller for interacting with the virtual content; the method comprising: outputting for display projected virtual content that extends from the peripheral device.
5. The method of claim 1, further comprising: receiving inertial measurement unit (IMU) data from the peripheral device, wherein the orientation of the peripheral device is determined using the IMU data.
6. The method of claim 1, wherein determining the location of the peripheral device in the real-world environment comprises: capturing an image of the peripheral device using a light sensor in the AR system, wherein the peripheral device comprises an illuminated light source detectable by the light sensor; and tracking the illuminated light source using the captured image to determine the location of the peripheral device.
7. A computer-readable storage medium comprising computer-readable program code embodied therewith, the computer-readable program code is configured to perform, when executed by a processor, an operation, the operation comprising: determining a location of a peripheral device of an AR system in a real-world environment; determining an orientation of the peripheral device in the real-world environment; orienting and sizing a pre-generated model of the peripheral device to match a size and the orientation of the peripheral device relative to a viewpoint of a user; generating a pass-through texture using the pre-generated model; inserting the pass-through texture into the virtual content, wherein the pass-through texture permits light emitted from the real-world environment to be seen by the user without being occluded by virtual content; and outputting for display the pass-through texture and the virtual content.
8. The computer-readable storage medium of claim 7, the operation further comprising: determining that the peripheral device has a location in the real-world environment that overlaps a location of the virtual content in an AR image.
9. The computer-readable storage medium of claim 7, wherein the pre-generated model is a 3D model, and wherein generating the pass-through texture comprises: converting the pre-generated model into the pass-through texture, wherein the pass-through texture comprises a 2D displayable texture; and assigning a color to the pass-through texture such that a display in the AR system is black in an area displaying the pass-through texture.
10. The computer-readable storage medium of claim 7, wherein the peripheral device comprises a game controller for interacting with the virtual content; the operation comprising: outputting for display projected virtual content that extends from the peripheral device.
11. The computer-readable storage medium of claim 7, the operation further comprising: receiving inertial measurement unit (IMU) data from the peripheral device, wherein the orientation of the peripheral device is determined using the IMU data.
12. The computer-readable storage medium of claim 7, wherein determining the location of the peripheral device in the real-world environment comprises: capturing an image of the peripheral device using a light sensor in the AR system, wherein the peripheral device comprises an illuminated light source detectable by the light sensor; and tracking the illuminated light source using the captured image to determine the location of the peripheral device.
13. A computer-readable storage medium comprising computer-readable program code embodied therewith, the computer-readable program code is configured to perform, when executed by a processor, an operation, the operation comprising: determining a location of a physical object in a real-world environment; determining an orientation of the physical object in the real-world environment; orienting and sizing a pre-generated model of the physical object to match a size and the orientation of the physical object relative to a viewpoint of a user, wherein the pre-generated model was generated before determining the location and orientation of the physical object; generating a black texture using the pre-generated model; inserting the black texture into a portion of virtual content that corresponds to the location of the physical object in the real-world environment; and outputting for display the black texture and the virtual content to generate an AR image.
14. The computer-readable storage medium of claim 13, wherein the pre-generated model is a 3D model, and wherein generating the black texture comprises: converting the pre-generated model into the black texture, wherein the black texture comprises a 2D displayable texture; and assigning black to the black texture such that a display in an AR system outputs black in an area displaying the black texture.
15. The computer-readable storage medium of claim 13, wherein the physical object comprises a game controller for interacting with the virtual content; the operation comprising: outputting for display projected virtual content that extends from the peripheral device.
16. The computer-readable storage medium of claim 13, the operation further comprising: receiving inertial measurement unit (IMU) data from the physical object, wherein the orientation of the physical object is determined using the IMU data.
17. The computer-readable storage medium of claim 13, wherein determining the location of the physical object in the real-world environment comprises: capturing an image of the physical object using a light sensor, tracking the physical object using the captured image to determine the location of the physical object.
18. The computer-readable storage medium of claim 17, wherein the physical object comprises an illuminated light source detectable by the light sensor, and wherein tracking the physical object using the captured image comprises: tracking the light source using the captured image.
19. The computer-readable storage medium of claim 13, the operation further comprising: estimating a location and an orientation of a body part of the user supporting the physical object; and modifying the black texture based on the estimating.
20. The computer-readable storage medium of claim 19, wherein the modified black texture, when displayed, prevents the virtual content from occluding the body part of the user.