CV occlusion technique for AR using pre-generated 3D models.
To prevent virtual content from occluding physical objects in an augmented reality (AR) system, the embodiments herein describe generating a pass-through texture using a pre-generated model of a physical object. In one embodiment, the AR system determines the location of the physical object as well as its orientation. With this information, the system rotates and re-sizes the pre-generated model of the object to match its location and orientation in the real-world. The system then generates the pass-through texture from the pre-generated model and inserts the texture into the virtual content. When the virtual content is displayed, the pass-through texture permits light from the real-world view to pass through substantially unaffected by the virtual content. In this manner, the physical object (which aligns with the location of the pass-through texture in the virtual content) can be seen by the user—i.e., is not occluded by the virtual content.
BACKGROUNDField of the Invention(1) The present disclosure generally relates to the occlusion of a physical object by virtual content in an augmented reality (AR) and/or virtual reality (VR) system.Description of the Related Art(2) Computer graphics technology has significantly progressed since the first video games were developed. Relatively inexpensive 3D graphics engines now provide nearly photo-realistic interactive gameplay on hand-held video game systems, home video game systems, and personal computer hardware platforms costing only a few hundred dollars. These video game systems typically include a hand-held controller, game controller, or, in the case of a hand-held video game systems, an integrated controller. A user interacts with the controller to send commands or other instructions to the video game system to control a video game or other simulation. For example, the controller may include a joystick and buttons operated by the user.(3) While video games allow the user to interact directly with the video game system, such interactions primarily influence the graphical depiction shown on the video game device (or on a connected display), and rarely influence any other objects outside of the virtual world. That is, a user may specify an input to the video game system, indicating that the user's avatar should perform a jump action, and in response the video game system could display the user's avatar jumping. However, such interactions are typically limited to the vi
1. A method, comprising: determining a location of a handheld peripheral device of an augmented reality (AR) system in a real-world environment; determining an orientation of the handheld peripheral device in the real-world environment based on data received from a first sensor disposed on the handheld peripheral device; orienting and sizing a pre-generated model of the handheld peripheral device to match a size and the orientation of the handheld peripheral device relative to a viewpoint of a user; generating a pass-through texture using the pre-generated model; determining a location of fingers of the user on the handheld peripheral device using a second sensor disposed on the handheld peripheral device; estimating an orientation and location of an arm of the user based on an estimation of a length of the arm, a distance between a headset and the handheld peripheral device, and the orientation of the handheld peripheral device; modifying the pass-through texture to include (i) the fingers of the user based on the determined location of the fingers and (ii) the arm of the user; inserting the modified pass-through texture into virtual content, wherein the modified pass-through texture permits light emitted from the real-world environment to be seen by the user without being occluded by the virtual content; and outputting for display the modified pass-through texture and the virtual content.
7. A non-transitory computer-readable storage medium comprising computer-readable program code embodied therewith, the computer-readable program code is configured to perform, when executed by a processor, an operation, the operation comprising: determining a location of a handheld peripheral device of an AR system in a real-world environment; determining an orientation of the handheld peripheral device in the real-world environment based on data received from a first sensor disposed on the handheld peripheral device; orienting and sizing a pre-generated model of the handheld peripheral device to match a size and the orientation of the handheld peripheral device relative to a viewpoint of a user; generating a pass-through texture using the pre-generated model; determining a location of fingers of the user on the handheld peripheral device using a second sensor disposed on the handheld peripheral device; estimating an orientation and location of an arm of the user based on an estimation of a length of the arm, a distance between a headset and the handheld peripheral device, and the orientation of the handheld peripheral device; modifying the pass-through texture to include (i) the fingers of the user based on the determined location of the fingers and (ii) the arm of the user; inserting the modified pass-through texture into virtual content, wherein the modified pass-through texture permits light emitted from the real-world environment to be seen by the user without being occluded by the virtual content; and outputting for display the modified pass-through texture and the virtual content.
13. A non-transitory computer-readable storage medium comprising computer-readable program code embodied therewith, the computer-readable program code is configured to perform, when executed by a processor, an operation, the operation comprising: determining a location of a handheld physical object in a real-world environment; determining an orientation of the handheld physical object in the real-world environment based on data from a first sensor disposed on the handheld physical object; orienting and sizing a pre-generated model of the handheld physical object to match a size and the orientation of the handheld physical object relative to a viewpoint of a user, wherein the pre-generated model was generated before determining the location and orientation of the handheld physical object; generating a black texture using the pre-generated model; determining a location of fingers of the user on the handheld physical object using a second sensor disposed on the handheld physical object; estimating an orientation and location of an arm of the user based on an estimation of a length of the arm, a distance between a headset and the handheld physical object, and the orientation of the handheld physical object; modifying the black texture to include (i) the fingers of the user based on the determined location of the fingers of the user and (ii) the arm of the user; inserting the modified black texture into a portion of virtual content that corresponds to the location of the handheld physical object in the real-world environment; and outputting for display the modified black texture and the virtual content to generate an AR image.