Real-time rendering with compressed animated light fields

Real-time light-field rendering technique.

Systems, methods, and articles of manufacture for real-time rendering using compressed animated light fields are disclosed. One embodiment provides a pipeline, from offline rendering of an animated scene from sparse optimized viewpoints to real-time rendering of the scene with freedom of movement, that includes three stages: offline preparation and rendering, stream compression, and real-time decompression and reconstruction. During offline rendering, optimal placements for cameras in the scene are determined, and color and depth images are rendered using such cameras. Color and depth data is then compressed using an integrated spatial and temporal scheme permitting high performance on graphics processing units for virtual reality applications. The compressed content may be decoded and reconstructed in real-time by selecting, using heuristics, cameras that provide useful data for a viewer, selecting grid cells from those cameras that are visible to the viewer, and using a ray marching technique to reconstruct the scene.

BACKGROUND(1) Field(2) Aspects of the disclosure presented herein relate to image rendering and, more specifically, to rendering images using compressed animated light fields.(3) Description of the Related Art(4) Recently, virtual reality (VR) has become practical for widespread consumer adoption with a variety of hardware releases of head-mounted displays (HMDs). However, the real-time rendering performance requirements for VR content are typically much higher than for non-VR rendering. As a result, VR content has traditionally been rendered with lower-complexity visuals than non-VR rendering. For example, cinematic-quality VR graphics could not be rendered in real time.(5) Immersive 360-degree videos (monoscopic or stereoscopic) have also emerged as a popular form of content. However, such videos are typically captured assuming specific eye locations. As a result, traditional immersive 360-degree videos lack motion parallax and can result in immersion breaking and the feeling that the content is flat, or even discomfort, when viewers' eyes diverge from the specific eye locations from which the videos were captured. To prevent such immersion breaking and discomfort, the content being displayed to the user must be modified to appear correct from any given eye location and head angle. However, this typically requires capturing and rendering the content from a very large number of eye locations, thereby increasing authoring, storage, processing, and bandwidth costs, among other