- Number
- 20190313080
- Published
- 2019-10-10
- Filed
- 2018-09-21
- Assignee
- Disney Enterprises, Inc.
- Inventors
- MITCHELL; Kenneth J., KONIARIS; Charalampos, KOSEK; Malgorzata E., SINCLAIR; David A.
- CPC
- H04N19/103; G06T9/40; H03M13/6513; H04N13/161; H04N19/96; H04N19/12; H04N13/232; G06T17/005; H04N19/19; H04N19/1883; H04N19/11; H04N19/597; H03M13/6572
- Verdict
- Medium Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Depth codec for real-time light field reconstruction.
Abstract
Systems, methods, and articles of manufacture are disclosed that enable the compression of depth data and real-time reconstruction of high-quality light fields. In one aspect, spatial compression and decompression of depth images is divided into the following stages: generating a quadtree data structure for each depth image captured by a light field probe and difference mask associated with the depth image, with each node of the quadtree approximating a corresponding portion of the depth image data using an approximating function; generating, from the quadtree for each depth image, a runtime packed form that is more lightweight and has a desired maximum error; and assembling multiple such runtime packed forms into per-probe stream(s); and decoding at runtime the assembled per-probe stream(s). Further, a block compression format is disclosed for approximating depth data by augmenting the block compression format 3DC+(BC4) with a line and two pairs of endpoints.
Background
BACKGROUNDField of the Disclosure
Aspects of the disclosure presented herein relate to image rendering and, more specifically, to a depth codec for real-time, high-quality light field reconstruction.Description of the Related Art
Recent advances in virtual and augmented reality software and hardware have sparked interest in high-quality mixed reality experiences that seamlessly blend together characters and environments. For immersive and interactive experiences that allow free user movement with six degrees of freedom, video content that is presented needs to be adapted for consumption from any point of view.
Video codec development has been strongly focused on color data, rather than depth data. However, depth estimation from color in real-time can be prohibitively expensive, and traditional video codecs have demonstrated poor performance when the amount of data to be decoded becomes large.SUMMARY
One aspect of this disclosure provides a computer-implemented method that generally includes determining a mask for a received depth image frame based, at least in part, on differences between the received depth image frame and one or more previous depth image frames. The method further includes generating a quadtree data structure that includes nodes associated with pixels of the depth image frame masked by the determined mask. Each node of the generated quadtree indicates one of a plurality of functions used to approximate a corresponding portion of the r
Claims
1. A computer-implemented method, comprising: determining a mask for a received depth image frame based, at least in part, on differences between the received depth image frame and one or more previous depth image frames; and generating a quadtree data structure that includes nodes associated with pixels of the depth image frame masked by the determined mask, wherein each node of the generated quadtree indicates one of a plurality of functions used to approximate a corresponding portion of the received depth image frame and coefficient values used in the approximation.
2. The computer-implemented method of claim 1, further comprising, generating a runtime packed form of the quadtree by: traversing the quadtree top-to-bottom and depth-first; and copying nodes of the quadtree that are able to approximate corresponding portions of the received depth image within an error to the runtime packed form.
3. The computer-implemented method of claim 2, further comprising, assembling the generated runtime packed form and other runtime packed forms associated with a light field probe into a video stream.
4. The computer-implemented method of claim 3, further comprising, decompressing at least a portion of the assembled video stream by performing steps including: selecting, for a video frame to be displayed to a user, one or more portions of faces of the light field probe that are visible; determining one or more of the visible portions that need to be updated; building indices to index the assembled video stream; determining one or more portions of the video stream to decompress based, at least in part, on the indices and the one or more of the visible portions that need to be updated; and spawning one or more thread groups for the determined one or more portions of the video stream, wherein each of the one or more thread groups decompresses a respective pixel tile in the determined one or more portions of the video stream in parallel with the other thread groups.
5. The computer-implemented method of claim 4, wherein each thread in each thread group of the one or more thread groups reads from the same memory and performs the same decompression operations as other threads in the thread group.
6. The computer-implemented method of claim 1, wherein: the quadtree is generated from bottom to top; and when the quadtree is being generated, a node of the quadtree is approximated only if its child nodes in the determined mask can be approximated by one of the plurality of functions within a predefined error.
7. The computer-implemented method of claim 1, wherein the one of the plurality of functions used to approximate the portion of the received depth image frame corresponding to each node of the quadtree is selected by performing steps including: determining, for each function of the plurality of functions, a respective maximum absolute distance in log-space between the portion of the received depth image frame and an approximation of the portion of the received depth image frame using the function; retaining one or more of the plurality of functions associated with determined maximum absolute distances that are less than a threshold value; determining, for each function of the retained functions, a respective mean-squared error between the portion of the received depth image frame and the approximation of the portion of the received depth image frame using the function; and selecting the one of the plurality of functions based, at least in part, on the determined mean-squared errors for the retained functions.
8. The computer-implemented method of claim 1, wherein the plurality of functions includes at least one of a raw, a platelet, a biquadratic, and a BC4-Wedged (BC4w) function.
9. The computer-implemented method of claim 8, wherein: the BC4w function approximates depth data for a given pixel tile using a line and two pairs of endpoints; and for each pixel of the given pixel tile in the BC4w function approximation: one of the pairs of endpoints is selected based on a halfspace in which the given pixel lies with respect to the line, and a depth value in the depth data for the pixel is snapped to a nearest palette index associated with the selected pair of endpoints.
10. The computer-implemented method of claim 9, wherein the BC4w function is evaluated by sampling a parameter space to determine optimal values for the line, two depth ranges represented by the pairs of endpoint, and a respective palette index associated with one of the pairs of endpoints for each pixel of the given pixel tile.
11. A non-transitory computer-readable storage medium storing a program, which, when executed by a processor performs operations comprising: determining a mask for a received depth image frame based, at least in part, on differences between the received depth image frame and one or more previous depth image frames; and generating a quadtree data structure that includes nodes associated with pixels of the depth image frame masked by the determined mask, wherein each node of the generated quadtree indicates one of a plurality of functions used to approximate a corresponding portion of the received depth image frame and coefficient values used in the approximation.
12. The computer-readable storage medium of claim 11, the operations further comprising, generating a runtime packed form of the quadtree by: traversing the quadtree top-to-bottom and depth-first; and copying nodes of the quadtree that are able to approximate corresponding portions of the received depth image within an error to the runtime packed form.
13. The computer-readable storage medium of claim 12, the operations further comprising, assembling the generated runtime packed form and other runtime packed forms associated with a light field probe into a video stream.
14. The computer-readable storage medium of claim 13, the operations further comprising, decompressing at least a portion of the assembled video stream by performing steps including: selecting, for a video frame to be displayed to a user, one or more portions of faces of the light field probe that are visible; determining one or more of the visible portions that need to be updated; building indices to index the assembled video stream; determining one or more portions of the video stream to decompress based, at least in part, on the indices and the one or more of the visible portions that need to be updated; and spawning one or more thread groups for the determined one or more portions of the video stream, wherein each of the one or more thread groups decompresses a respective pixel tile in the determined one or more portions of the video stream in parallel with the other thread groups.
15. The computer-readable storage medium of claim 11, wherein: the quadtree is generated from bottom to top; and when the quadtree is being generated, a node of the quadtree is approximated only if its child nodes in the determined mask can be approximated by one of the plurality of functions within a predefined error.
16. The computer-readable storage medium of claim 11, wherein the one of the plurality of functions used to approximate the portion of the received depth image frame corresponding to each node of the quadtree is selected by performing steps including: determining, for each function of the plurality of functions, a respective maximum absolute distance in log-space between the portion of the received depth image frame and an approximation of the portion of the received depth image frame using the function; retaining one or more of the plurality of functions associated with determined maximum absolute distances that are less than a threshold value; determining, for each function of the retained functions, a respective mean-squared error between the portion of the received depth image frame and the approximation of the portion of the received depth image frame using the function; and selecting the one of the plurality of functions based, at least in part, on the determined mean-squared errors for the retained functions.
17. The computer-readable storage medium of claim 11, wherein the plurality of functions includes at least one of a raw, a platelet, a biquadratic, and a BC4-Wedged (BC4w) function.
18. The computer-readable storage medium of claim 17, wherein: the BC4w function approximates depth data for a given pixel tile using a line and two pairs of endpoints; and for each pixel of the given pixel tile in the BC4w function approximation: one of the pairs of endpoints is selected based on a halfspace in which the given pixel lies with respect to the line, and a depth value in the depth data for the pixel is snapped to a nearest palette index associated with the selected pair of endpoints.
19. The computer-readable storage medium of claim 18, wherein the BC4w function is evaluated by sampling a parameter space to determine optimal values for the line, two depth ranges represented by the pairs of endpoint, and a respective palette index associated with one of the pairs of endpoints for each pixel of the given pixel tile.
20. A computer-implemented method for compressing a pixel tile in a depth image, comprising: sampling a parameter space of values for a line, two pairs of endpoints on respective sides of the line, and a respective palette index for each pixel of the pixel tile; determining, based on the sampling, values for the line, the two pairs of endpoints, and the respective palette index for each pixel that minimize an error, wherein the respective palette index that is determined for each pixel is associated with one of the two pairs of endpoints that are determined; and storing the determined values for the line, the two pairs of endpoints, and the respective palette index for each pixel.