Hybrid Two-Dimensional And Three-Dimensional Denoiser

Number: 20240394850
Published: 2024-11-28
Filed: 2024-05-21
Assignee: Disney Enterprises, Inc.
Inventors: Papas; Marios et al.
CPC: G06T3/067; G06T5/70; G06T5/60; G06T9/00; G06T15/00; G06T19/20
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Hybrid 2D/3D rendering-denoising technique.

Abstract

A system includes a pre-processor configured to receive three-dimensional (3-D) image data, flatten the 3-D image data to produce corresponding two-dimensional (2-D) image data, and concatenate the 3-D image data and the corresponding 2-D image data to provide concatenated image data. The system further includes an encoder including one or more first neural networks (NNs), the encoder configured to use the one or more first NNs to encode the concatenated image data to provide encoded data, a decoder including one or more second NNs, the decoder configured to use the one or more second NNs to decode the encoded data to provide decoded data, and a reconstructor including a plurality of hybrid 2-D/3-D reconstructors configured to reconstruct the decoded data to provide a denoised 3-D output image corresponding to the 3-D image data.

Background

BACKGROUND

Compositing is an important step in the production of animated films and visual effects, in which different parts of a frame are post-processed and fine-tuned independently before being merged together. Three-dimensional (3-D) images, such as deep-Z images for example, contain a variable number of bins per pixel at different depths, each of which records the color and opacity, or “alpha,” at the corresponding depth. As a result, 3-D images can advantageously provide more accurate opacity and avoid edge artifacts in compositing because those 3-D images can cleanly separate distinct geometric boundaries in different bins.

However, path-traced 3-D images generated by renderers presently used in production suffer from the same problem as flat two-dimensional (2-D) images, i.e., noise. Noise reduces the quality of the compositing operations and increases the difficulty of achieving a desired artistic effect. The absence in the conventional art of a denoising solution for 3-D images that can compete with the quality of denoisers on flat 2-D images is one of the primary factors inhibiting the use of 3-D images in production. For example, the present state-of-the-art deep-Z image denoising approach, which filters each bin based on information from neighboring bins, produces artifacts such as residual noise or splotches and is computationally expensive.

Although it is possible to apply state-of-the-art neural network-based denoisers for flat 2-D images

Claims

1. A system comprising: a pre-processor configured to: receive three-dimensional (3-D) image data; flatten the 3-D image data to produce corresponding two-dimensional (2-D) image data; and concatenate the 3-D image data and the corresponding 2-D image data to provide concatenated image data; an encoder including one or more first neural networks (NNs), the encoder configured to use the one or more first NNs to encode the concatenated image data to provide encoded data; a decoder including one or more second NNs, the decoder configured to use the one or more second NNs to decode the encoded data to provide decoded data; and a reconstructor including a plurality of hybrid 2-D/3-D reconstructors configured to reconstruct the decoded data to provide a denoised 3-D output image corresponding to the 3-D image data. || 11. A method for use by a system to denoise three-dimensional (3-D) image data, the method comprising: receiving the 3-D image data; flattening the 3-D image data to produce a corresponding two-dimensional (2-D) image data; concatenating the 3-D image data and the corresponding 2-D image data to provide concatenated image data; encoding the concatenated image data to provide encoded data; decoding the encoded data to provide decoded data; and reconstructing the decoded data, using a plurality of hybrid 2-D/3-D reconstructors of the system, to provide a denoised 3-D output image corresponding to the 3-D image data.