Denoising Monte Carlo renderings with neural networks (Pixar).
A modular architecture is provided for denoising Monte Carlo renderings using neural networks. The temporal approach extracts and combines feature representations from neighboring frames rather than building a temporal context using recurrent connections. A multiscale architecture includes separate single-frame or temporal denoising modules for individual scales, and one or more scale compositor neural networks configured to adaptively blend individual scales. An error-predicting module is configured to produce adaptive sampling maps for a renderer to achieve more uniform residual noise distribution. An asymmetric loss function may be used for training the neural networks, which can provide control over the variance-bias trade-off during denoising.
The following four U.S. patent applications (including this one) are being filed concurrently, and the entire disclosure of the other application is incorporated by reference into this application for all purposes:
application Ser. No. ______, filed on Jul. 31, 2018, entitled “TEMPORAL TECHNIQUES OF DENOISING MONTE CARLO RENDERINGS USING NEURAL NETWORKS” (Attorney Docket No. 088298-P18074US1-1085252),
application Ser. No. ______, filed on Jul. 31, 2018, entitled “MULTI-SCALE ARCHITECTURE OF DENOISING MONTE CARLO RENDERINGS USING NEURAL NETWORKS” (Attorney Docket No. 088298-P18074US2-1087809),
application Ser. No. ______, filed on Jul. 31, 2018, entitled “DENOISING MONTE CARLO RENDERINGS USING NEURAL NETWORKS WITH ASYMMETRIC LOSS” (Attorney Docket No. 088298-P18074US3-1087810), and
application Ser. No. ______, filed on Jul. 31, 2018, entitled “ADAPTIVE SAMPLING IN MONTE CARLO RENDERINGS USING ERROR-PREDICTING NEURAL NETWORKS” (Attorney Docket No. 088298-P18074US4-1087811).BACKGROUND
Monte Carlo (MC) path tracing is a technique for rendering images of three-dimensional scenes by tracing paths of light through pixels on an image plane. This technique is capable of producing high quality images that are nearly indistinguishable from photographs. In MC path tracing, the color of a pixel is computed by randomly sampling light paths that connect the camera to light sources through multiple interactions with the scene. The mean intensity of many s
1. A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising: receiving an input image rendered by MC path tracing, the input image including a plurality of pixels and input color data for each pixel; receiving a reference image corresponding to the input image, the reference image including the plurality of pixels and reference color data for each pixel; receiving a proportionality constant factor for each pixel, the proportionality constant factor being a positive real number; configuring a neural network comprising: an input layer configured to receive the input image; a plurality of hidden layers including a plurality of nodes, a first layer of the plurality of hidden layers coupled to the input layer; and an output layer coupled to a last layer of the plurality of hidden layers and configured to output an output image corresponding to the input image, the output image comprising the plurality of pixels and including output color data for each pixel; and training the neural network using the input image and the reference image, the training comprising: performing a first iteration of the training to obtain a first output image and an intermediate set of parameters associated with the plurality of nodes of the plurality of hidden layers; determining values of a loss function for the plurality of pixels by: for each respective pixel of the plurality of pixels: determining a first difference between output color data and reference color data for the respective pixel; determining a second difference between input color data and reference color data for the respective pixel; upon determining that the first difference and the second difference have a same sign, assigning a first respective value of the loss function for the respective pixel that relates to an absolute value of the first difference multiplied by a first proportionality constant of unity; and upon determining that the first difference and the second difference have opposite signs, assigning a second respective value of the loss function for the respective pixel that relates to the absolute value of the first difference multiplied by a second proportionality constant that is equal to unity plus the proportionality constant factor for the respective pixel; back-propagating a gradient of the loss function through the neural network to obtain an updated intermediate set of parameters associated with the plurality of nodes of the plurality of hidden layers; and performing additional iterations of the training to obtain a final output image and an optimized set of parameters associated with the plurality of nodes of the plurality of hidden layers.
2. The method of claim 1, further comprising: receiving a new input image rendered by MC path tracing; and generating a denoised image corresponding to the new input image by passing the new input image through the neural network using the optimized set of parameters.
3. The method of claim 1, wherein the proportionality constant factor varies from pixel to pixel.
4. The method of claim 1, wherein the proportionality constant factor is same for all pixels.
5. The method of claim 1, wherein the neural network comprises a convolutional neural network.
6. The method of claim 5, wherein the neural network comprises: a kernel prediction module configured to generate a set of weights associated with a neighborhood of pixels around each pixel; and a reconstruction module configured to reconstruct the output image using the set of weights.
7. The method of claim 1, wherein the neural network comprises a multilayer perceptron neural network.
8. The method of claim 1, wherein the loss function comprises relative squared error (rSE).
9. The method of claim 1, wherein the loss function comprises symmetric absolute percentage error (SAPE).
10. The method of claim 1, wherein the loss function comprises structural similarity (SSIM) error or multi-scale structural similarity (MS-SSIM) error.