Neural-network Monte Carlo render denoising, VFX rendering research.
A modular architecture is provided for denoising Monte Carlo renderings using neural networks. The temporal approach extracts and combines feature representations from neighboring frames rather than building a temporal context using recurrent connections. A multiscale architecture includes separate single-frame or temporal denoising modules for individual scales, and one or more scale compositor neural networks configured to adaptively blend individual scales. An error-predicting module is configured to produce adaptive sampling maps for a renderer to achieve more uniform residual noise distribution. An asymmetric loss function may be used for training the neural networks, which can provide control over the variance-bias trade-off during denoising.
(1) The following four U.S. patent applications (including this one) are being filed concurrently, and the entire disclosure of the other application is incorporated by reference into this application for all purposes:(2) application Ser. No. 16/050,314, filed on Jul. 31, 2018, entitled “TEMPORAL TECHNIQUES OF DENOISING MONTE CARLO RENDERINGS USING NEURAL NETWORKS”,(3) U.S. Pat. No. 16,050,332, filed on Jul. 31, 2018, entitled “MULTI-SCALE ARCHITECTURE OF DENOISING MONTE CARLO RENDERINGS USING NEURAL NETWORKS”,(4) application Ser. No. 16/050,336, filed on Jul. 31, 2018, entitled “DENOISING MONTE CARLO RENDERINGS USING NEURAL NETWORKS WITH ASYMMETRIC LOSS”, and(5) application Ser. No. 16/050,362, filed on Jul. 31, 2018, entitled “ADAPTIVE SAMPLING IN MONTE CARLO RENDERINGS USING ERROR-PREDICTING NEURAL NETWORKS”.BACKGROUND(6) Monte Carlo (MC) path tracing is a technique for rendering images of three-dimensional scenes by tracing paths of light through pixels on an image plane. This technique is capable of producing high quality images that are nearly indistinguishable from photographs. In MC path tracing, the color of a pixel is computed by randomly sampling light paths that connect the camera to light sources through multiple interactions with the scene. The mean intensity of many such samples constitutes a noisy estimate of the total illumination of the pixel. Unfortunately, in realistic scenes with complex light transport, these samples might have large variance, and the va
1. A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising: receiving an input image rendered by MC path tracing, the input image including a plurality of pixels and input color data for each pixel; receiving a reference image corresponding to the input image, the reference image including the plurality of pixels and reference color data for each pixel; receiving a proportionality constant factor for each pixel, the proportionality constant factor being a positive real number; configuring a neural network comprising: an input layer configured to receive the input image; a plurality of hidden layers including a plurality of nodes, a first layer of the plurality of hidden layers coupled to the input layer; and an output layer coupled to a last layer of the plurality of hidden layers and configured to output an output image corresponding to the input image, the output image comprising the plurality of pixels and including output color data for each pixel; and training the neural network using the input image and the reference image, the training comprising: performing a first iteration of the training to obtain a first output image and an intermediate set of parameters associated with the plurality of nodes of the plurality of hidden layers; determining values of a loss function for the plurality of pixels by: for each respective pixel of the plurality of pixels: determining a first difference between output color data and reference color data for the respective pixel; determining a second difference between input color data and reference color data for the respective pixel; upon determining that the first difference and the second difference have a same sign, assigning a first respective value of the loss function for the respective pixel that relates to an absolute value of the first difference multiplied by a first proportionality constant of unity; and upon determining that the first difference and the second difference have opposite signs, assigning a second respective value of the loss function for the respective pixel that relates to the absolute value of the first difference multiplied by a second proportionality constant that is equal to unity plus the proportionality constant factor for the respective pixel; back-propagating a gradient of the loss function through the neural network to obtain an updated intermediate set of parameters associated with the plurality of nodes of the plurality of hidden layers; and performing additional iterations of the training to obtain a final output image and an optimized set of parameters associated with the plurality of nodes of the plurality of hidden layers.
11. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that when executed control a computer system to denoise images rendered by Monte Carlo (MC) path tracing, the instructions comprising: receiving an input image rendered by MC path tracing, the input image including a plurality of pixels and input color data for each pixel; receiving a reference image corresponding to the input image, the reference image including the plurality of pixels and reference color data for each pixel; receiving a proportionality constant factor for each pixel, the proportionality constant factor being a positive real number; configuring a neural network comprising: an input layer configured to receive the input image; a plurality of hidden layers including a plurality of nodes, a first layer of the plurality of hidden layers coupled to the input layer; and an output layer coupled to a last layer of the plurality of hidden layers and configured to output an output image corresponding to the input image, the output image comprising the plurality of pixels and including output color data for each pixel; and training the neural network using the input image and the reference image, the training comprising: performing a first iteration of the training to obtain a first output image and an intermediate set of parameters associated with the plurality of nodes of the plurality of hidden layers; determining values of a loss function for the plurality of pixels by: for each respective pixel of the plurality of pixels: determining a first difference between output color data and reference color data for the respective pixel; determining a second difference between input color data and reference color data for the respective pixel; upon determining that the first difference and the second difference have a same sign, assigning a first respective value of the loss function for the respective pixel that relates to an absolute value of the first difference multiplied by a first proportionality constant of unity; and upon determining that the first difference and the second difference have opposite signs, assigning a second respective value of the loss function for the respective pixel that relates to the absolute value of the first difference multiplied by a second proportionality constant that is equal to unity plus the proportionality constant factor for the respective pixel; back-propagating a gradient of the loss function through the neural network to obtain an updated intermediate set of parameters associated with the plurality of nodes of the plurality of hidden layers; and performing additional iterations of the training to obtain a final output image and an optimized set of parameters associated with the plurality of nodes of the plurality of hidden layers.