GAN-based Monte Carlo render denoising (Pixar).
Supervised machine learning using neural networks is applied to denoising images rendered by MC path tracing. Specialization of neural networks may be achieved by using a modular design that allows reusing trained components in different networks and facilitates easy debugging and incremental building of complex structures. Specialization may also be achieved by using progressive neural networks. In some embodiments, training of a neural-network based denoiser may use importance sampling, where more challenging patches or patches including areas of particular interests within a training dataset are selected with higher probabilities than others. In some other embodiments, generative adversarial networks (GANs) may be used for training a machine-learning based denoiser as an alternative to using pre-defined loss functions.
BACKGROUND(1) Monte Carlo (MC) path tracing is a technique for rendering images of three-dimensional scenes by tracing paths of light through pixels on an image plane. This technique is capable of producing high quality images that are nearly indistinguishable from photographs. In MC path tracing, the color of a pixel is computed by randomly sampling light paths that connect the camera to light sources through multiple interactions with the scene. The mean intensity of many such samples constitutes a noisy estimate of the total illumination of the pixel. Unfortunately, in realistic scenes with complex light transport, these samples might have large variance, and the variance of their mean only decreases linearly with respect to the number of samples per pixel. Typically, thousands of samples per pixel are required to achieve a visually converged rendering. This can result in prohibitively long rendering times. Therefore, there is a need to reduce the number of samples needed for MC path tracing while still producing high-quality images.SUMMARY(2) Supervised machine learning using neural networks is applied to denoising images rendered by MC path tracing. Specialization of neural networks may be achieved by using a modular design that allows reusing trained components in different networks and facilitates easy debugging and incremental building of complex structures. Specialization may also be achieved by using progressive neural networks. In some embodiments, training of a ne
1. A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising: receiving an input image rendered by MC path tracing and a corresponding reference image; configuring a generative adversarial network (GAN), the GAN comprising: a generator comprising a first neural network having a first set of parameters to be optimized, the generator configured to receive the input image and produce an output image corresponding to the input image using the first set of parameters; and a discriminator coupled to the generator, the discriminator comprising a second neural network having a second set of parameters to be optimized, the discriminator configured to: receive the input image, the reference image, and the output image produced by the generator; generate a quality metric based on a comparison of the output image or the reference image with the input image using the second set of parameters, the quality metric indicating a relative probability of the output image or the reference image belonging to a first class of denoised images as compared to a second class of ground truth images; and output the quality metric to the generator, wherein the generator is further configured to update the first set of parameters based on the quality metric and to produce an updated output image using the updated first set of parameters; training the GAN to obtain an optimized first set of parameters and an optimized second set of parameters, such that a statistical value of the quality metric generated by the discriminator approaches a predetermined value; receiving a new input image rendered by MC path tracing; and generating a denoised image corresponding to the new input image by passing the new input image through the generator using the optimized first set of parameters.
15. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that when executed control a computer system to denoise images rendered by Monte Carlo (MC) path tracing, the instructions comprising: receiving an input image rendered by MC path tracing and a corresponding reference image; configuring a generative adversarial network (GAN), the GAN comprising: a generator comprising a first neural network having a first set of parameters to be optimized, the generator configured to receive the input image and produce an output image corresponding to the input image using the first set of parameters; and a discriminator coupled to the generator, the discriminator comprising a second neural network having a second set of parameters to be optimized, the discriminator configured to: receive the input image, the reference image, and the output image produced by the generator; generate a quality metric based on a comparison of the output image or the reference image with the input image using the second set of parameters, the quality metric indicating a relative probability of the output image or the reference image belonging to a first class of denoised images as compared to a second class of ground truth images; and output the quality metric to the generator, wherein the generator is further configured to update the first set of parameters based on the quality metric and to produce an updated output image using the updated first set of parameters; training the GAN to obtain an optimized first set of parameters and an optimized second set of parameters, such that a statistical value of the quality metric generated by the discriminator approaches a predetermined value; receiving a new input image rendered by MC path tracing; and generating a denoised image corresponding to the new input image by passing the new input image through the generator using the optimized first set of parameters.