- Number
- 10572979
- Published
- 2020-02-25
- Filed
- 2018-04-05
- Assignee
- Pixar
- Inventors
- Vogels; Thijs, Rousselle; Fabrice, McWilliams; Brian, Meyer; Mark, Novak; Jan
- CPC
- G06F18/2113; G06F18/2148; G06F18/2413; G06N3/045; G06N3/0464; G06N3/047; G06N3/0475; G06N3/084; G06N3/09; G06N3/094; G06N3/096; G06T15/06; G06T5/50; G06T5/60; G06T5/70; G06T7/0002; G06V10/454; G06V10/72; G06V10/764; G06V10/7747; G06V10/82
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Neural-network Monte Carlo render denoising (Pixar).
Abstract
Supervised machine learning using neural networks is applied to denoising images rendered by MC path tracing. Specialization of neural networks may be achieved by using a modular design that allows reusing trained components in different networks and facilitates easy debugging and incremental building of complex structures. Specialization may also be achieved by using progressive neural networks. In some embodiments, training of a neural-network based denoiser may use importance sampling, where more challenging patches or patches including areas of particular interests within a training dataset are selected with higher probabilities than others. In some other embodiments, generative adversarial networks (GANs) may be used for training a machine-learning based denoiser as an alternative to using pre-defined loss functions.
Background
BACKGROUND(1) Monte Carlo (MC) path tracing is a technique for rendering images of three-dimensional scenes by tracing paths of light through pixels on an image plane. This technique is capable of producing high quality images that are nearly indistinguishable from photographs. In MC path tracing, the color of a pixel is computed by randomly sampling light paths that connect the camera to light sources through multiple interactions with the scene. The mean intensity of many such samples constitutes a noisy estimate of the total illumination of the pixel. Unfortunately, in realistic scenes with complex light transport, these samples might have large variance, and the variance of their mean only decreases linearly with respect to the number of samples per pixel. Typically, thousands of samples per pixel are required to achieve a visually converged rendering. This can result in prohibitively long rendering times. Therefore, there is a need to reduce the number of samples needed for MC path tracing while still producing high-quality images.SUMMARY(2) Supervised machine learning using neural networks is applied to denoising images rendered by MC path tracing. Specialization of neural networks may be achieved by using a modular design that allows reusing trained components in different networks and facilitates easy debugging and incremental building of complex structures. Specialization may also be achieved by using progressive neural networks. In some embodiments, training of a ne
Claims
1. A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising: receiving a set of input images rendered by MC path tracing and a set of reference images, each reference image corresponding to a respective input image; configuring a neural network comprising: an input layer configured to receive the set of input images; a plurality of hidden layers, each hidden layer having a respective number of nodes, each node associated with a respective parameter, a first layer of the plurality of hidden layers coupled to the input layer; and an output layer coupled to a last layer of the plurality of hidden layers and configured to output a respective denoised image corresponding to a respective input image; training the neural network using the set of input images and the set of reference images, the training comprising: obtaining one or more image metrics for each respective input image of the set of input images or for a reference image corresponding to the respective input image; selecting a first input image among the set of input images according to a probability function based on the one or more image metrics; performing a first iteration of the training using the first input image and a corresponding first reference image to obtain a first intermediate set of parameters associated with the nodes of the plurality of hidden layers; selecting additional input images among the set of input images according to the probability function; and performing additional iterations of the training using each of the additional input images and a corresponding reference image to obtain a final set of parameters associated with the nodes of the plurality of hidden layers; receiving a new input image rendered by MC path tracing; and generating a new denoised image corresponding to the new input image by passing the new input image through the neural network using the final set of parameters.
11. A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising: receiving a set of input images rendered by MC path tracing and a set of reference images, each reference image corresponding to a respective input image; configuring a neural network comprising: an input layer configured to receive the set of input images; a plurality of hidden layers, each hidden layer having a respective number of nodes, each node associated with a respective parameter, a first layer of the plurality of hidden layers coupled to the input layer; and an output layer coupled to a last layer of the plurality of hidden layers, the output layer configured to output a respective denoised image corresponding to a respective input image; training the neural network using the set of input images and the set of reference images, the training comprising: performing one or more initial iterations of the training by randomly sampling the set of input images using a uniform probability to obtain a set of intermediate denoised images, each intermediate denoised image corresponding to a respective input image; for each respective input image, evaluating an error gradient by comparing a corresponding intermediate denoised image to the respective input image; and performing one or more additional iterations of the training by sampling the set of input images according to a probability function based on the error gradient of each input image of the set of input images to obtain a set of optimized parameters associated with the nodes of the plurality of hidden layers; receiving a new input image rendered by MC path tracing; and generating a new denoised image corresponding to the new input image by passing the new input image through the neural network using the set of optimized parameters.
16. A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising: receiving a set of input images rendered by MC path tracing and a set of reference images, each reference image corresponding to a respective input image; configuring a neural network comprising: an input layer configured to receive the set of input images; a plurality of hidden layers, each hidden layer having a respective number of nodes, each node associated with a respective parameter, a first layer of the plurality of hidden layers coupled to the input layer; and an output layer coupled to a last layer of the plurality of hidden layers, the output layer configured to output a respective denoised image corresponding to a respective input image; training the neural network using the set of input images and the set of reference images, the training comprising: assigning a relevance score to each respective input image of the set of input images, the relevance score indicating a degree of relevance to one or more areas of interests; and performing the training by sampling the set of input images according to a probability function that is proportional to the relevance score of each respective input image to obtain a set of optimized parameters associated with the nodes of the plurality of hidden layers; receiving a new input image rendered by MC path tracing; and generating a denoised image corresponding to the new input image by passing the new input image through the neural network using the set of optimized parameters.