- Number
- 11037274
- Published
- 2021-06-15
- Filed
- 2020-02-12
- Assignee
- Pixar
- Inventors
- Vogels; Thijs, Rousselle; Fabrice, McWilliams; Brian, Meyer; Mark, Novak; Jan
- CPC
- G06F18/2113; G06F18/2148; G06F18/2413; G06N3/045; G06N3/0464; G06N3/047; G06N3/0475; G06N3/084; G06N3/09; G06N3/094; G06N3/096; G06T15/06; G06T5/50; G06T5/60; G06T5/70; G06T7/0002; G06V10/454; G06V10/72; G06V10/764; G06V10/7747; G06V10/82
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Progressive-NN Monte Carlo render denoising (Pixar, granted).
Abstract
Supervised machine learning using neural networks is applied to denoising images rendered by MC path tracing. Specialization of neural networks may be achieved by using a modular design that allows reusing trained components in different networks and facilitates easy debugging and incremental building of complex structures. Specialization may also be achieved by using progressive neural networks. In some embodiments, training of a neural-network based denoiser may use importance sampling, where more challenging patches or patches including areas of particular interests within a training dataset are selected with higher probabilities than others. In some other embodiments, generative adversarial networks (GANs) may be used for training a machine-learning based denoiser as an alternative to using pre-defined loss functions.
Background
BACKGROUND (1) Monte Carlo (MC) path tracing is a technique for rendering images of three-dimensional scenes by tracing paths of light through pixels on an image plane. This technique is capable of producing high quality images that are nearly indistinguishable from photographs. In MC path tracing, the color of a pixel is computed by randomly sampling light paths that connect the camera to light sources through multiple interactions with the scene. The mean intensity of many such samples constitutes a noisy estimate of the total illumination of the pixel. Unfortunately, in realistic scenes with complex light transport, these samples might have large variance, and the variance of their mean only decreases linearly with respect to the number of samples per pixel. Typically, thousands of samples per pixel are required to achieve a visually converged rendering. This can result in prohibitively long rendering times. Therefore, there is a need to reduce the number of samples needed for MC path tracing while still producing high-quality images. SUMMARY (2) Supervised machine learning using neural networks is applied to denoising images rendered by MC path tracing. Specialization of neural networks may be achieved by using a modular design that allows reusing trained components in different networks and facilitates easy debugging and incremental building of complex structures. Specialization may also be achieved by using progressive neural networks. In some embodiments, training of a
Claims
1. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that when executed control a computer system to denoise images rendered by Monte Carlo (MC) path tracing, the instructions comprising: receiving a first set of input images rendered by MC path tracing, and a first set of corresponding reference images; receiving a second set of input images rendered by MC path tracing, and a second set of corresponding reference images; configuring a first neural network including a first plurality of layers and a first number of nodes associated with a first number of parameters, the first neural network configured to receive the first set of input images; configuring a second neural network including a second plurality of layers and a second number of nodes associated with a second number of parameters, the second neural network configured to receive the second set of input images; configuring a third neural network including a third plurality of layers and a third number of nodes associated with a third number of parameters, the third neural network configured to: receive output from one of the first neural network and the second neural network; and output an output image corresponding to a respective input image; training the first neural network, the second neural network, and the third neural network to obtain a first number of optimized parameters associated with the first number of nodes of the first neural network, a second number of optimized parameters associated with the second number of nodes of the second neural network, and a third number of optimized parameters associated with the third number of nodes of the third neural network, wherein the training uses the first set of input images and the first set of reference images while the third neural network receives output from the first neural network, and uses the second set of input images and the second set of reference images while the third neural network receives output from the second neural network; receiving a third set of input images rendered by MC path tracing and a third set of corresponding reference images; configuring a fourth neural network including a fourth plurality of layers and a fourth number of nodes associated with a fourth number of parameters, the fourth neural network configured to receive the third set of input images, wherein the third neural network is further configured to receive output from the fourth neural network; and training the fourth neural network using the third set of input images and the third set of reference images to obtain a fourth number of optimized parameters associated with the fourth number of nodes of the fourth neural network, while the third neural network is configured to receive output from the fourth neural network and the third number of optimized parameters associated with the third number of nodes of the third neural network are fixed. ||
11. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that when executed control a computer system to denoise images rendered by Monte Carlo (MC) path tracing, the instructions comprising: receiving a first set of input images rendered by MC path tracing, and a first set of corresponding reference images; configuring a first neural network including a first plurality of layers and a first number of nodes associated with a first number of parameters, the first neural network configured to receive the first set of input images; configuring a second neural network including a second plurality of layers and a second number of nodes associated with a second number of parameters, the second neural network configured to receive output from the first neural network and output an output image corresponding to a respective input image; training the first neural network and the second neural network using the first set of input images and the first set of reference images to obtain a first number of optimized parameters associated with the first number of nodes of the first neural network and a second number of optimized parameters associated with the second number of nodes of the second neural network; receiving a second set of input images rendered by MC path tracing and a second set of corresponding reference images; configuring a third neural network including a third plurality of layers and a third number of nodes associated with a third number of parameters, the third neural network configured to receive the second set of input images, wherein the second neural network is further configured to receive output from the third neural network; and training the third neural network using the second set of input images and the second set of reference images to obtain a third number of optimized parameters associated with the third number of nodes of the third neural network, while the second neural network is configured to receive output from the third neural network and the second number of optimized parameters associated with the second number of nodes of the second neural network are fixed. ||
16. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that when executed control a computer system to denoise images rendered by Monte Carlo (MC) path tracing, the instructions comprising: receiving a first set of input images rendered by MC path tracing, and a first set of corresponding reference images; configuring a first neural network comprising: a first input layer configured to receive the first set of input images; a plurality of first hidden layers, each first hidden layer having a respective number of first nodes associated with a respective number of first parameters, a first layer of the plurality of first hidden layers coupled to the first input layer, each respective first hidden layer receiving input from a preceding first hidden layer; and a first output layer coupled to a last layer of the plurality of first hidden layers, the first output layer configured to output a respective output image corresponding to a respective input image; training the first neural network using the first set of input images and the first set of reference images to obtain a number of optimized first parameters associated with the number of first nodes of each of the plurality of first hidden layers; receiving a second set of input images rendered by MC path tracing, and a second set of corresponding reference images; configuring a second neural network including: a second input layer configured to receive the second set of input images; a plurality of second hidden layers, each second hidden layer corresponding to a respective first hidden layer and having a respective number of second nodes associated with a respective number of second parameters, a first layer of the plurality of second hidden layers coupled to the second input layer and the first input layer of the first neural network, each respective second hidden layer receiving input from a preceding second hidden layer and a corresponding first hidden layer of the first neural network; and a second output layer coupled to a last layer of the plurality of second hidden layers and the last layer of the plurality of first hidden layers of the first neural network, the second output layer configured to output a respective output image corresponding to a respective input image; and training the second neural network using the second set of input images and the second set of reference images to obtain a number of optimized second parameters associated with the number of second nodes of each of the plurality of second hidden layers, while the number of optimized first parameters associated with the number of first nodes of each of the plurality of first hidden layers of the first neural network are fixed.