Multi-scale architecture of denoising monte carlo renderings using neural networks

Number: 10672109
Published: 2020-06-02
Filed: 2018-07-31
Assignee: Pixar
Inventors: Vogels; Thijs, Rousselle; Fabrice, Novak; Jan, McWilliams; Brian, Meyer; Mark, Harvill; Alex
CPC: G06F17/18; G06N20/00; G06N3/044; G06N3/045; G06N3/0455; G06N3/0464; G06N3/084; G06N3/09; G06N3/096; G06N5/046; G06N7/01; G06T15/06; G06T15/506; G06T5/50; G06T5/60; G06T5/70
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Neural-network Monte Carlo render denoising (Pixar, VFX rendering research).

Abstract

A modular architecture is provided for denoising Monte Carlo renderings using neural networks. The temporal approach extracts and combines feature representations from neighboring frames rather than building a temporal context using recurrent connections. A multiscale architecture includes separate single-frame or temporal denoising modules for individual scales, and one or more scale compositor neural networks configured to adaptively blend individual scales. An error-predicting module is configured to produce adaptive sampling maps for a renderer to achieve more uniform residual noise distribution. An asymmetric loss function may be used for training the neural networks, which can provide control over the variance-bias trade-off during denoising.

Background

BACKGROUND(1) Monte Carlo (MC) path tracing is a technique for rendering images of three-dimensional scenes by tracing paths of light through pixels on an image plane. This technique is capable of producing high quality images that are nearly indistinguishable from photographs. In MC path tracing, the color of a pixel is computed by randomly sampling light paths that connect the camera to light sources through multiple interactions with the scene. The mean intensity of many such samples constitutes a noisy estimate of the total illumination of the pixel. Unfortunately, in realistic scenes with complex light transport, these samples might have large variance, and the variance of their mean only decreases linearly with respect to the number of samples per pixel. Typically, thousands of samples per pixel are required to achieve a visually converged rendering. This can result in prohibitively long rendering times. Therefore, there is a need to reduce the number of samples needed for MC path tracing while still producing high-quality images.SUMMARY(2) A modular architecture is provided for denoising Monte Carlo renderings using neural networks. A source-aware encoding module may be configured to extract low-level features and embed them into a feature space common between sources, which may allow for quickly adapting a trained network to novel data. A spatial module may be configured to extract abstract, high-level features for reconstruction.(3) According to some embodiments, a t

Claims

1. A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising: receiving an input image rendered by MC path tracing and a corresponding reference image, the input image including a set of first color buffers, each first color buffer including a first number of rows and a first number of columns of pixels; generating a down-sampled image corresponding to the input image by down-sampling the input image, the down-sampled image including a set of second color buffers, each second color buffer including a second number of rows and a second number of columns of pixels, the second number of rows being less than the first number of rows, and the second number of columns being less than the first number of columns; configuring a neural network including a plurality of nodes, the neural network configured to: receive the input image and the down-sampled image; generate a first denoised image corresponding to the input image, the first denoised image including the first number of rows and the first number of columns of pixels; generate a second denoised image corresponding to the down-sampled image, the second denoised image including the second number of rows and the second number of columns of pixels; generate a set of per-pixel weights for each pixel of the first number of rows and the first number of columns of pixels; and blend the first denoised image and the second denoised image to obtain a final denoised image using the set of per-pixel weights, the final denoised image including the first number of rows and the first number of columns of pixels; and training the neural network to obtain a plurality of optimized parameters associated with the plurality of nodes of the neural network, wherein the training uses the input image and the corresponding reference image. 11. A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising: receiving a sequence of frames rendered by MC path tracing, the sequence of frames including a center frame and one or more temporal neighboring frames, each frame including a first number of rows and a first number of columns of pixels; receiving a reference image corresponding to the center frame; configuring a plurality of first neural networks, each respective first neural network configured to extract a respective first set of features from a respective frame of the sequence of frames; configuring a second neural network, the second neural network configured to: extract a first set of temporal features from the first sets of features; and output a first denoised frame corresponding to the center frame based on the first set of temporal features, the first denoised frame including the first number of rows and the first number of columns of pixels; for each respective frame of the sequence of frames: generating a respective down-sampled frame by down-sampling the respective frame, the respective down-sampled frame including a second number of rows and a second number of columns of pixels, the second number of rows being less than the first number of rows, and the second number of columns being less than the first number of columns; configuring a plurality of third neural networks, each respective third neural network configured to extract a respective second set of features from a respective down-sampled frame; configuring a fourth neural network, the fourth neural network configured to: extract a second set of temporal features from the second sets of features; and output a second denoised frame corresponding to the center frame based on the second set of temporal features, the second denoised frame including the second number of rows and the second number of columns of pixels; configuring a fifth neural network including a plurality of layers and a plurality of nodes, the fifth neural network configured to: receive the first denoised frame and the second denoised frame; generate a set of per-pixel weights for each pixel of the first number of rows and the first number of columns of pixels; and blend the first denoised frame and the second denoised frame to obtain a final denoised frame using the set of per-pixel weights, the final denoised frame including the first number of rows and the first number of columns of pixels; and training the fifth neural network to obtain a plurality of optimized parameters associated with the plurality of nodes of the fifth neural network, wherein the training uses the sequence of frames and the reference image. 21. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that when executed control a computer system to denoise images rendered by Monte Carlo (MC) path tracing, the instructions comprising: receiving an input image rendered by MC path tracing and a corresponding reference image, the input image including a set of first color buffers, each first color buffer including a first number of rows and a first number of columns of pixels; generating a down-sampled image corresponding to the input image by down-sampling the input image, the down-sampled image including a set of second color buffers, each second color buffer including a second number of rows and a second number of columns of pixels, the second number of rows being less than the first number of rows, and the second number of columns being less than the first number of columns; configuring a neural network including a plurality of nodes, the neural network configured to: receive the input image and the down-sampled image; generate a first denoised image corresponding to the input image, the first denoised image including the first number of rows and the first number of columns of pixels; generate a second denoised image corresponding to the down-sampled image, the second denoised image including the second number of rows and the second number of columns of pixels; generate a set of per-pixel weights for each pixel of the first number of rows and the first number of columns of pixels; and blend the first denoised image and the second denoised image to obtain a final denoised image using the set of per-pixel weights, the final denoised image including the first number of rows and the first number of columns of pixels; and training the neural network to obtain a plurality of optimized parameters associated with the plurality of nodes of the neural network, wherein the training uses the input image and the corresponding reference image. 27. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that when executed control a computer system to denoise images rendered by Monte Carlo (MC) path tracing, the instructions comprising: receiving a sequence of frames rendered by MC path tracing, the sequence of frames including a center frame and one or more temporal neighboring frames, each frame including a first number of rows and a first number of columns of pixels; receiving a reference image corresponding to the center frame; configuring a plurality of first neural networks, each respective first neural network configured to extract a respective first set of features from a respective frame of the sequence of frames; configuring a second neural network, the second neural network configured to: extract a first set of temporal features from the first sets of features; and output a first denoised frame corresponding to the center frame based on the first set of temporal features, the first denoised frame including the first number of rows and the first number of columns of pixels; for each respective frame of the sequence of frames: generating a respective down-sampled frame by down-sampling the respective frame, the respective down-sampled frame including a second number of rows and a second number of columns of pixels, the second number of rows being less than the first number of rows, and the second number of columns being less than the first number of columns; configuring a plurality of third neural networks, each respective third neural network configured to extract a respective second set of features from a respective down-sampled frame; configuring a fourth neural network, the fourth neural network configured to: extract a second set of temporal features from the second sets of features; and output a second denoised frame corresponding to the center frame based on the second set of temporal features, the second denoised frame including the second number of rows and the second number of columns of pixels; configuring a fifth neural network including a plurality of layers and a plurality of nodes, the fifth neural network configured to: receive the first denoised frame and the second denoised frame; generate a set of per-pixel weights for each pixel of the first number of rows and the first number of columns of pixels; and blend the first denoised frame and the second denoised frame to obtain a final denoised frame using the set of per-pixel weights, the final denoised frame including the first number of rows and the first number of columns of pixels; and training the fifth neural network to obtain a plurality of optimized parameters associated with the plurality of nodes of the fifth neural network, wherein the training uses the sequence of frames and the reference image.