KERNEL-PREDICTING CONVOLUTIONAL NEURAL NETWORKS FOR DENOISING

Number: 20200027198
Published: 2020-01-23
Filed: 2019-09-26
Assignee: Disney Enterprises, Inc.
Inventors: Vogels; Thijs, Novák; Jan, Rousselle; Fabrice, McWilliams; Brian
CPC: G06F17/10; G06F18/214; G06F18/24; G06N3/045; G06N3/0464; G06N3/08; G06N3/09; G06T15/005; G06T15/06; G06T15/50; G06T5/60; G06T5/70; G06V10/30; G06V10/764; G06V10/82
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Kernel-predicting CNN for render denoising (landmark Disney Research work).

Abstract

Supervised machine learning using convolutional neural network (CNN) is applied to denoising images rendered by MC path tracing. The input image data may include pixel color and its variance, as well as a set of auxiliary buffers that encode scene information (e.g., surface normal, albedo, depth, and their corresponding variances). In some embodiments, a CNN directly predicts the final denoised pixel value as a highly non-linear combination of the input features. In some other embodiments, a kernel-prediction neural network uses a CNN to estimate the local weighting kernels, which are used to compute each denoised pixel from its neighbors. In some embodiments, the input image can be decomposed into diffuse and specular components. The diffuse and specular components are then independently preprocessed, filtered, and postprocessed, before recombining them to obtain a final denoised image.

Background

BACKGROUND

Monte Carlo (MC) path tracing is a technique for rendering images of three-dimensional scenes by tracing paths of light through pixels on an image plane. This technique is capable of producing high quality images that are nearly indistinguishable from photographs. In MC path tracing, the color of a pixel is computed by randomly sampling light paths that connect the camera to light sources through multiple interactions with the scene. The mean intensity of many such samples constitutes a noisy estimate of the total illumination of the pixel. Unfortunately, in realistic scenes with complex light transport, these samples might have large variance, and the variance of their mean only decreases linearly with respect to the number of samples per pixel. Typically, thousands of samples per pixel are required to achieve a visually converged rendering. This can result in prohibitively long rendering times. Therefore, there is a need to reduce the number of samples needed for MC path tracing while still producing high-quality images.SUMMARY

Supervised machine learning using convolutional neural networks (CNNs) is applied to denoising images rendered by MC path tracing. The input image data may include pixel color and its variance, as well as a set of auxiliary buffers that encode scene information (e.g., surface normal, albedo, depth, and their corresponding variances). A repeated-block architecture and a residual-block architecture may be employed in the neural n

Claims

1. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that when executed control a computer system to perform a method of denoising images rendered by Monte Carlo (MC) path-tracing, the instructions comprising: receiving a plurality of input images, each input image having a first number of pixels and including input image data for each respective pixel obtained by MC path-tracing; receiving a plurality of reference images, each reference image corresponding to a respective input image and having a second number of pixels, each reference image including reference image data for each respective pixel; training a convolutional neural network (CNN) using the plurality of input images and the plurality of reference images, the CNN including: an input layer having a first number of input nodes for receiving input image data for each respective pixel of a respective input image; a plurality of hidden layers, each hidden layer having a respective number of nodes and having a respective receptive field, each respective hidden layer applying a convolution operation to a preceding hidden layer, with a first hidden layer of the plurality of hidden layers applying a convolution operation to the input layer, each node of a respective hidden layer processing data of a plurality of nodes of a preceding hidden layer within the respective receptive field using a plurality of parameters associated with the respective receptive field; an output layer having a second number of output nodes, the output layer applying a convolution operation to a last hidden layer of the plurality of hidden layers to obtain a plurality of output values associated with the second number of output nodes; and a reconstruction module coupled to the output layer for generating a respective output image corresponding to the respective input image using the plurality of output values, the respective output image having the second number of pixels and including output image data for each respective pixel; wherein training the CNN includes, for each respective input image, optimizing the plurality of parameters associated with the respective receptive field of each hidden layer by comparing the respective output image to a corresponding reference image to obtain a plurality of optimized parameters; receiving a new input image obtained by MC path-tracing; and generating a new output image corresponding to the new input image by passing the new input image through the CNN using the plurality of optimized parameters, the new output image being less noisy than the new input image. 12. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that when executed control a computer system to perform a method of denoising images rendered by Monte Carlo (MC) path-tracing, the instructions comprising: receiving a plurality of input images, each input image having a first number of pixels and including input image data for each respective pixel obtained by MC path-tracing, the input image data comprising color data for each respective pixel; receiving a plurality of reference images, each reference image corresponding to a respective input image and having a second number of pixels, each reference image including reference image data for each respective pixel; training a neural network using the plurality of input images and the plurality of reference images, the neural network including: an input layer having a first number of input nodes for receiving input image data for each respective pixel of a respective input image; a plurality of hidden layers, each hidden layer having a respective number of nodes, each node of a respective hidden layer processing data of a plurality of nodes of a preceding hidden layer using a plurality of parameters associated with the plurality of nodes, with each node of a first hidden layer of the plurality of hidden layers processing data of a plurality of nodes of the input layer; an output layer having a second number of output nodes associated with a neighborhood of pixels around each pixel of the input image, each node of the output layer processing data of a plurality of nodes of a last hidden layer of the plurality of hidden layers to obtain a respective output value; and a reconstruction module coupled to the output layer for generating a respective output image corresponding to the respective input image, the respective output image having the second number of pixels, each respective pixel having color data relating to a weighted combination of the color data for the neighborhood of pixels around a corresponding pixel of the input image using the output values associated with the second number of output nodes as weights; wherein training the neural network includes, for each respective input image, optimizing the plurality of parameters associated with the plurality of nodes of each hidden layer by comparing the respective output image to a corresponding reference image to obtain a plurality of optimized parameters; receiving a new input image obtained by MC path-tracing; and generating a new output image corresponding to the new input image by passing the new input image through the neural network using the plurality of optimized parameters, the new output image being less noisy than the new input image. 19. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that when executed control a computer system to perform a method of denoising images rendered by Monte Carlo (MC) path-tracing, the instructions comprising: receiving a plurality of input images, each input image having a first number of pixels and including a diffuse buffer and a specular buffer, the diffuse buffer including diffuse input image data for each respective pixel, the specular buffer including specular input image data for each respective pixel, the diffuse input image data comprising diffuse color data for each respective pixel, and the specular input image data comprising specular color data for each respective pixel; receiving a plurality of reference images, each reference image corresponding to a respective input image and having a second number of pixels, each reference image including a diffuse buffer and a specular buffer, the diffuse buffer including diffuse reference image data for each respective pixel, the specular buffer including specular reference image data for each respective pixel; training a first neural network using the diffuse buffers of the plurality of input images and the diffuse buffers of the plurality of reference images, the first neural network including: a diffuse input layer for receiving diffuse input image data for each respective pixel of a respective input image; a plurality of diffuse hidden layers, each diffuse hidden layer including a plurality of nodes, each node of a respective diffuse hidden layer processing data of a plurality of nodes of a preceding diffuse hidden layer using a plurality of first parameters associated with the plurality of nodes, with each node of a first diffuse hidden layer of the plurality of diffuse hidden layers processing data of a plurality of nodes of the diffuse input layer; a diffuse output layer having a first number of output nodes associated with a first neighborhood of pixels around each pixel of the input image, each node of the diffuse output layer processing data of a plurality of nodes of a last diffuse hidden layer of the plurality of hidden layers to obtain a respective diffuse output value; and a diffuse reconstruction module coupled to the diffuse output layer for generating a respective diffuse output image corresponding to the respective input image, the respective diffuse output image having the second number of pixels, each respective pixel having diffuse color data relating to a weighted combination of the diffuse color data for the first neighborhood of pixels around a corresponding pixel of the input image using the diffuse output values associated with the first number of output nodes as weights; wherein training the first neural network includes, for each respective input image, optimizing the plurality of first parameters associated with the plurality of nodes of each diffuse hidden layer by comparing the respective diffuse output image to the diffuse buffer of a corresponding reference image to obtain a plurality of optimized first parameters; training a second neural network using the specular buffers of the plurality of input images and the specular buffers of the plurality of reference images, the second neural network including: a specular input layer for receiving specular input image data for each respective pixel of a respective input image; a plurality of specular hidden layers, each specular hidden layer including a plurality of nodes, each node of a respective specular hidden layer processing data of a plurality of nodes of a preceding specular hidden layer using a plurality of second parameters associated with the plurality of nodes, with each node of a first specular hidden layer of the plurality of specular hidden layers processing data of a plurality of nodes of the specular input layer; a specular output layer having a second number of output nodes associated with a second neighborhood of pixels around each pixel of the input image, each node of the specular output layer processing data of a plurality of nodes of a last specular hidden layer of the plurality of specular hidden layers to obtain a respective specular output value; and a specular reconstruction module coupled to the specular output layer for generating a respective specular output image corresponding to the respective input image, the respective specular output image having the second number of pixels, each respective pixel having specular color data relating to a weighted combination of the specular color data for the second neighborhood of pixels around a corresponding pixel of the input image using the specular output values associated with the second number of output nodes as weights; wherein training the second neural network includes, for each respective input image, optimizing the plurality of second parameters associated with the plurality of nodes of each specular hidden layer by comparing the respective specular output image to the specular buffer of a corresponding reference image to obtain a plurality of optimized second parameters; receiving a new input image obtained by MC path-tracing, the new input image including a diffuse buffer and a specular buffer; generating a new diffuse output image corresponding to the new input image by passing the diffuse buffer of the new input image through the first neural network using the plurality of optimized first parameters; generating a new specular output image corresponding to the new input image by passing the specular buffer of the new input image through the second neural network using the plurality of optimized second parameters; and generating a new output image by combining the new diffuse output image and the new specular output image, the new output image being less noisy than the new input image.