Temporal techniques of denoising Monte Carlo renderings using neural networks

Number: 11532073
Published: 2022-12-20
Filed: 2018-07-31
Assignee: Pixar
Inventors: Vogels; Thijs, Rousselle; Fabrice, Novak; Jan, McWilliams; Brian, Meyer; Mark, Harvill; Alex
CPC: G06F17/18; G06N20/00; G06N3/044; G06N3/045; G06N3/0455; G06N3/0464; G06N3/084; G06N3/09; G06N3/096; G06N5/046; G06N7/01; G06T15/06; G06T15/506; G06T5/50; G06T5/60; G06T5/70
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Temporal Monte Carlo denoising technique via neural networks (Pixar VFX rendering).

Abstract

A modular architecture is provided for denoising Monte Carlo renderings using neural networks. The temporal approach extracts and combines feature representations from neighboring frames rather than building a temporal context using recurrent connections. A multiscale architecture includes separate single-frame or temporal denoising modules for individual scales, and one or more scale compositor neural networks configured to adaptively blend individual scales. An error-predicting module is configured to produce adaptive sampling maps for a renderer to achieve more uniform residual noise distribution. An asymmetric loss function may be used for training the neural networks, which can provide control over the variance-bias trade-off during denoising.

Background

BACKGROUND (1) Monte Carlo (MC) path tracing is a technique for rendering images of three-dimensional scenes by tracing paths of light through pixels on an image plane. This technique is capable of producing high quality images that are nearly indistinguishable from photographs. In MC path tracing, the color of a pixel is computed by randomly sampling light paths that connect the camera to light sources through multiple interactions with the scene. The mean intensity of many such samples constitutes a noisy estimate of the total illumination of the pixel. Unfortunately, in realistic scenes with complex light transport, these samples might have large variance, and the variance of their mean only decreases linearly with respect to the number of samples per pixel. Typically, thousands of samples per pixel are required to achieve a visually converged rendering. This can result in prohibitively long rendering times. Therefore, there is a need to reduce the number of samples needed for MC path tracing while still producing high-quality images. SUMMARY (2) A modular architecture is provided for denoising Monte Carlo renderings using neural networks. A source-aware encoding module may be configured to extract low-level features and embed them into a feature space common between sources, which may allow for quickly adapting a trained network to novel data. A spatial module may be configured to extract abstract, high-level features for reconstruction. (3) According to some embodiments,

Claims

1. A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising: receiving a sequence of frames rendered by MC path tracing, the sequence of frames including a center frame and one or more temporal neighboring frames; receiving a reference image corresponding to the center frame; configuring a plurality of first neural networks, each respective first neural network configured to extract a respective set of first features from a respective frame of the sequence of frames; configuring a second neural network including a plurality of layers and a plurality of nodes, the second neural network configured to: extract a set of temporal features from the sets of first features; and output an output frame corresponding to the center frame; and training the second neural network to obtain a plurality of optimized parameters associated with the plurality of nodes of the second neural network using the sequence of frames and the reference image corresponding to the center frame. || 16. A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising: receiving a sequence of frames rendered by MC path tracing, the sequence of frames including a center frame and one or more temporal neighboring frames; receiving a reference image corresponding to the center frame; configuring a plurality of first neural networks, each respective first neural network comprising a respective first plurality of layers and a respective first set of nodes, each respective first neural network configured to extract a respective set of first features from a respective frame of the sequence of frames; configuring a second neural network including a second plurality of layers and a second set of nodes, the second neural network configured to: extract a set of temporal features from the sets of first features; and output an output frame corresponding to the center frame; and training the plurality of first neural networks and the second neural network to obtain a respective first set of optimized parameters associated with each respective first set of nodes of the respective first neural network, and a second set of optimized parameters associated with the second set of nodes of the second neural network, wherein the training uses the sequence of frames and the reference image corresponding to the center frame.