Temporal techniques of denoising Monte Carlo renderings using neural networks

Number: 12169914
Published: 2024-12-17
Filed: 2022-11-09
Assignee: PIXAR
Inventors: Vogels; Thijs et al.
CPC: G06F17/18; G06N20/00; G06N3/044; G06N3/045; G06N3/0455; G06N3/0464; G06N3/084; G06N3/09; G06N3/096; G06N5/046; G06N7/01; G06T15/06; G06T15/506; G06T5/50; G06T5/60; G06T5/70
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Temporal Monte Carlo denoising technique via neural networks (Pixar VFX rendering, granted).

Abstract

A modular architecture is provided for denoising Monte Carlo renderings using neural networks. The temporal approach extracts and combines feature representations from neighboring frames rather than building a temporal context using recurrent connections. A multiscale architecture includes separate single-frame or temporal denoising modules for individual scales, and one or more scale compositor neural networks configured to adaptively blend individual scales. An error-predicting module is configured to produce adaptive sampling maps for a renderer to achieve more uniform residual noise distribution. An asymmetric loss function may be used for training the neural networks, which can provide control over the variance-bias trade-off during denoising.

Background

BACKGROUND (1) Monte Carlo (MC) path tracing is a technique for rendering images of three-dimensional scenes by tracing paths of light through pixels on an image plane. This technique is capable of producing high quality images that are nearly indistinguishable from photographs. In MC path tracing, the color of a pixel is computed by randomly sampling light paths that connect the camera to light sources through multiple interactions with the scene. The mean intensity of many such samples constitutes a noisy estimate of the total illumination of the pixel. Unfortunately, in realistic scenes with complex light transport, these samples might have large variance, and the variance of their mean only decreases linearly with respect to the number of samples per pixel. Typically, thousands of samples per pixel are required to achieve a visually converged rendering. This can result in prohibitively long rendering times. Therefore, there is a need to reduce the number of samples needed for MC path tracing while still producing high-quality images. SUMMARY (2) A modular architecture is provided for denoising Monte Carlo renderings using neural networks. A source-aware encoding module may be configured to extract low-level features and embed them into a feature space common between sources, which may allow for quickly adapting a trained network to novel data. A spatial module may be configured to extract abstract, high-level features for reconstruction. (3) According to some embodiments,

Claims

1. A method of denoising images rendered by path tracing, the method comprising: receiving a sequence of frames rendered by path tracing, the sequence of frames including a first frame and one or more temporal neighboring frames; receiving a reference image corresponding to the first frame; extracting, using one or more first neural networks, sets of first features from the sequence of frames; storing a second neural network including a plurality of layers and a plurality of nodes, the second neural network configured to: extract a set of temporal features from the sets of first features; and training the second neural network to obtain a plurality of optimized parameters associated with the plurality of nodes of the second neural network using the sequence of frames and the reference image corresponding to the first frame. || 16. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that, when executed, cause a computer system to perform a method of denoising images rendered by path tracing, the method comprising: receiving a sequence of frames rendered by path tracing, the sequence of frames including a first frame and one or more temporal neighboring frames; receiving a reference image corresponding to the first frame; extracting, using one or more first neural networks, sets of first features from the sequence of frames; storing a second neural network including a plurality of layers and a plurality of nodes, the second neural network configured to: extract a set of temporal features from the sets of first features; and training the second neural network to obtain a plurality of optimized parameters associated with the plurality of nodes of the second neural network using the sequence of frames and the reference image corresponding to the first frame.