Adaptive sampling in Monte Carlo renderings using error-predicting neural networks

Number: 10706508
Published: 2020-07-07
Filed: 2018-07-31
Assignee: Disney Enterprises, Inc.
Inventors: Vogels; Thijs, Rousselle; Fabrice, Novak; Jan, McWilliams; Brian, Meyer; Mark, Harvill; Alex
CPC: G06F17/18; G06N20/00; G06N3/044; G06N3/045; G06N3/0455; G06N3/0464; G06N3/084; G06N3/09; G06N3/096; G06N5/046; G06N7/01; G06T15/06; G06T15/506; G06T5/50; G06T5/60; G06T5/70
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Adaptive sampling Monte Carlo render denoising, VFX rendering research.

Abstract

A modular architecture is provided for denoising Monte Carlo renderings using neural networks. The temporal approach extracts and combines feature representations from neighboring frames rather than building a temporal context using recurrent connections. A multiscale architecture includes separate single-frame or temporal denoising modules for individual scales, and one or more scale compositor neural networks configured to adaptively blend individual scales. An error-predicting module is configured to produce adaptive sampling maps for a renderer to achieve more uniform residual noise distribution. An asymmetric loss function may be used for training the neural networks, which can provide control over the variance-bias trade-off during denoising.

Background

BACKGROUND(1) Monte Carlo (MC) path tracing is a technique for rendering images of three-dimensional scenes by tracing paths of light through pixels on an image plane. This technique is capable of producing high quality images that are nearly indistinguishable from photographs. In MC path tracing, the color of a pixel is computed by randomly sampling light paths that connect the camera to light sources through multiple interactions with the scene. The mean intensity of many such samples constitutes a noisy estimate of the total illumination of the pixel. Unfortunately, in realistic scenes with complex light transport, these samples might have large variance, and the variance of their mean only decreases linearly with respect to the number of samples per pixel. Typically, thousands of samples per pixel are required to achieve a visually converged rendering. This can result in prohibitively long rendering times. Therefore, there is a need to reduce the number of samples needed for MC path tracing while still producing high-quality images.SUMMARY(2) A modular architecture is provided for denoising Monte Carlo renderings using neural networks. A source-aware encoding module may be configured to extract low-level features and embed them into a feature space common between sources, which may allow for quickly adapting a trained network to novel data. A spatial module may be configured to extract abstract, high-level features for reconstruction.(3) According to some embodiments, a t

Claims

1. A method of rendering images by Monte Carlo (MC) path tracing, the method comprising: rendering, using a renderer, a first input image by MC path tracing, the first input image including a plurality of pixels and input color data for each pixel, the first input image rendered with a first number of samples across the plurality of pixels; generating a first denoised image corresponding to the first input image, the first denoised image including the plurality of pixels and denoised color data for each pixel; predicting a first sampling map using a first neural network, the first sampling map including the plurality of pixels, wherein the first neural network is configured to: receive the first input image and the first denoised image; and generate a respective error value for each respective pixel of the first sampling map, the respective error value relating to a difference between denoised color data and input color data for the respective pixel; and rendering, using the renderer, a second input image corresponding to the first input image by MC path tracing, the second input image including the plurality of pixels, the second input image rendered with a second number of samples greater than the first number, wherein the second number of samples is distributed across the plurality of pixels based on the first sampling map. 15. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that when executed control a computer system to render images by Monte Carlo (MC) path tracing, the instructions comprising: rendering, using a renderer, a first input image by MC path tracing, the first input image including a plurality of pixels and input color data for each pixel, the first input image rendered with a first number of samples across the plurality of pixels; generating a first denoised image corresponding to the first input image, the first denoised image including the plurality of pixels and denoised color data for each pixel; predicting a first sampling map using a first neural network, the first sampling map including the plurality of pixels, wherein the first neural network is configured to: receive the first input image and the first denoised image; and generate a respective error value for each respective pixel of the first sampling map, the respective error value relating to a difference between denoised color data and input color data for the respective pixel; and rendering, using the renderer, a second input image corresponding to the first input image by MC path tracing, the second input image including the plurality of pixels, the second input image rendered with a second number of samples greater than the first number, wherein the second number of samples is distributed across the plurality of pixels based on the first sampling map.