- Number
- 20240161391
- Published
- 2024-05-16
- Filed
- 2023-11-08
- Assignee
- DISNEY ENTERPRISES, INC.
- Inventors
- Bradley; Derek Edward et al.
- CPC
- G06T7/90; G06T15/506; G06T15/50; G06T19/20
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Relightable neural radiance field (NeRF) rendering technique.
Abstract
The present invention sets forth a technique for generating two-dimensional (2D) renderings of a three-dimensional (3D) scene from an arbitrary camera position under arbitrary lighting conditions. This technique includes determining, based on a plurality of 2D representations of a 3D scene, a radiance field function for a neural radiance field (NeRF) model. This technique further includes determining, based on a plurality of 2D representations of a 3D scene, a radiance field function for a “one light at a time” (OLAT) model. The technique further includes rendering a 2D representation of the scene based on a given camera position and illumination data. The technique further includes computing a rendering loss based on the difference between the rendered 2D representation and an associated one of the plurality of 2D representations of the scene. The technique further includes modifying at least one of the NeRF and OLAT models based on the rendering loss.
Background
BACKGROUND Field of the Various Embodiments
Embodiments of the present disclosure relate generally to machine learning and computer vision and, more specifically, to techniques for creating representations of one or more three-dimensional (3D) objects in a scene from one or more two-dimensional (2D) representations of the scene. Description of the Related Art
Generating a 3D representation of a scene including one or more 3D objects is a common task in the fields of computer vision and computer graphics. This representation of the scene may be generated from one or more 2D representations of the scene. In some applications, different representations of a scene are generated from different viewpoints, where a viewpoint is a combination of a specific camera location and a specific orientation of the camera relative to the scene. For instance, multiple 2D representations of a scene may have been captured by placing one or more cameras at specific locations and with specific orientations relative to the scene. The captured 2D representations can then be used to generate additional 2D representations of the scene from different camera viewpoints. Further, generating different 2D representations of a scene also allows creators to modify the scene. For example, objects (either real or computer-generated) may be added to the scene, objects may be removed from the scene, or the relative positions of objects in the scene may be altered.
Existing techniques for gene
Claims
1. A computer-implemented method for performing scene rendering, the computer-implemented method comprising: determining, for each of a plurality of three-dimensional (3D) locations in a 3D scene, a density value associated with the 3D location based on an output of a first trained machine learning model; determining, for each of the plurality of 3D locations, a diffuse color value and a specular color value associated with the 3D location based on a given camera location, a given lighting map, and an output of a second trained machine learning model; determining a pixel color value for each of a plurality of pixels in a two-dimensional (2D) representation of the scene based on the density values, the diffuse color values, and the specular color values associated with the plurality of 3D locations; and generating, based on the pixel color values associated with the plurality of pixels, a 2D rendering of the scene. ||
8. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: determining, for each of a plurality of three-dimensional (3D) locations in a 3D scene, a density value associated with the 3D location based on an output of a first trained machine learning model; determining, for each of the plurality of 3D locations, a diffuse color value and a specular color value associated with the 3D location based on a given camera location, a given lighting map, and an output of a second trained machine learning model; determining a pixel color value for each of a plurality of pixels in a two-dimensional (2D) representation of the scene based on the density values, diffuse color values, and specular color values associated with the plurality of 3D locations; and generating, based on the pixel color values associated with the plurality of pixels, a 2D rendering of the scene. ||
15. A computer-implemented method for performing scene rendering, the computer-implemented method comprising: determining, based on a plurality of two-dimensional (2D) representations of a three-dimensional (3D) scene, a first radiance field function associated with a first machine learning model; determining, based on the plurality of 2D representations of the 3D scene, a second radiance field function associated with a second machine learning model; generating a combined radiance field function based on the radiance field functions associated with the first machine learning model and the second machine learning model; generating, based on the combined radiance field function, a color value for a pixel in a 2D rendering of the scene; computing a rendering loss based on a difference between the color value for the pixel and a ground truth color value associated with a corresponding pixel in a corresponding one of the plurality of 2D representations; and modifying at least one of the first machine learning model or the second machine learning model based on the rendering loss.