DECOMPOSING AND RECOMPOSING SCENES USING NEURAL RADIANCE FIELDS

Number: 20250104343
Published: 2025-03-27
Filed: 2023-09-27
Assignee: DISNEY ENTERPRISES, INC.
Inventors: MEGARO; Martina et al.
CPC: G06T17/00; G06V10/25; G06T7/55; G06T7/90; G06T15/08; G06T15/20; G06T19/20
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

NeRF-based scene decomposition/recomposition rendering technique.

Abstract

The present invention sets forth a technique for performing scene decomposition. This technique includes determining, based on a plurality of two-dimensional (2D) representations of a three-dimensional (3D) scene, a different radiance field function for each of a plurality of neural radiance fields (NeRFs). The technique also includes generating a combined radiance field function based on the radiance field functions associated with the plurality of NeRFs. The technique further includes generating a color value for a given 3D location and viewing angle in the 3D scene. The technique further includes computing a decomposition loss based on the difference between the color value and a ground truth color value associated with the 3D location and viewing angle. The technique further includes modifying at least one of the plurality of NeRFs based on the decomposition loss such that each NeRF is associated with a different object in the 3D scene.

Background

BACKGROUND Field of the Various Embodiments

Embodiments of the present disclosure relate generally to machine learning and computer vision and, more specifically, to techniques for creating representations of one or more three-dimensional (3D) objects in a scene from one or more two-dimensional (2D) representations of the scene. Description of the Related Art

Generating a 3D representation of a scene including one or more 3D objects is a common task in the fields of computer vision and computer graphics. This representation of the scene may be generated from one or more 2D representations of the scene. One goal of generating representations of a scene is the ability to generate different viewpoints of the scene. For instance, a particular 2D representation of a scene may have been captured by placing a camera at a specific location and with a specific orientation relative to the scene. The captured 2D representation can then be used to generate additional 2D representations of the scene from different camera viewpoints. Further, generating different representations of a scene also allows creators to modify the scene. For example, objects (either real or computer-generated) may be added to the scene, objects may be removed from the scene, or the relative positions of objects in the scene may be altered.

Existing techniques for generating 3D representations of scenes may make use of photogrammetry. Photogrammetry uses a collection of 2D representations of a

Claims

1. A computer-implemented method for performing scene decomposition, the computer-implemented method comprising: determining, based on a plurality of two-dimensional (2D) representations of a three-dimensional (3D) scene, a different radiance field function associated with each of a plurality of neural radiance fields (NeRFs); generating a combined radiance field function based on the radiance field functions associated with the plurality of NeRFs; generating, based on the combined radiance field function, a color value for a given 3D location and a given viewing angle in the 3D scene; computing a decomposition loss based on a difference between the color value and a ground truth color value associated with the given 3D location and the given viewing angle; and modifying at least one of the plurality of NeRFs based on the decomposition loss such that each NeRF is associated with a different object in the 3D scene. || 10. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: determining, based on a plurality of two-dimensional representations of a three-dimensional (3D) scene, a different radiance field function associated with each of a plurality of neural radiance fields (NeRFs); generating a combined radiance field function based on the radiance field functions associated with the plurality of NeRFs; generating, based on the combined radiance field function, a color value for a given 3D location and a given viewing angle in the 3D scene; computing a decomposition loss based on the difference between the color value and a ground truth color value associated with the given 3D location and the given viewing angle; and modifying at least one of the plurality of NeRFs based on the decomposition loss such that each NeRF is associated with a different object in the 3D scene. || 19. A computer-implemented method for performing scene composition, the computer-implemented method comprising: modifying a single neural radiance field (NeRF) based on a given plurality of NeRFs, each of the plurality of NeRFs representing a discrete object in a 3D scene; and generating, based on the modified single NeRF, a composited 3D scene that includes each of the discrete objects represented by the plurality of NeRFs.