TECHNIQUES FOR PERFORMING POINT-BASED INVERSE RENDERING

Number: 20210065434
Published: 2021-03-04
Filed: 2019-09-27
Assignee: DISNEY ENTERPRISES, INC.
Inventors: ÖZTIRELI; Ahmet Cengiz, SORKINE-HORNUNG; Olga, WU; Shihao, WANG; Yifan
CPC: G06T15/205; G06T7/90; G06T5/73; G06T11/40
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Point-based inverse rendering technique.

Abstract

In various embodiments, a differentiable rendering application enables an inverse rendering application to infer attributes associated with a 3D scene. In operation, the differentiable rendering application renders an image based on a first set of points associated with the 3D scene. The differentiable rendering application then generates an artificial gradient that approximates a change in a value of a first pixel included in the image with respect to a change in an attribute of a first point included in the first set of points. Subsequently, the inverse rendering application performs optimization operation(s) on the first point based on the artificial gradient to generate a second set of points. Notably, an error associated with the second set of points is less than an error associated with the first set of points.

Background

BACKGROUND Field of the Various Embodiments

Embodiments relate generally to computer science and computer vision and, more specifically, to techniques for performing point-based inverse rendering. Description of the Related Art

Neural networks trained to recognize patterns in input data based on example patterns included in training data can be used to solve a wide variety of “computer vision” problems. In computer vision, computers are programmed to infer, analyze, and/or process attributes of three-dimensional (“3D scenes”) from digital two-dimensional (“2D”) images. Some examples of these types of attributes include, without limitation, coordinates, normals, colors of points, camera position and orientation, and lighting.

For instance, in an iterative training process that includes a forward rendering pass and a backward inverse rendering pass, a neural network can be trained to infer attributes associated with a 3D scene based on a 2D reference image. In the forward rendering pass, a rendering function can be used to generate a 2D image based on a set of inferred attributes. In the backward inverse rendering pass, an error can be computed based on the difference between the generated image and the reference image and optimization operations can be performed on the set of inferred attributes to reduce that error.

With respect to the backward inverse rendering pass, optimizing the inferred attributes may require the rendering function to be diffe

Claims

1. A computer-implemented method for inferring attributes associated with a three-dimensional (3D) scene, the method comprising: rendering a first image based on a first plurality of points associated with the 3D scene; generating a first artificial gradient that approximates a change in a first value of a first pixel included in the first image with respect to a change in a first attribute of a first point included in the first plurality of points; and performing one or more optimization operations on the first point based on the first artificial gradient to generate a second plurality of points, wherein an error associated with the second plurality of points is less than an error associated with the first plurality of points. || 11. One or more non-transitory computer readable media including instructions that, when executed by one or more processors, cause the one or more processors to infer attributes associated with a three-dimensional (3D) scene by performing the steps of: rendering a first image based on a first plurality of points associated with the 3D scene; and generating a first artificial gradient that approximates a change in a first value of a first pixel included in the first image with respect to a change in a first attribute of a first point included in the first plurality of points; wherein one or more optimization operations are performed on the first point based on the first artificial gradient to generate a second plurality of points, wherein an error associated with the second plurality of points is less than an error associated with the first plurality of points. || 20. A system, comprising: one or more memories storing instructions; and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to: perform one or more rendering operations on a first point cloud associated with a 3D scene to generate a first image; compute an artificial gradient that approximates a change in a first value of a first pixel included in the first image with respect to a change in a first attribute of a first point included in the first point cloud; and perform one or more optimization operations on the first point based on the artificial gradient to generate a second point cloud, wherein an error associated with the second point cloud is less than an error associated with the first point cloud.