Techniques for multi-view neural object modeling

Number: 12236517
Published: 2025-02-25
Filed: 2022-11-08
Assignee: Disney Enterprises, INC.
Inventors: Bradley; Derek Edward et al.
CPC: G06T7/62; G06T15/08; G06T15/04; G06T15/06
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Multi-view neural object-modeling rendering technique.

Abstract

Techniques are disclosed for generating photorealistic images of objects, such as heads, from multiple viewpoints. In some embodiments, a morphable radiance field (MoRF) model that generates images of heads includes an identity model that maps an identifier (ID) code associated with a head into two codes: a deformation ID code encoding a geometric deformation from a canonical head geometry, and a canonical ID code encoding a canonical appearance within a shape-normalized space. The MoRF model also includes a deformation field model that maps a world space position to a shape-normalized space position based on the deformation ID code. Further, the MoRF model includes a canonical neural radiance field (NeRF) model that includes a density multi-layer perceptron (MLP) branch, a diffuse MLP branch, and a specular MLP branch that output densities, diffuse colors, and specular colors, respectively. The MoRF model can be used to render images of heads from various viewpoints.

Background

BACKGROUND Technical Field (1) Embodiments of the present disclosure relate generally to computer science and image generation and, more specifically, to techniques for multi-view neural object modeling. Description of the Related Art (2) Realistic digital images of heads (e.g., human heads) are required for various computer graphics and computer vision applications. As used herein, an image of a head can include regions corresponding to the skin of a face as well as non-skin regions that can correspond to eyes, ears, scalp hair, facial hair, inside of the mouth, parts of the neck and shoulder, etc. For example, digital images of heads are oftentimes used in the virtual scenes of film productions and in video games, among other things. (3) One approach for generating digital images of heads involves capturing faces and rendering the captured faces in images. However, conventional facial capture techniques are limited to capturing the skin regions of faces, while non-skin regions are typically not captured. To generate an image of a head that includes both skin and non-skin regions, the non-skin regions that are not captured need to be filled in, or “inpainted,” after the captured skin regions are rendered. Conventional techniques for inpainting the non-skin regions oftentimes cannot generate photorealistic heads. In addition, to generate images of heads having desired inpainting details (e.g., eyes and hair that look realistic), manual effort is typically required to tune the

Claims

1. A computer-implemented method for rendering an image of an object, the method comprising: tracing a ray through a pixel into a virtual scene; sampling one or more positions along the ray; applying a machine learning model to the one or more positions and an identifier (ID) code associated with an object to determine, for each position included in the one or more positions, a density, a diffuse color, and a specular color, wherein the ID code is used to determine a geometric deformation from a canonical object geometry, wherein the geometric deformation is associated with the object; and computing a color of a pixel based on the density, the diffuse color, and the specular color corresponding to each position included in the one or more positions. || 11. One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processing units, cause the one or more processing units to perform steps for rendering an image of an object, the steps comprising: tracing a ray through a pixel into a virtual scene; sampling one or more positions along the ray; applying a machine learning model to the one or more positions and an identifier (ID) code associated with an object to determine, for each position included in the one or more positions, a density, a diffuse color, and a specular color, wherein the ID code is used to determine a geometric deformation from a canonical object geometry, wherein the geometric deformation is associated with the object; and computing a color of a pixel based on the density, the diffuse color, and the specular color corresponding to each position included in the one or more positions. || 19. A computer-implemented method for training a machine learning model, the method comprising: receiving a first set of images of one or more object that are captured from a plurality of viewpoints; generating a second set of images of the one or more object from another plurality of viewpoints; and training, based on the first set of images and the second set of images, a machine learning model, wherein the machine learning model comprises a neural radiance field model and an identity model, and wherein the identity model maps an identifier (ID) code to (i) a deformation ID code that encodes a geometric deformation from a canonical object geometry, and (ii) a canonical ID code that encodes an appearance within a space associated with the canonical object geometry.