SHAPE AND APPEARANCE RECONSTRUCTION WITH DEEP GEOMETRIC REFINEMENT

Number: 20230252714
Published: 2023-08-10
Filed: 2022-02-10
Assignee: DISNEY ENTERPRISES, INC.
Inventors: Bradley; Derek Edward et al.
CPC: G06T15/205; G06T17/00; G06V40/169; G06T15/04; G06T7/97
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Deep geometric-refinement shape/appearance reconstruction technique (VFX).

Abstract

One embodiment of the present invention sets forth a technique for performing shape and appearance reconstruction. The technique includes generating a first set of renderings associated with an object based on a set of parameters that represent a reconstruction of the object in a first target image. The technique also includes producing, via a neural network, a first set of corrections associated with at least a portion of the set of parameters based on the first target image and the first set of renderings. The technique further includes generating an updated reconstruction of the object based on the first set of corrections.

Background

BACKGROUND Field of the Various Embodiments

Embodiments of the present disclosure relate generally to machine learning and computer vision, more specifically, to shape and appearance reconstruction with deep geometric refinement. Description of the Related Art

Realistic digital faces are required for various computer graphics and computer vision applications. For example, digital faces are oftentimes used in virtual scenes of film or television productions and in video games.

To capture photorealistic faces, a typical facial capture system employs a specialized light stage and hundreds of lights that are used to capture numerous images of an individual face under multiple illumination conditions. The facial capture system additionally employs multiple calibrated camera views, uniform or controlled patterned lighting, and a controlled setting. Further, a given face is typically scanned during a scheduled block of time, in which the corresponding individual can be guided into different expressions to capture images of individual faces. The resulting images can then be used to determine three-dimensional (3D) geometry and appearance maps that are needed to synthesize digital versions of the face.

Because existing facial capture systems require controlled settings and the physical presence of the corresponding individuals, these facial capture systems cannot be used to perform facial reconstruction under various uncontrolled “in the wild” conditions th

Claims

1. A computer-implemented method for performing shape and appearance reconstruction, the computer-implemented method comprising: generating a first set of renderings associated with an object based on a set of parameters that represent a reconstruction of the object in a first target image; producing, via a neural network, a first set of corrections associated with at least a portion of the set of parameters based on the first target image and the first set of renderings; and generating an updated reconstruction of the object based on the first set of corrections. || 11. One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: generating a first set of renderings associated with an object based on a set of parameters that represent a reconstruction of the object in a first target image; producing, via a neural network, a first set of corrections associated with at least a portion of the set of parameters based on the first target image and the first set of renderings; and generating an updated reconstruction of the object based on the first set of corrections. || 20. A system, comprising: one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to: generate a first set of renderings associated with an object based on a set of parameters that represent a reconstruction of the object in a first target image; produce, via a neural network, a first set of corrections associated with at least a portion of the set of parameters based on the first target image and the first set of renderings; and generate an updated reconstruction of the object based on the first set of corrections.