SYNTHESIZING SEQUENCES OF IMAGES FOR MOVEMENT-BASED PERFORMANCE

Number: 20230154090
Published: 2023-05-18
Filed: 2021-11-15
Assignee: DISNEY ENTERPRISES, INC.
Inventors: Bradley; Derek Edward et al.
CPC: G06N3/045; G06T13/40; G06T7/215; G06N3/088; G06T17/20; G06T19/20; G06N3/047; G06N3/084; G06T13/80; G06N3/08; G06T15/04
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Movement/performance-driven image-sequence synthesis technique.

Abstract

A technique for rendering an input geometry includes generating a first segmentation mask for a first input geometry and a first set of texture maps associated with one or more portions of the first input geometry. The technique also includes generating, via one or more neural networks, a first set of neural textures for the one or more portions of the first input geometry. The technique further includes rendering a first image corresponding to the first input geometry based on the first segmentation mask, the first set of texture maps, and the first set of neural textures.

Background

BACKGROUND Field of the Various Embodiments

Embodiments of the present disclosure relate generally to machine learning and animation and, more specifically, to synthesizing sequences of images for movement-based performance. Description of the Related Art

Realistic digital faces are required for various computer graphics and computer vision applications. For example, digital faces are oftentimes used in virtual scenes of film or television productions and in video games.

To capture photorealistic faces, a typical facial capture system employs a specialized light stage and hundreds of lights that are used to capture numerous images of an individual face under multiple illumination conditions. The facial capture system additionally employs multiple calibrated camera views, uniform or controlled patterned lighting, and a controlled setting in which the face can be guided into different expressions to capture images of individual faces. These images can then be used to determine three-dimensional (3D) geometry and appearance maps that are needed to synthesize digital versions of the faces.

Machine learning models have also been developed to synthesize digital faces. These machine learning models can include a large number of tunable parameters and thus require a large amount and variety of data to train. However, collecting training data for these machine learning models can be time- and resource-intensive. For example, a deep neural network could be t

Claims

1. A computer-implemented method for rendering an input geometry, the computer-implemented method comprising: generating a first segmentation mask for a first input geometry and a first set of texture maps associated with one or more portions of the first input geometry; generating, via one or more neural networks, a first set of neural textures for the one or more portions of the first input geometry; and rendering a first image corresponding to the first input geometry based on the first segmentation mask, the first set of texture maps, and the first set of neural textures. || 11. One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: generating one or more maps associated with one or more portions of a first input geometry; generating, via one or more neural networks, a first set of neural textures for the one or more portions of the first input geometry; and rendering a first image corresponding to the first input geometry based on the one or more maps and the first set of neural textures. || 20. A system, comprising: one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to: generate a first segmentation mask for a first input geometry and a first set of texture maps associated with one or more portions of the first input geometry; generate, via one or more neural networks, a first set of neural textures for the one or more portions of the first input geometry; and render a first image corresponding to the first input geometry based on the first segmentation mask, the first set of texture maps, and the first set of neural textures.