RELIGHTABLE AND REANIMATABLE NEURAL HEADS

Number: 20250166273
Published: 2025-05-22
Filed: 2024-11-18
Assignee: DISNEY ENTERPRISES, INC.
Inventors: BRADLEY; Derek Edward et al.
CPC: G06T15/506; G06T13/40; G06T17/20; G06T15/20
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Relightable/reanimatable neural character-head rendering technique.

Abstract

The present invention sets forth techniques for generating an animation sequence. The techniques include receiving one or more three-dimensional (3D) input meshes, wherein each input mesh includes a representation of an object included in a 3D scene. The techniques also include receiving, for each of the 3D input meshes, a virtual camera position associated with the 3D input mesh and one or more virtual lighting positions associated with the 3D input mesh. The techniques further include generating, for each of the 3D input meshes and via a trained machine learning model, one or more rendered frames associated with the 3D input mesh, wherein each rendered frame includes a two-dimensional (2D) representation of the object as viewed from the virtual camera position and illuminated by one or more virtual lights located at the one or more virtual lighting positions, and generating an output animation sequence based on the rendered frames.

Background

BACKGROUND Field of the Various Embodiments

Embodiments of the present disclosure relate generally to machine learning and computer vision and, more specifically, to techniques for creating animatable and relightable representations of one or more three-dimensional (3D) objects included in a scene from one or more two-dimensional (2D) representations of the scene. Description of the Related Art

Generating a 3D representation of a scene including one or more 3D objects is a common task in the fields of computer vision and computer graphics. This representation of the scene may be generated from one or more 2D representations of the scene. In some applications, different representations of a scene are generated from different viewpoints, where a viewpoint is a combination of a specific camera location and a specific orientation of the camera relative to the scene. For instance, multiple 2D representations of a scene may be captured by placing one or more cameras at specific locations and with specific orientations relative to the scene. The captured 2D representations may then be used to generate additional 2D representations of the scene from different camera viewpoints. Further, generating different 2D representations of a scene also allows creators to modify the scene. For example, objects (either real or computer-generated) may be added to the scene, objects may be removed from the scene, or the relative positions of objects in the scene may be altered.

Claims

1. A computer-implemented method for generating an animation sequence, the method comprising: receiving one or more three-dimensional (3D) input meshes, wherein each input mesh includes a representation of an object included in a 3D scene; receiving, for each of the one or more 3D input meshes, a virtual camera position associated with the 3D input mesh and one or more virtual lighting positions associated with the 3D input mesh; generating, for each of the one or more 3D input meshes and via a trained machine learning model, one or more rendered frames associated with the 3D input mesh, wherein each of the one or more rendered frames includes a two-dimensional (2D) representation of the object as viewed from the virtual camera position and illuminated by one or more virtual lights located at the one or more virtual lighting positions; and generating an output animation sequence based on the one or more rendered frames. || 11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: receiving one or more three-dimensional (3D) input meshes, wherein each input mesh includes a representation of an object included in a 3D scene; receiving, for each of the one or more 3D input meshes, a virtual camera position associated with the 3D input mesh and one or more virtual lighting positions associated with the 3D input mesh; generating, for each of the multiple 3D input meshes and via a trained machine learning model, one or more rendered frames associated with the 3D input mesh, wherein each of the one or more rendered frames includes a two-dimensional (2D) representation of the object as viewed from the virtual camera position and illuminated by one or more virtual lights located at the one or more virtual lighting positions; and generating an output animation sequence based on the one or more rendered frames. || 19. A system comprising: one or more memories storing instructions; and one or more processors for executing the instructions to: receive one or more three-dimensional (3D) input meshes, wherein each input mesh includes a representation of an object included in a 3D scene; receive, for each of the one or more 3D input meshes, a virtual camera position associated with the 3D input mesh and one or more virtual lighting positions associated with the 3D input mesh; generate, for each of the one or more 3D input meshes and via a trained machine learning model, one or more rendered frames associated with the 3D input mesh, wherein each of the one or more rendered frames includes a two-dimensional (2D) representation of the object as viewed from the virtual camera position and illuminated by one or more virtual lights located at the one or more virtual lighting positions; and generate an output animation sequence based on the multiple rendered frames.