TECHNIQUES FOR FEATURE-BASED NEURAL RENDERING

Number: 20210019928
Published: 2021-01-21
Filed: 2019-07-15
Assignee: DISNEY ENTERPRISES, INC.
Inventors: BORER; Dominik Tobias, GUAY; Martin, BUHMANN; Jakob Joachim, SUMNER; Robert Walker
CPC: G06F17/15; G06N20/00; G06N3/0455; G06N3/0464; G06N3/09; G06N3/094; G06T13/40; G06T19/006
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Feature-based neural rendering technique.

Abstract

Techniques are disclosed for learning a machine learning model that maps control data, such as renderings of skeletons, and associated three-dimensional (3D) information to two-dimensional (2D) renderings of a character. The machine learning model may be an adaptation of the U-Net architecture that accounts for 3D information and is trained using a perceptual loss between images generated by the machine learning model and ground truth images. Once trained, the machine learning model may be used to animate a character, such as in the context of previsualization or a video game, based on control of associated control points.

Background

BACKGROUND Technical Field

Embodiments of the present disclosure relate generally to image rendering and, more specifically, to techniques for feature-based neural rendering. Description of the Related Art

Feature animation films generally include high-definition, high-fidelity characters. Typically, such characters are animated using high-resolution models and textures as well as complex proprietary rigs and deformation algorithms.

The characters from feature animation films are not easily adaptable to real-time applications, such as computer-based games and previsualation. Previsualization, also referred to herein as 'previs,' is the visualization of scenes prior to final animation or filming. For example, motion capture with a single camera may be employed to visualize a character's movement in the early stages of story authoring and storyboarding. Rendering engines used in real-time applications typically support only linear blend skinning and blend shapes, not the proprietary rigs and deformation algorithms used to render feature animation films. Further, real-time rendering engines may require lower-resolution models and textures.

Currently, no effective techniques exist for decimating or otherwise exporting the high-definition, high-fidelity assets used in feature animation films to low-resolution assets compatible with real-time rendering engines. As a result, low-resolution assets, such as characters re-modeled with fewer polygons, re-text

Claims

1. A computer-implemented method for rendering an image that includes at least one character, the method comprising: determining pose information for a first character based on a control signal; and processing the pose information using a trained machine learning model to generate a rendered image of the first character based on a 3D model of the first character. || 11. A computer-implemented method for training a machine learning model, the method comprising: receiving training data that includes a plurality of rendered images and an associated set of control points for each rendered image; and training the machine learning model based on a perceptual loss between one or more images generated by the machine learning model and one or more associated rendered images included in the training data. || 19. A non-transitory computer-readable storage medium including instructions that, when executed by a processing unit, cause the processing unit to train a machine learning model by performing steps comprising: receiving training data that includes a plurality of rendered images and an associated set of control points for each rendered image; and training the machine learning model based on a perceptual loss between one or more images generated by the machine learning model and one or more associated rendered images included in the training data.