Techniques for feature-based neural rendering

Number: 11704853
Published: 2023-07-18
Filed: 2019-07-15
Assignee: DISNEY ENTERPRISES, INC.
Inventors: Borer; Dominik Tobias et al.
CPC: G06F17/15; G06N20/00; G06N3/0455; G06N3/0464; G06N3/09; G06N3/094; G06T13/40; G06T19/006
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Feature-based neural rendering technique (granted).

Abstract

Techniques are disclosed for learning a machine learning model that maps control data, such as renderings of skeletons, and associated three-dimensional (3D) information to two-dimensional (2D) renderings of a character. The machine learning model may be an adaptation of the U-Net architecture that accounts for 3D information and is trained using a perceptual loss between images generated by the machine learning model and ground truth images. Once trained, the machine learning model may be used to animate a character, such as in the context of previsualization or a video game, based on control of associated control points.

Background

BACKGROUND Technical Field (1) Embodiments of the present disclosure relate generally to image rendering and, more specifically, to techniques for feature-based neural rendering. Description of the Related Art (2) Feature animation films generally include high-definition, high-fidelity characters. Typically, such characters are animated using high-resolution models and textures as well as complex proprietary rigs and deformation algorithms. (3) The characters from feature animation films are not easily adaptable to real-time applications, such as computer-based games and previsualation. Previsualization, also referred to herein as “previs,” is the visualization of scenes prior to final animation or filming. For example, motion capture with a single camera may be employed to visualize a character's movement in the early stages of story authoring and storyboarding. Rendering engines used in real-time applications typically support only linear blend skinning and blend shapes, not the proprietary rigs and deformation algorithms used to render feature animation films. Further, real-time rendering engines may require lower-resolution models and textures. (4) Currently, no effective techniques exist for decimating or otherwise exporting the high-definition, high-fidelity assets used in feature animation films to low-resolution assets compatible with real-time rendering engines. As a result, low-resolution assets, such as characters re-modeled with fewer polygons, re-textured with lo

Claims

1. A computer-implemented method for rendering an image that includes at least one character, the method comprising: determining, by one or more computing devices, pose information for a first character based on a control signal; and inputting the pose information and three-dimensional (3D) information associated with a 3D model of the first character into a trained machine learning model, wherein the trained machine learning model generates a rendered image of the first character based on the pose information and the 3D information, and wherein the trained machine learning model is trained based on a perceptual loss between images generated by the trained machine learning model and associated rendered images included in a set of training data. || 8. A computer-implemented method for rendering an image that includes at least one character, the method comprising: determining, by one or more computing devices, pose information for a first character based on a control signal; and inputting the pose information and three-dimensional (3D) information associated with a 3D model of the first character into a trained machine learning model, wherein the trained machine learning model generates a rendered image of the first character based on the pose information and the 3D information, and wherein the trained machine learning model is trained based on a perceptual loss between images generated by the trained machine learning model and associated rendered images included in a set of training data; wherein the trained machine learning model includes a two-dimensional (2D) U-Net architecture configured to incorporate the 3D information; the 3D information is processed via layers distinct from the 2D U-Net architecture; and the processed 3D information is used to augment the 2D U-Net architecture via skip connections. || 10. A computer-implemented method for training a machine learning model, the method comprising: receiving, by one or more computing devices, training data that includes a plurality of rendered images and an associated set of control points for each rendered image; and training, by the one or more computing devices, the machine learning model based on a perceptual loss between one or more images generated by the machine learning model and one or more associated rendered images included in the training data. || 18. A non-transitory computer-readable storage medium including instructions that, when executed by one or more processors, cause the one or more processors to train a machine learning model by performing steps comprising: receiving training data that includes a plurality of rendered images and an associated set of control points for each rendered image; and training the machine learning model based on a perceptual loss between one or more images generated by the machine learning model and one or more associated rendered images included in the training data.