FRAME INTERPOLATION FOR RENDERED CONTENT

Number: 20220383573
Published: 2022-12-01
Filed: 2021-05-19
Assignee: DISNEY ENTERPRISES, INC.
Inventors: SCHROERS; Christopher Richard, BRIEDIS; Karlis Martins, DJELOUAH; Abdelaziz, MCGONIGAL; Ian, MEYER; Mark, PAPAS; Marios, PLACK; Markus
CPC: G06T7/20; G06T13/00; G06T3/18; G06T13/80; G06T7/248
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Frame-interpolation rendering technique.

Abstract

One embodiment of the present invention sets forth a technique for performing frame interpolation. The technique includes generating (i) a first set of feature maps based on a first set of rendering features associated with a first key frame, (ii) a second set of feature maps based on a second set of rendering features associated with a second key frame, and (iii) a third set of feature maps based on a third set of rendering features associated with a target frame. The technique also includes applying one or more neural networks to the first, second, and third set of feature maps to generate a set of mappings from a first set of pixels in the first key frame to a second set of pixels in the target frame. The technique further includes generating the target frame based on the set of mappings.

Background

BACKGROUND Field of the Various Embodiments

Embodiments of the present disclosure relate generally to frame interpolation and, more specifically, to frame interpolation for rendered content. Description of the Related Art

Computer-animated video content is commonly created via a multi-stage process that involves generating models of characters, objects, and settings; applying textures, lighting, shading, and other effects to the models; animating the models; and rendering individual frames based on the animated models. During this process, the rendering stage typically incurs significantly more computational overhead than earlier stages. This computational overhead has also increased over time, as more complex and sophisticated rendering techniques are developed to improve the realism and detail in the rendered frames. For example, a single frame of computer-animated video content could require multiple hours to multiple days to render on a single processor. A feature-length film with over 100,000 frames would thus consume hundreds of years of processor hours.

To reduce rendering overhead associated with computer-animated video content, a subset of “key frames” in the computer-animated video content can be rendered, and remaining frames between pairs of consecutive key frames can be generated via less computationally expensive frame interpolation techniques. However, an interpolated frame is not created using the same amount of information (e.g., models,

Claims

1. A computer-implemented method for performing frame interpolation, the method comprising: generating (i) a first set of feature maps based on a first set of rendering features associated with a first key frame, (ii) a second set of feature maps based on a second set of rendering features associated with a second key frame, and (iii) a third set of feature maps based on a third set of rendering features associated with a target frame to be interpolated based on the first key frame and the second key frame; applying one or more neural networks to the first set of feature maps, the second set of feature maps, and the third set of feature maps to generate a set of mappings from a first set of pixels in the first key frame to a second set of pixels in the target frame; and generating the target frame based on the set of mappings. || 13. One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: generating (i) a first set of feature maps based on a first set of rendering features associated with a first key frame, (ii) a second set of feature maps based on a second set of rendering features associated with a second key frame, and (iii) a third set of feature maps based on a third set of rendering features associated with a target frame to be interpolated based on the first key frame and the second key frame; applying one or more neural networks to the first set of feature maps, the second set of feature maps, and the third set of feature maps to generate a set of mappings from a first set of pixels in the first key frame to a second set of pixels in the target frame; and generating the target frame based on the set of mappings. || 20. A system, comprising: a memory that stores instructions, and a processor that is coupled to the memory and, when executing the instructions, is configured to: generating (i) a first set of feature maps based on a first set of rendering features associated with a first key frame, (ii) a second set of feature maps based on a second set of rendering features associated with a second key frame, and (iii) a third set of feature maps based on a third set of rendering features associated with a target frame to be interpolated based on the first key frame and the second key frame; applying one or more neural networks to the first set of feature maps, the second set of feature maps, and the third set of feature maps to generate a set of mappings from a first set of pixels in the first key frame to a second set of pixels in the target frame; and generating the target frame based on the set of mappings.