Uncertainty-Guided Frame Interpolation for Video Rendering

Number: 20240163395
Published: 2024-05-16
Filed: 2023-11-10
Assignee: Disney Enterprises, Inc.
Inventors: Djelouah; Abdelaziz et al.
CPC: G06T3/4007; H04N7/0135; G06T3/18; G06T3/40
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Uncertainty-guided frame-interpolation rendering technique.

Abstract

A system includes a hardware processor, a memory storing software code, and a machine learning (ML) model-based video frame interpolator. The hardware processor executes the software code to provide first and second frames of a video sequence including a plurality of frames, respective binary masks for the first and second frames, and optionally an intermediate frame of the video sequence between the first and second frames and a binary mask for the intermediate frame, as interpolation inputs to the ML model-based video frame interpolator. The hardware processor further executes the software code to generate, using the ML model-based video frame interpolator and the interpolation inputs, an interpolated frame and an error map for the interpolated frame, wherein generating the interpolated frame and the error map includes a cross-backward warping of respective latent feature representations of each of the plurality of frames.

Background

BACKGROUND

Video frame interpolation enables many practical applications, such as video editing, novel-view synthesis, video retiming, and slow motion generation, for example. Recently, different deep learning video frame interpolation methods have been proposed. However, those conventional methods fail to generalize their interpolation results to animated data. In addition, retraining a method for each specific use case is not a viable solution, as the data statistics in video content or can vary drastically, sometimes even within the same scene. Thus, despite recent advances in the field, video frame interpolation remains an open challenge due to the complex lighting effects and large motion that are ubiquitous in video content and can introduce severe artifacts for existing methods.

Claims

1. A system comprising: a hardware processor; a system memory storing a software code; and a machine learning (ML) model-based video frame interpolator; the hardware processor configured to execute the software code to: provide a first frame of a video sequence including a plurality of frames, a binary mask for the first frame, a second frame of the video sequence, and a binary mask for the second frame, as interpolation inputs to the ML model-based video frame interpolator; generate, using the ML model-based video frame interpolator and the interpolation inputs, an interpolated frame and an error map for the interpolated frame; wherein generating the interpolated frame and the error map includes a cross-backward warping of respective latent feature representations of each of the plurality of frames. || 11. A method for use by a system including a hardware processor and a system memory storing a software code and a machine learning (ML) model-based video frame interpolator, the method comprising: a hardware processor; providing, by the software code executed by the hardware processor, a first frame of a video sequence including a plurality of frames, a binary mask for the first frame, a second frame of the video sequence, and a binary mask for the second frame as interpolation inputs to the ML model-based video frame interpolator; generating, by the software code executed by the hardware processor and using the ML model-based video frame interpolator and the interpolation inputs, an interpolated frame and an error map for the interpolated frame; wherein generating the interpolated frame and the error map includes a cross-backward warping of respective latent feature representations of each of the plurality of frames.