Controllable Video Frame Interpolation with Latent Blending and Motion Alignment

Number: 20250356455
Published: 2025-11-20
Filed: 2025-02-21
Assignee: Disney Enterprises, Inc.
Inventors: Djelouah; Abdelaziz et al.
CPC: G06T3/4007; G06T3/18; G06T7/248; G06T7/74; H04N7/0127; H04N7/014
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Controllable frame-interpolation rendering technique (related cluster).

Abstract

A system includes a processor and a memory storing software code including a video frame interpolation machine-learning (ML) model. The processor executes the software code to receive an input video sequence including a first video frame and a second video frame, obtain point tracks between the first video frame and the second video frame, identify a target position for an interpolated video frame and determine, using the point tracks, a first optical flow between the target position and the first video frame, and a second optical flow between the target position and the second video frame. The processor further executes the software code to warp, using the first optical flow and the second optical flow, respectively, the first video frame and the second video frame, respectively, and predict, using the video frame interpolation ML model, the warped first video frame and the warped second video frame, the interpolated video frame.

Background

BACKGROUND

Video frame interpolation is a commonly used post-processing technique that can be used for frame rate adjustment, novel-view synthesis and the generation of artistic slow-motion effects, for example. Although advances in video frame interpolation made in recent years have significantly improved the quality of interpolated frames, finding correspondences for large displacements between keyframes and compensating for that motion remains a challenging problem. Moreover, because video frame interpolation is an ill-posed problem, it can result in generation of plausible intermediate frames that can differ disturbingly from user expectations. Nevertheless, to date little research has been directed to solutions for controlling interpolated outputs. Thus, there remains a need in the art for a video frame interpolation solution that is both controllable and provides motion alignment.

Claims

1. A system comprising: a hardware processor; and a system memory storing a software code including a video frame interpolation machine-learning (ML) model; the hardware processor configured to execute the software code to: receive an input video sequence including at least a first video frame and a second video frame; obtain a plurality of point tracks between the first video frame and the second video frame; identify a target position for an interpolated video frame between the first video frame and the second video frame; determine, using the plurality of point tracks, a first optical flow between the target position and the first video frame, and a second optical flow between the target position and the second video frame; warp, using the first optical flow and the second optical flow, respectively, the first video frame and the second video frame, respectively; and predict, using the video frame interpolation ML model, the warped first video frame and the warped second video frame, the interpolated video frame. || 11. A method for use by a system including a hardware processor and a system memory storing a software code including a video frame interpolation machine learning (ML) model, the method comprising: receiving, by the software code executed by the hardware processor, an input video sequence including at least a first video frame and a second video frame; obtaining, by the software code executed by the hardware processor, a plurality of point tracks between the first video frame and the second video frame; identifying, by the software code executed by the hardware processor, a target position for an interpolated video frame between the first video frame and the second video frame; determining, by the software code executed by the hardware processor and using the plurality of point tracks, a first optical flow between the target position and the first video frame, and a second optical flow between the target position and the second video frame; warping, by the software code executed by the hardware processor and using the first optical flow and the second optical flow, respectively, the first video frame and the second video frame, respectively; and predicting, by the software code executed by the hardware processor and using the video frame interpolation ML model, the warped first video frame and the warped second video frame, the interpolated video frame.