- Number
- 12367544
- Published
- 2025-07-22
- Filed
- 2023-05-15
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Schroers; Christopher Richard et al.
- CPC
- G06T3/4007; G06T5/50; G06T7/248; G06T7/90
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Kernel-based adaptive frame-interpolation rendering technique.
Abstract
A system includes a processor and a memory storing software code. The processor executes the software code to receive a first rendered image having first color and auxiliary buffers, receive a second rendered image having second color and auxiliary buffers, provide a third auxiliary buffer for an intermediate image to be interpolated, and identify one or more context features from each of the color buffers and the auxiliary buffers. The software code further interpolates, based on the context features, the intermediate image to provide an interpolated image, determines, based on the one or more context features from each of the first, second, and third auxiliary buffers, a rendering mask for producing one or more rendered sub-portions of the intermediate image, partially renders, using the rendering mask, the intermediate image to provide a partially rendered image, and blends the interpolated image and the partially rendered image to provide a composite image.
Background
BACKGROUND (1) Due to the ever increasing demand for high quality rendered content, there is an ongoing effort to reduce the very high costs associated with Monte Carlo renderings. Existing methods for reducing these costs include image denoising and adaptive sampling, neither of which leverages the redundancies present across multiple frames. A more recent approach for re-using information across multiple frames is based on rendered image interpolation or extrapolation, where the renderer produces a temporally downsampled sequence from which the missing frames are reconstructed. Despite this progress, challenges remain. (2) For example, in a production context, frame interpolation cannot be limited to the final color. Other feature channels (e.g., alpha channel, decomposed per light contributions) also need to be interpolated, and the result must remain consistent between all channels in order to be used in subsequent processing such as compositing. Most of the existing video frame interpolation approaches employ a direct or residual prediction neural network for the final frame synthesis, which must be retrained for every channel combination despite being unable to provide any guarantee for aligned outputs. Moreover, direct prediction networks with their unconstrained output range can undesirably produce color artifacts. Thus, there remains a need in the art for a frame interpolation solution capable of providing high quality results reliably, at acceptable cost.
Claims
1. A system comprising: a hardware processor; and a system memory storing a software code; the hardware processor configured to execute the software code to: receive a first rendered image having a first color buffer and a first auxiliary buffer; receive a second rendered image having a second color buffer and a second auxiliary buffer; provide a third auxiliary buffer for an intermediate image to be interpolated, the intermediate image being between the first rendered image and the second rendered image; identify one or more context features from each of the first color buffer, the first auxiliary buffer, the second color buffer, the second auxiliary buffer, and the third auxiliary buffer to provide a plurality of context features; interpolate, based on the plurality of context features, the intermediate image to provide an interpolated image; determine, based on the identified one or more context features from each of the first auxiliary buffer, the second auxiliary buffer, and the third auxiliary buffer, a rendering mask for producing one or more rendered sub-portions of the intermediate image; partially render, using the rendering mask, the intermediate image to provide a partially rendered image including the one or more rendered sub-portions; and blend the interpolated image and the partially rendered image to provide a composite image. ||
11. A method for use by a system including a hardware processor and a system memory storing a software code, the method comprising: receiving, by the software code executed by the hardware processor, a first rendered image having a first color buffer and a first auxiliary buffer; receiving, by the software code executed by the hardware processor, a second rendered image having a second color buffer and a second auxiliary buffer; providing, by the software code executed by the hardware processor, a third auxiliary buffer for an intermediate image to be interpolated, the intermediate image being between the first rendered image and the second rendered image; identifying, by the software code executed by the hardware processor, one or more context features from each of the first color buffer, the first auxiliary buffer, the second color buffer, the second auxiliary buffer, and the third auxiliary buffer to provide a plurality of context features; interpolating, by the software code executed by the hardware processor, based on the plurality of context features, the intermediate image to provide an interpolated image; determining, by the software code executed by the hardware processor, based on the identified one or more context features from each of the first auxiliary buffer, the second auxiliary buffer, and the third auxiliary buffer, a rendering mask for producing one or more rendered sub-portions of the intermediate image; partially rendering, by the software code executed by the hardware processor and using the rendering mask, the intermediate image to provide a partially rendered image including the one or more rendered sub-portions; and blending, by the software code executed by the hardware processor, the interpolated image and the partially rendered image to provide a composite image.