Edge-aware spatio-temporal filtering and optical flow estimation in real time

Number: 10547871
Published: 2020-01-28
Filed: 2017-05-05
Assignee: Disney Enterprises, Inc.
Inventors: Aydin; Tunc Ozan, Scheidegger; Florian Michael, Schaffner; Michael Stefano Fritz, Cavigelli; Lukas, Benini; Luca, Smolic; Aljosa Aleksej Andrej
CPC: G06T5/20; G06T5/70; H04N19/51; H04N19/615; H04N19/80
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Real-time optical-flow/rendering filtering technique.

Abstract

The disclosure provides an approach for edge-aware spatio-temporal filtering. In one embodiment, a filtering application receives as input a guiding video sequence and video sequence(s) from additional channel(s). The filtering application estimates a sparse optical flow from the guiding video sequence using a novel binary feature descriptor integrated into the Coarse-to-fine PatchMatch method to compute a quasi-dense nearest neighbor field. The filtering application then performs spatial edge-aware filtering of the sparse optical flow (to obtain a dense flow) and the additional channel(s), using an efficient evaluation of the permeability filter with only two scan-line passes per iteration. Further, the filtering application performs temporal filtering of the optical flow using an infinite impulse response filter that only requires one filter state updated based on new guiding video sequence video frames. The resulting optical flow may then be used in temporal edge-aware filtering of the additional channel(s) using the nonlinear infinite impulse response filter.

Background

BACKGROUND(1) Field of the Invention(2) Embodiments of the disclosure presented herein relate to image and video processing and, more specifically, to efficient edge-aware spatio-temporal filtering and optical flow estimation that may be performed in real time.(3) Description of the Related Art(4) Edge-aware filters are an important tool in many image and video applications, such as high-dynamic range (HDR) tone mapping, stylization, and detail manipulation. As used herein, filtering generally refers to combining pixel values, over space or time, according to a filter kernel that defines filter weights. Edge-aware filters in particular make use of information about image edges in the filtering process and are space-variant (i.e., their kernel is dependent on the input data).(5) Traditional spatio-temporal edge-aware filters have two major drawbacks that impede their use in real-time filtering of videos. First, such filters either operate iteratively on complete video volumes (i.e., all the frames in a video) or on sliding windows of video frames, which require the entire video volume or sliding window of video frames to be stored in memory, thereby incurring high processing latency and requiring access to a large memory with high bandwidth. Second, such filters require accurate optical flow information as input in order to align neighboring frames for temporal filtering, but accurate optical flow is difficult to obtain efficiently with traditional techniques.

Claims

1. A computer-implemented method for spatio-temporal video filtering, comprising: receiving an optical flow for a video frame in a video sequence; temporally filtering the optical flow using at least an edge-aware filter in time, wherein the edge-aware filter in time uses at least an optical flow from a previous time step to align a filter state from the previous time step to the video frame; and spatially and temporally filtering video frames, from one or more additional video channels, using at least the temporally filtered optical flow. 10. A non-transitory computer-readable medium storing a program, which, when executed by a processor performs operations for spatio-temporal video filtering, the operations comprising: receiving an optical flow for a video frame in a video sequence; temporally filtering the optical flow using at least an edge-aware filter in time, wherein the edge-aware filter in time uses at least an optical flow from a previous time step to align a filter state from the previous time step to the video frame; and spatially and temporally filtering video frames, from one or more additional video channels, using at least the temporally filtered optical flow. 18. A computer-implemented method of determining correspondences between a first image and a second image in a video sequence, comprising: determining, for each of a plurality of points in the first image and in the second image, binary descriptor values of a binary descriptor formulated as a collection of binarized octal orientation maps on spatial bins around a center of an associated region of the image; and comparing the determined binary descriptor values of points in the first image with the determined binary descriptor values of points in the second image to determine correspondences in estimating an optical flow for at least one of the first and second images, wherein images from one or more additional video channels are spatially and temporally filtered using at least the optical flow. 20. A computer-implemented method for spatio-temporal video filtering, comprising: estimating an optical flow for a video frame in a video sequence, wherein correspondences between the video frame and a previous frame are determined in the optical flow estimation using at least a binary descriptor formulated as a collection of binarized octal orientation maps on spatial bins around a center of an associated region of the video frame; spatially filtering the estimated optical flow using an edge-aware spatial filter; and temporally filtering the spatially filtered optical flow using at least an edge-aware filter in time, wherein the edge-aware filter in time uses at least an optical flow from a previous time step to align a filter state from the previous time step to the video frame.