Panoramic video from unstructured camera arrays with globally consistent parallax removal

Number: 9900505
Published: 2018-02-20
Filed: 2014-07-23
Assignee: DISNEY ENTERPRISES, INC.
Inventors: Perazzi; Federico; Sorkine Hornung; Alexander; Zimmer; Henning; Wang; Oliver; Kaufmann; Peter; Watson; Scott
CPC: G06T3/4038; H04N23/45; H04N23/698; H04N23/90; H04N5/2628; H04N5/265
Verdict: Medium Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Panoramic video stitching from camera arrays.

Abstract

Systems and methods for generating a panoramic video from unstructured camera arrays. The systems and methods are configured to statically align corresponding image-frames of respective input video streams, warp the aligned image-frames according to a warping-order, and relax the warped image-frames thereby generating a temporally coherent panoramic video. Methods according to embodiments this invention utilize a new parallax-warping-error metric that is devised to capture structural differences created by parallax artifacts. The parallax-warping-error metric is effective in finding an optimal warping-order and in driving the warping process, resulting in a panoramic video with minimal parallax artifacts.

Background

FIELD OF INVENTION(1) Embodiments of the present invention relate to methods and systems for video image stitching directed at panoramic video construction out of multiple video streams.BACKGROUND OF INVENTION(2) Panoramic presentations are popular for the immersive experience that they offer viewers. Museums, recreational parks, and theaters have been playing documentaries and entertainment content in panoramic views for quite some time now. This content has been projected onto large spherical or semi-spherical domes wherein viewers experience motion sensation and engulf themselves in a virtual reality. Other applications of interest for panoramic video are in the domain of low scale television production. Therein, a high-resolution high-quality panoramic video is constructed as an intermediate video feed from which a region-of-interest may be cut to create the final program. Such a system for live panorama construction not only allows for efficient production of low-scale events, but also provides flexibility in generating more than one program from a given panorama as well as producing playbacks of unpredictable happenings in the scene that otherwise may have been missed by traditional television coverage practices.(3) Broadcast quality cameras are limited by the field-of-view they cover. Using wide-angle lenses produces spatial distortion and image blur. Therefore, it is common in the art to combine images from multiple views to form a high-quality wide-angle panoramic vi

Claims

1. A computer-implemented method for constructing a panoramic video, comprising: receiving a plurality of input image-frames, wherein the plurality of input image-frames are temporally corresponding frames of respective input video streams; aligning the plurality of input image-frames to create a plurality of pairs of aligned image-frames and a reference panorama frame; warping the plurality of pairs of aligned image-frames according to a warping-order to create an intermediate panorama frame including a plurality of warped image-frames, wherein the warping-order is determined based on an overlap-region size corresponding to the plurality of pairs of aligned image-frames; relaxing the plurality of warped image-frames to create a relaxed panorama frame that is temporally coherent and including a plurality of relaxed image-frames; and blending the plurality of relaxed image-frames, to create an output panorama frame. 10. A system for constructing a panoramic video, comprising: a camera rig, configured to hold a plurality of cameras in an unstructured configuration, wherein the cameras operate to capture input video streams; and a processor for constructing the panoramic video, wherein the processor receives the input video streams and is configured to: buffer the input video streams to produce a plurality of input image-frames, wherein the input image-frames are temporally corresponding frames of respective input video streams; align the plurality of input image-frames to create a plurality of pairs of aligned image-frames and a reference panorama frame; warp the plurality of pairs of aligned image-frames according to a warping-order to create an intermediate panorama frame including a plurality of warped image-frames, wherein the warping-order is determined based on an overlap-region size corresponding to the plurality of pairs of aligned image-frames; relax the plurality of warped image-frames to create a relaxed panorama frame that is temporally coherent and including a plurality of relaxed image-frames; and blend the plurality of relaxed image-frames to create an output panorama frames. 19. A non-transitory computer-readable storage medium storing a set of instructions that is executable by a processor, the set of instructions, when executed by the processor, causing the processor to perform operations comprising: receiving a plurality of input image-frames, wherein the plurality of input image-frames are temporally corresponding frames of respective input video streams; aligning the plurality of input image-frames to create a plurality of pairs of aligned image-frames and a reference panorama; warping the plurality of pairs of aligned image-frames according to a warping-order to create an intermediate panorama frame including a plurality of warped image-frames, wherein the warping-order is determined based on an overlap-region size corresponding to the plurality of pairs of aligned image-frames; relaxing the plurality of warped image-frames, to create a relaxed panorama frame that is temporally coherent and including a plurality of relaxed image-frames; and blending the plurality of relaxed image-frames, to create an output panorama frame.