VIDEO SEGMENTATION FROM AN UNCALIBRATED CAMERA ARRAY

Number: 20170353670
Published: 2017-12-07
Filed: 2016-06-07
Assignee: Disney Enterprises, Inc.
Inventors: ZIMMER; Henning, SORKINE HORNUNG; Alexander, BOTSCH; Mario, PERAZZI; Federico
CPC: H04N5/272; H04N23/90; G06T7/215; G06T7/143; G06T7/11; G06T7/55
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Segments video from an uncalibrated multi-camera array by computing and fusing pseudo depth maps across cameras.

Abstract

The disclosure provides an approach for image segmentation from an uncalibrated camera array. In one aspect, a segmentation application computes a pseudo depth map for each frame of a video sequence recorded with a camera array based on dense correspondences between cameras in the array. The segmentation application then fuses such pseudo depth maps computed for satellite cameras of the camera array to obtain a pseudo depth map at a central camera. Further, the segmentation application interpolates virtual green screen positions for an entire frame based on user input which provides control points and pseudo depth thresholds atthe control points. The segmentation application then computes an initial segmentation based on a thresholding using the virtual green screen positions, and refines the initial segmentation by solving a binary labeling problem in a Markov random field to better align the segmentation with image edges and provide temporal coherency for the segmentation.

Background

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited aspects are attained and can be understood in detail, a more particular description of aspects of this disclosure, briefly summarized above, may be had by reference to the appended drawings.

It is to be noted, however, that the appended drawings illustrate only typical aspectsand are therefore not to be considered limiting of its scope, for the invention may admitto other equally effective aspects.

FIG. 1 illustrates the layout of an example camera array, according to an aspect of this disclosure.

FIG. 2 illustrates an approachfor computing pseudo depths based on dense correspondences from an optical flow, according to an aspect of this disclosure.

FIG. 3 illustrates fusing of multiple pseudo depths into a single pseudo depth at a reference camera, according to an aspect of this disclosure.

FIG. 4 illustrates an approach for interpolating virtual green screen positions for the one or more images based on user-defined control points, according to an aspect of this disclosure.

FIG. 5 illustrates an approach for refining image segmentation, according to an aspect of this disclosure.

FIG. 6 illustrates a method for segmenting video from an uncalibrated camera array, according to an aspect of this disclosure.

FIG. 7 illustrates a computer system in which an aspect of this disclosure may be implemented.

To facilitate understanding, identical

Claims

1. A computer-implemented method for image segmentation, comprising: receiving images captured by cameras in a camera array; determining dense correspondences between pairs of the images captured at the same time by two or more cameras in the camera array; computing one or more pseudo depth maps from the dense correspondences, wherein pseudo depths ineach of the pseudo depth maps indicate relative depths of pixels with respect to other pixels in the pseudo depth map but not actual depth measurements; and segmenting one or moreof the images based on the computed pseudo depth maps. 11. A non-transitory computer-readable storage medium storing a program, which, when executed by a processor performs operations for image segmentation, the operations comprising: receiving images captured by cameras in a camera array; determining dense correspondences between pairs of the images captured at the same time by two or more cameras in the camera array; computing one or more pseudo depth maps from the dense correspondences, wherein pseudo depths in each of the pseudo depth maps indicate relative depths of pixels with respect to other pixels in the pseudo depth map but not actual depth measurements; and segmenting one or more of the images basedon the computed pseudo depth maps. 20. A system, comprising: a processor; and a memory, wherein the memory includes a program configured to perform operations for image segmentation, the operations comprising: receiving images captured by cameras in a camera array, determining dense correspondences between pairs of the images captured at the same time by two or more cameras in the camera array, computing one or more pseudo depth maps from the dense correspondences, wherein pseudo depths in each of the pseudo depth maps indicate relative depths of pixels with respect to other pixels in the pseudo depth map but not actual depth measurements, and segmenting one or more of the images based on the computed pseudo depth maps. 21. A computer-implemented method for segmenting an image, comprising: receiving the image and an associated depth map; receiving user input of one or more control points in the image and depth thresholds at the control points; interpolating virtual green screen positions for the image based on the user input; and segmenting the image based on thresholding using the virtual green screen positions for the image.