- Number
- 10091435
- Published
- 2018-10-02
- Filed
- 2016-06-07
- Assignee
- Disney Enterprises, Inc.; ETH Zurich (Eidgenoessische Technische Hochschule Zurich)
- Inventors
- Zimmer; Henning; Sorkine Hornung; Alexander; Botsch; Mario; Perazzi; Federico
- CPC
- H04N23/90; G06T7/55; G06T7/11; H04N5/272; G06T7/215; G06T7/143
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Uncalibrated camera-array video segmentation (CV).
Abstract
The disclosure provides an approach for image segmentation from an uncalibrated camera array. In one aspect, a segmentation application computes a pseudo depth map for each frame of a video sequence recorded with a camera array based on dense correspondences between cameras in the array. The segmentation application then fuses such pseudo depth maps computed for satellite cameras of the camera array to obtain a pseudo depth map at a central camera. Further, the segmentation application interpolates virtual green screen positions for an entire frame based on user input which provides control points and pseudo depth thresholds at the control points. The segmentation application then computes an initial segmentation based on a thresholding using the virtual green screen positions, and refines the initial segmentation by solving a binary labeling problem in a Markov random field to better align the segmentation with image edges and provide temporal coherency for the segmentation.
Background
BACKGROUNDField of the Invention(1) Techniques presented herein relate to the field of video processing. More specifically, this disclosure presents techniques for segmenting video frames captured by an uncalibrated camera array.Description of the Related Art(2) Chroma keying using green screens is an important tool for simplifying video compositing and visual effects. Traditional chroma keying works by covering or painting parts of a set with a specific color (e.g., green), and then using color separation techniques to segment out the colored parts. Doing so simplifies the task of video segmentation, as compared to other techniques such as rotoscoping and matting, which usually require tedious manual effort. However, setting up green screens can be a significant and expensive effort, particularly for large outdoor sets. Further, for indoor sets, color spill from masked areas onto foreground objects is a common problem. In addition, actors tend to dislike green screens, which can create unnatural acting environments.SUMMARY(3) One aspect of this disclosure provides a computer-implemented method for image segmentation from a camera array. The method generally includes receiving images captured by cameras in the camera array, and determining dense correspondences between pairs of the images captured at the same time by two or more cameras in the camera array. The method further includes computing one or more pseudo depth maps from the dense correspondences, where pseudo depths
Claims
1. A computer-implemented method for image segmentation, comprising: receiving images captured by cameras in a camera array; determining optical flow fields between pairs of the images captured simultaneously by two or more cameras in the camera array; computing one or more pseudo depth maps from the optical flow fields, wherein pseudo depths in each pseudo depth map of the one or more pseudo depth maps indicate relative depths of pixels with respect to other pixels in the pseudo depth map but not actual depth measurements, and wherein at least one of the one or more pseudo depth maps is computed by performing steps including: removing from an associated one of the optical flow fields a global translational component of motion, thereby producing a residual flow field, offsetting the residual flow field by a component-wise minimum of the residual flow field, and normalizing a magnitude of the offset residual flow field to a range; and segmenting one or more of the images based, at least in part, on one or more of the computed pseudo depth maps.
10. A non-transitory computer-readable storage medium storing a program, which, when executed by a processor performs operations for image segmentation, the operations comprising: receiving images captured by cameras in a camera array; determining optical flow fields between pairs of the images captured simultaneously by two or more cameras in the camera array; computing one or more pseudo depth maps from the optical flow fields, wherein pseudo depths in each pseudo depth map of the one or more pseudo depth maps indicate relative depths of pixels with respect to other pixels in the pseudo depth map but not actual depth measurements, and wherein at least one of the one or more pseudo depth maps is computed by performing steps including: removing from an associated one of the optical flow fields a global translational component of motion, thereby producing a residual flow field, offsetting the residual flow field by a component-wise minimum of the residual flow field, and normalizing a magnitude of the offset residual flow field to a range; and segmenting one or more of the images based, at least in part, on one or more of the computed pseudo depth maps.
18. A system, comprising: a processor; and a memory, wherein the memory includes a program configured to perform operations for image segmentation, the operations comprising: receiving images captured by cameras in a camera array, determining optical flow fields between pairs of the images captured simultaneously by two or more cameras in the camera array, computing one or more pseudo depth maps from the optical flow fields, wherein pseudo depths in each pseudo depth map of the one or more pseudo depth maps indicate relative depths of pixels with respect to other pixels in the pseudo depth map but not actual depth measurements, and wherein at least one of the one or more pseudo depth maps is computed by performing steps including: removing from an associated one of the optical flow fields a global translational component of motion, thereby producing a residual flow field; offsetting the residual flow field by a component-wise minimum of the residual flow field; and normalizing a magnitude of the offset residual flow field to a range, and segmenting one or more of the images based, at least in part, on one or more of the computed pseudo depth maps.
19. A computer-implemented method for segmenting an image, comprising: receiving the image and an associated depth map; receiving user input of control points in the depth map, each of the control points having an associated depth threshold value; determining virtual green screen positions for the image based, at least in part, on an interpolation of the user input control points that generates a smooth manifold; and segmenting the image based, at least in part, on thresholding using the virtual green screen positions for the image.
22. A computer-implemented method for image segmentation, comprising: receiving images captured by cameras in a camera array; determining dense correspondences between pairs of the images captured simultaneously by two or more cameras in the camera array; computing one or more pseudo depth maps from the dense correspondences, wherein pseudo depths in each pseudo depth map of the one or more pseudo depth maps indicate relative depths of pixels with respect to other pixels in the pseudo depth map but not actual depth measurements, and wherein computing the one or more pseudo depth maps includes: computing initial pseudo depth maps for each satellite camera of the camera array based, at least in part, on dense correspondences to neighboring cameras and to a central camera of the camera array, and fusing the initial pseudo depth maps to obtain one or more pseudo depth maps at the central camera; and segmenting one or more of the images based, at least in part, on one or more of the computed pseudo depth maps.
25. A computer-implemented method for image segmentation, comprising: receiving images captured by cameras in a camera array; determining optical flow fields between pairs of the images captured simultaneously by two or more cameras in the camera array; computing a respective confidence value for each flow vector in the optical flow fields based, at least in part, on a forward-backward consistency measure specifying that a forward flow should be inverse to a backward counterpart of the foward flow; computing one or more pseudo depth maps from the dense correspondences and using the computed confidence values, wherein pseudo depths in each pseudo depth map of the one or more pseudo depth maps indicate relative depths of pixels with respect to other pixels in the pseudo depth map but not actual depth measurements; and segmenting one or more of the images based, at least in part, on one or more of the computed pseudo depth maps and the computed confidence values.