Methods and systems of performing video object segmentation

Number: 9881380
Published: 2018-01-30
Filed: 2016-02-16
Assignee: DISNEY ENTERPRISES, INC.; ETH ZÜRICH (EIDGENÖSSISCHE TECHNISCHE HOCHSCHULE ZÜRICH
Inventors: Sorkine Hornung; Alexander; Perazzi; Federico; Wang; Oliver
CPC: G06T7/143; G06T7/11; G06F18/295; G06V10/85; G06V20/49; G06F18/2414; G06T7/162; G06T7/269; G06V10/764; G06T7/215; G06T7/194; G06T7/90; G06V10/426
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Video object segmentation (CV/VFX).

Abstract

Techniques and systems are described for performing video segmentation using fully connected object proposals. For example, a number of object proposals for a video sequence are generated. A pruning step can be performed to retain high quality proposals that have sufficient discriminative power. A classifier can be used to provide a rough classification and subsampling of the data to reduce the size of the proposal space, while preserving a large pool of candidate proposals. A final labeling of the candidate proposals can then be determined, such as a foreground or background designation for each object proposal, by solving for a posteriori probability of a fully connected conditional random field, over which an energy function can be defined and minimized.

Background

FIELD(1) The present disclosure relates generally to video segmentation, and more specifically to performing video object segmentation using fully connected object proposals.BACKGROUND(2) Video segmentation includes separating video into disjoint portions representing a foreground object and background regions. Video segmentation is a building block for numerous applications, including video editing and post-processing, video retrieval, analysis of large video collections, activity recognition, and other applications. Some approaches to video segmentation are based on background subtraction, tracking of feature points and homogeneous regions, spatiotemporal graph-cuts, hierarchical clustering, object proposals, or the like.SUMMARY(3) Techniques and systems are described for performing video segmentation using fully connected object proposals. For example, given a group of video frames as input, a number of object proposals for each video frame are generated. In some examples, one or more ground proof object proposals can also be provided as input and used to generate the number of object proposals across the group of video frames. A pruning step can be performed to retain only high quality proposals that have sufficient discriminative power (e.g., object proposals that do not overlap with both the foreground and background regions of a frame). The pruning step can include a rough classification and subsampling of the data using a classifier to reduce the size of the proposal

Claims

1. A computer-implemented method of segmenting video data, the method comprising: obtaining input data, the input data including a plurality of video frames; computing object proposals for the plurality of video frames, an object proposal including a region of a video frame that is a candidate for containing at least a portion of a background object or at least a portion of a foreground object; refining the computed object proposals into a refined set of object proposals, the refined set including fewer object proposals than the computed object proposals; generating a fully-connected random field, the fully-connected random field connecting the object proposals of the refined set of object proposals across the plurality of video frames; and solving for a posteriori probability of the fully-connected random field, the posteriori probability providing a background designation or a foreground designation for each object proposal of the refined set of object proposals. 8. A system for segmenting video data, comprising: a memory storing a plurality of instructions; and one or more processors configurable to: obtain input data, the input data including a plurality of video frames; compute object proposals for the plurality of video frames, an object proposal including a region of a video frame that is a candidate for containing at least a portion of a background object or at least a portion of a foreground object; refine the computed object proposals into a refined set of object proposals, the refined set including fewer object proposals than the computed object proposals; generate a fully-connected random field, the fully-connected random field connecting the object proposals of the refined set of object proposals across the plurality of video frames; and solve for a posteriori probability of the fully-connected random field, the posteriori probability providing a background designation or a foreground designation for each object proposal of the refined set of object proposals. 15. A computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising: instructions that cause the one or more processors to obtain input data, the input data including a plurality of video frames; instructions that cause the one or more processors to compute object proposals for the plurality of video frames, an object proposal including a region of a video frame that is a candidate for containing at least a portion of a background object or at least a portion of a foreground object; instructions that cause the one or more processors to refine the computed object proposals into a refined set of object proposals, the refined set including fewer object proposals than the computed object proposals; instructions that cause the one or more processors to generate a fully-connected random field, the fully-connected random field connecting the object proposals of the refined set of object proposals across the plurality of video frames; and instructions that cause the one or more processors to solve for a posteriori probability of the fully-connected random field, the posteriori probability providing a background designation or a foreground designation for each object proposal of the refined set of object proposals.