Learning-based sampling for image matting

Number: 10984558
Published: 2021-04-20
Filed: 2019-05-09
Assignee: Disney Enterprises, Inc.
Inventors: Aydin; Tunc Ozan, Öztireli; Ahmet Cengiz, Tang; Jingwei, Aksoy; Yagiz
CPC: G06T7/11; G06N3/0455; G06N3/0464; G06N3/0475; G06N3/08; G06N3/09; G06N3/094; G06N20/00; G06T3/40; G06T7/194; G06T7/90; G06T11/10; G06T11/60; H04N5/272; H04N5/275
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Learning-based image matting technique for VFX compositing (granted).

Abstract

Techniques are disclosed for image matting. In particular, embodiments decompose the matting problem of estimating foreground opacity into the targeted subproblems of estimating a background using a first trained neural network, estimating a foreground using a second neural network and the estimated background as one of the inputs into the second neural network, and estimating an alpha matte using a third neural network and the estimated background and foreground as two of the inputs into the third neural network. Such a decomposition is in contrast to traditional sampling-based matting approaches that estimated foreground and background color pairs together directly for each pixel. By decomposing the matting problem into subproblems that are easier for a neural network to learn compared to traditional data-driven techniques for image matting, embodiments disclosed herein can produce better opacity estimates than such data-driven techniques as well as sampling-based and affinity-based matting approaches.

Background

COLOR DRAWINGS (1) The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. BACKGROUND Field (2) This disclosure provides techniques for image matting. Description of the Related Art (3) Image matting seeks to estimate the opacities of user-defined foreground objects in an (natural or synthetic) image, i.e., to estimate the soft transitions between foreground that is user-defined and a background, with the soft transitions defining the opacity of the foreground at each pixel. An alpha matte indicating the opacity of the foreground at each pixel (with, e.g., white indicating complete opacity, black indicating complete transparency, and shades of gray indicating partial transparency) may be obtained via image matting, and such an alpha matte is useful in many image and video editing workflows. For example, matting is a fundamental operation for various tasks during the post-production stage of feature films, such as compositing live-action and rendered elements together, and performing local color corrections. (4) One traditional approach to image matting is through sampling, in which color samples are gathered from known-opacity regions to predict foreground and background layer colors to use for alpha estimation. Such sampling-based matting typically involves selecting a color pair by using the color

Claims

1. A computer-implemented method for image matting, the method comprising: processing a received image and associated trimap using, at least in part, a first machine learning model which outputs a predicted background; processing the received image, the associated trimap, and the predicted background using, at least in part, a second machine learning model which outputs a predicted foreground; and processing the received image, the associated trimap, the predicted background, and the predicted foreground using, at least in part, a third machine learning model which outputs an alpha matte. || 11. A non-transitory computer-readable storage medium storing a program, which, when executed by a processor performs operations for image matting, the operations comprising: processing a received image and associated trimap using, at least in part, a first machine learning model which outputs a predicted background; processing the received image, the associated trimap, and the predicted background using, at least in part, a second machine learning model which outputs a predicted foreground; and processing the received image, the associated trimap, the predicted background, and the predicted foreground using, at least in part, a third machine learning model which outputs an alpha matte. || 16. A computer-implemented method of training machine learning models for image matting, the method comprising: training a first machine learning model using at least a portion of training data which includes images and corresponding trimaps; training a second machine learning model using at least a portion of the training data and backgrounds predicted by the first machine learning model subsequent to the training of the first machine learning model; and training a third machine learning model using at least a portion of the training data, backgrounds predicted by the first machine learning model, and foregrounds predicted by the second machine learning model subsequent to the training of the second machine learning model.