Learning-based sampling for image matting

Number: 11615555
Published: 2023-03-28
Filed: 2021-04-09
Assignee: DISNEY ENTERPRISES, INC.
Inventors: Aydin; Tunc Ozan et al.
CPC: G06T7/11; G06N3/0455; G06N3/0464; G06N3/0475; G06N3/08; G06N3/09; G06N3/094; G06N20/00; G06T3/40; G06T7/194; G06T7/90; G06T11/10; G06T11/60; H04N5/272; H04N5/275
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Learning-based image matting technique for VFX compositing (granted).

Abstract

A method of generating a training data set for training an image matting machine learning model includes receiving a plurality of foreground images, generating a plurality of composited foreground images by compositing randomly selected foreground images from the plurality of foreground images, and generating a plurality of training images by compositing each composited foreground image with a randomly selected background image. The training data set includes the plurality of training images.

Background

BACKGROUND Field (1) This disclosure provides techniques for image matting. Description of the Related Art (2) Image matting seeks to estimate the opacities of user-defined foreground objects in an (natural or synthetic) image, i.e., to estimate the soft transitions between foreground that is user-defined and a background, with the soft transitions defining the opacity of the foreground at each pixel. An alpha matte indicating the opacity of the foreground at each pixel (with, e.g., white indicating complete opacity, black indicating complete transparency, and shades of gray indicating partial transparency) may be obtained via image matting, and such an alpha matte is useful in many image and video editing workflows. For example, matting is a fundamental operation for various tasks during the post-production stage of feature films, such as compositing live-action and rendered elements together, and performing local color corrections. (3) One traditional approach to image matting is through sampling, in which color samples are gathered from known-opacity regions to predict foreground and background layer colors to use for alpha estimation. Such sampling-based matting typically involves selecting a color pair by using the color line assumption and other metrics from the spatial proximity of samples among others. However, an inherent shortcoming of the sampling-based matting approach is the lack of consideration for image structure and texture during the sample selection process

Claims

1. A method of training an image matting machine learning model, the method comprising: generating a training data set by: receiving a plurality of foreground images; generating a plurality of composited foreground images by compositing randomly selected foreground images from the plurality of foreground images; and generating a plurality of training images by compositing each composited foreground image with a randomly selected background image, wherein the training data set includes the plurality of training images; training, using the training data set, at least one of (1) a foreground sampling machine learning network configured to predict a foreground, or (2) a background sampling machine learning network configured to predict a background; and training the image matting machine learning model, at least in part, using the training data set and predictions made using the at least one of the foreground sampling machine learning network or the background sampling machine learning network. || 3. A method of generating a training data set for training an image matting machine learning model, the method comprising: receiving a plurality of foreground images; generating a plurality of composited foreground images by compositing randomly selected foreground images from the plurality of foreground images; and generating a plurality of training images by compositing each composited foreground image with a randomly selected background image, wherein the training data set comprises the plurality of training images. || 12. An apparatus for generating a training data set for training an image matting machine learning model, the apparatus comprising: a memory; and a hardware processor communicatively coupled to the memory, the hardware processor configured to: receive a plurality of foreground images; generate a plurality of composited foreground images by compositing randomly selected foreground images from the plurality of foreground images; and generate a plurality of training images by compositing each composited foreground image with a randomly selected background image, wherein the training data set comprises the plurality of training images.