- Number
- 10885708
- Published
- 2021-01-05
- Filed
- 2018-10-16
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Guay; Martin, Cimen; Gökcen, Maurhofer; Christoph, Ryffel; Mattia, Sumner; Robert W.
- CPC
- G06T7/75; G06N5/01; G06N3/02; G06T19/00; G06N3/0464
- Verdict
- Medium Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Shape-estimation technique for costume augmentation.
Abstract
An automated costume augmentation system includes a computing platform having a hardware processor and a system memory storing a software code. The hardware processor executes the software code to provide an image including a posed figure to an artificial neural network (ANN), receive from the ANN a 2D skeleton data including joint positions corresponding to the posed figure, and determine a 3D pose corresponding to the posed figure using an optimization algorithm applied to the skeleton data. The software code further identifies one or more proportion(s) of the posed figure based on the skeleton data, determines bone directions corresponding to the posed figure using another optimization algorithm applied to the 3D pose, parameterizes a costume for the posed figure based on the 3D pose, the proportion(s), and the bone directions, and outputs an enhanced image including the posed figure augmented with the fitted costume for rendering on a display.
Background
BACKGROUND (1) Despite the widespread use of augmented reality (AR) techniques to enhance many real world objects with virtual imagery, obstacles remain to realistically augmenting the figure of a living body with virtual enhancements. For example, due to the ambiguities associated with depth projection, the variations in body shapes, and the variety of poses a body may assume, three-dimensional (3D) shape estimation of a body from a red-green-blue (RGB) image is an under-constrained and ambiguous problem. As a result, augmenting the image of a human body, for example, with a virtual costume that is realistically fitted to the 3D shape of the human body presents significant challenges. (2) Although solutions for estimating a 3D human pose exist, they are insufficient to the goal of matching the borders and contours of a digital costume augmentation to the 3D shape of the human figure assuming the pose. For instance, applying a digital costume to a human figure by merely overlaying the costume onto an image of the human figure based on 3D pose matching typically results in clothing or skin of the human model remaining visible. SUMMARY (3) There are provided systems and methods for performing automated costume augmentation using shape estimation, substantially as shown in and/or described in connection with at least one of the figures, and as set forth more completely in the claims.
Claims
1. An automated costume augmentation system comprising a computing platform including a hardware processor and a system memory storing a software code, the hardware processor configured to execute the software code to: determine a three-dimensional (3D) pose corresponding to a two-dimensional (2D) skeleton data including a plurality of joint positions corresponding to a posed figure in an image, using a first optimization algorithm applied to the 2D skeleton data, the first optimization algorithm including a first objective function; identify at least one proportion of the posed figure based on the 2D skeleton data; determine a plurality of bone directions corresponding to the posed figure, using a second optimization algorithm applied to the 3D pose, wherein the second optimization algorithm includes the first objective function summed with a second objective function configured to match the plurality of bone directions; parameterize a costume for fitting to the posed figure based on the 3D pose, the at least one proportion, and the plurality of bone directions; and output an enhanced image including the posed figure augmented with the costume fitted to the posed figure for rendering on a display. ||
9. A method for use by an automated costume augmentation system including a computing platform having a hardware processor executing a software code stored in a system memory, the method comprising: determining a three-dimensional (3D) pose corresponding to a two-dimensional (2D) skeleton data including a plurality of joint positions corresponding to a posed figure in an image, using a first optimization algorithm applied to the 2D skeleton data, the first optimization algorithm including a first objective function; identifying at least one proportion of the posed figure based on the 2D skeleton data; determining a plurality of bone directions corresponding to the posed figure, using a second optimization algorithm applied to the 3D pose, wherein the second optimization algorithm includes the first objective function summed with a second objective function configured to match the plurality of bone directions; parameterizing a costume for fitting to the posed figure based on the 3D pose, the at least one proportion, and the plurality of bone directions; and outputting an enhanced image including the posed figure augmented with the costume fitted to the posed figure for rendering on a display. ||
17. A computer-readable non-transitory medium having stored thereon a software code including instructions, which when executed by a hardware processor, instantiate a method comprising: determining a three-dimensional (3D) pose corresponding to a two-dimensional (2D) skeleton data including a plurality of joint positions corresponding to a posed figure in an image, using a first optimization algorithm applied to the 2D skeleton data, the first optimization algorithm including a first objective function; identifying at least one proportion of the posed figure based on the 2D skeleton data; determining a plurality of bone directions corresponding to the posed figure, using a second optimization algorithm applied to the 3D pose, wherein the second optimization algorithm includes the first objective function summed with a second objective function configured to match the plurality of bone directions; parameterizing a costume for fitting to the posed figure based on the 3D pose, the at least one proportion, and the plurality of bone directions; and outputting an enhanced image including the posed figure augmented with the costume fitted to the posed figure for rendering on a display.