FLEXIBLE 3D LANDMARK DETECTION

Number: 20250118025
Published: 2025-04-10
Filed: 2024-10-04
Assignee: DISNEY ENTERPRISES, INC.
Inventors: CHANDRAN; Prashanth et al.
CPC: G06T7/70; G06V40/171; G06T17/20; G06V10/82
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Flexible 3D facial-landmark detection technique.

Abstract

One embodiment of the present invention sets forth a technique for performing landmark detection. The technique includes determining a first set of parameters associated with a depiction of a first face in a first image. The technique also includes generating, via execution of a first machine learning model, a first set of three-dimensional (3D) landmarks on the first face based on the first set of parameters, and projecting, based on the first set of parameters, the first set of 3D landmarks onto the first image to generate a first set of two-dimensional (2D) landmarks. The technique further includes training the first machine learning model based on one or more losses associated with the first set of 2D landmarks to generate a first trained machine learning model.

Background

BACKGROUND Field of the Various Embodiments

Embodiments of the present disclosure relate generally to machine learning and computer vision and, more specifically, to techniques for flexible 3D landmark detection. DESCRIPTION OF THE RELATED ART

Facial landmark detection refers to the detection of a set of specific key points, or landmarks, on a face that is depicted within an image and/or video. For example, a standard landmark detection technique may predict a set of 68 sparse landmarks that are spread across the face in a specific, predefined layout. The detected landmarks can then be used in various computer vision and computer graphics applications, such as (but not limited to) three-dimensional (3D) facial reconstruction, facial tracking, face swapping, segmentation, and/or facial re-enactment.

Deep learning approaches for predicting facial landmarks can generally be categorized into main types: direct prediction methods and heatmap prediction methods. In direct prediction methods, the x and y coordinates of the various landmarks are directly predicted by processing facial images. In heatmap prediction methods, the distribution of each landmark is first predicted, and the location of each landmark is subsequently extracted by maximizing that distribution function.

However, existing landmark detection techniques are associated with a number of drawbacks. First, most landmark detectors perform a face normalization pre-processing step that crops a

Claims

1. A computer-implemented method for performing landmark detection, the method comprising: determining a first set of parameters associated with a depiction of a first face in a first image; generating, via execution of a first machine learning model, a first set of three-dimensional (3D) landmarks on the first face based on the first set of parameters; projecting, based on the first set of parameters, the first set of 3D landmarks onto the first image to generate a first set of two-dimensional (2D) landmarks; and training the first machine learning model based on one or more losses associated with the first set of 2D landmarks to generate a first trained machine learning model. || 11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: determining a first set of parameters associated with a depiction of a first face in a first image; generating, via execution of a first machine learning model, a first set of three-dimensional (3D) landmarks on the first face based on the first set of parameters; projecting, based on the first set of parameters, the first set of 3D landmarks onto the first image to generate a first set of two-dimensional (2D) landmarks; and training the first machine learning model based on one or more losses associated with the first set of 2D landmarks to generate a first trained machine learning model. || 20. A system, comprising: one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform the steps of: determining a first machine learning model, wherein the first machine learning model is trained based on one or more losses associated with a projection of a first set of three-dimensional (3D) landmarks generated by the first machine learning model onto a two-dimensional (2D) space; determining a set of parameters associated with a depiction of a face in an image; generating, via execution of the first machine learning model, a second set of three-dimensional (3D) landmarks on the face based on the set of parameters; and reconstructing a 3D shape of the face based on the second set of 3D landmarks.