METHOD FOR TEMPORAL STABILIZATION OF LANDMARK LOCALIZATION

Number: 20220058822
Published: 2022-02-24
Filed: 2020-08-24
Assignee: DISNEY ENTERPRISES, INC.
Inventors: NARUNIEC; Jacek Krzysztof, SCHROERS; Christopher Richard, WEBER; Romann Matthew
CPC: G06F18/217; G06N20/00; G06N3/0442; G06N3/0455; G06N3/0464; G06N3/0475; G06N3/0895; G06N3/09; G06N3/0985; G06N5/04; G06T3/02; G06T3/20; G06T3/40; G06T3/60; G06T7/248; G06T7/73; G06V10/7747; G06V10/82; G06V40/165; G06V40/171
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Temporal stabilization technique for facial landmark tracking (VFX).

Abstract

Various embodiments set forth systems and techniques for training a landmark model. The techniques include determining, using the landmark model, a first landmark in a set of first landmarks associated with a first image; performing, on the first image, a first perturbation to obtain a second image; determining, using the landmark model, a second landmark in a set of second landmarks associated with the second image; determining, based on a first distance between the first landmark and the second landmark, a first loss function; and updating, based on the first loss function, a first parameter of the landmark model.

Background

BACKGROUND Field of the Various Embodiments

The various embodiments relate generally to computer science and landmark localization in images and, more specifically, to a method for temporal stabilization of landmark localization. Description of the Related Art

Many computer vision and computer graphics applications rely on landmark localization. Such applications include face swapping, face puppeteering, face recognition, face reenactment, face animation, digital avatars, or the like. Landmarks, such as facial landmarks, can be used as anchoring points for models, such as, 3D face appearance or autoencoders. Locations of localized landmarks are used, for instance, to spatially align faces. In some applications, temporary stable landmarks, such as facial landmarks, are important for enabling visual effects on faces, for tracking eye gaze, or the like.

However, precise landmark localization is a longstanding problem. For instance, facial landmark localization on image sequences often results in temporally unstable results. Such instability may be caused by various factors such as inconsistency or inaccuracy of manual ground truth labels on training data used to train the underlying landmark algorithm, insufficient training data, inaccuracies in the landmark algorithm, or the like.

The instability can result in undesired effects. For instance, instability in localized landmarks can result in degradation in face recognition performance. Such degradatio

Claims

1. A computer-implemented method for training a landmark model, the method comprising: determining, using the landmark model, a first landmark in a set of first landmarks associated with a first image; performing, on the first image, a first perturbation to obtain a second image; determining, using the landmark model, a second landmark in a set of second landmarks associated with the second image; determining, based on a first distance between the first landmark and the second landmark, a first loss function; and updating, based on the first loss function, a first parameter of the landmark model. || 11. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform the steps of: determining, using a landmark model, a first landmark in a set of first landmarks associated with a first image; performing, on the first image, a first perturbation to obtain a second image; determining, using the landmark model, a second landmark in a set of second landmarks associated with the second image; determining, based on a first distance between the first landmark and the second landmark, a first loss function; and updating, based on the first loss function, a first parameter of the landmark model. || 20. A system, comprising: a memory storing one or more software applications; and a processor that, when executing the one or more software applications, is configured to perform the steps of: determining, using a landmark model, a first landmark in a set of first landmarks associated with a first image; performing, on the first image, a first perturbation to obtain a second image; determining, using the landmark model, a second landmark in a set of second landmarks associated with the second image; determining, based on a first distance between the first landmark and the second landmark, a first loss function; and updating, based on the first loss function, a first parameter of the landmark model.