FACE SWAPPING WITH NEURAL NETWORK-BASED GEOMETRY REFINING

Number: 20250028787
Published: 2025-01-23
Filed: 2024-10-07
Assignee: DISNEY ENTERPRISES, INC.
Inventors: NARUNIEC; Jacek Krzysztof et al.
CPC: G06F18/21; G06N3/045; G06N3/0455; G06N3/0464; G06N3/08; G06N3/088; G06N3/09; G06T11/00; G06T11/10; G06T17/20; G06T19/20; G06V10/774; G06V10/82; G06V20/647; G06V40/161; G06V40/172
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Neural-network geometry-refining face-swap VFX technique.

Abstract

Various embodiments set forth systems and techniques for changing a face within an image. The techniques include receiving a first image including a face associated with a first facial identity; generating, via a machine learning model, at least a first texture map and a first position map based on the first image; rendering a second image including a face associated with a second facial identity based on the first texture map and the first position map, wherein the second facial identity is different from the first facial identity.

Background

BACKGROUND Technical Field

Embodiments of the present disclosure relate generally to computer science and computer graphics and, more specifically, to face swapping with neural network-based geometry refining. Description of the Related Art

Oftentimes, the facial identity of an individual needs to be changed in the frames of a video or in a standalone image while maintaining the performance of the individual within the video or image. As used herein, a “facial identity” refers to aspects of a facial appearance that arise from differences in personal identities, ages, lighting conditions, and the like. Thus, two different facial identities may be attributed to different individuals or to the same individual under different conditions, such as the same individual at different ages or under different lighting conditions. As used herein, the “performance” of an individual, which also is referred to as the dynamic “behavior” of an individual, includes the facial expressions and poses with which the individual appears in the frames of a video or in a standalone image.

One example scenario that requires the facial identity of an individual to be changed while maintaining the way the individual is performing is when the individual needs to be portrayed at a younger age in a particular scene within video content (e.g., a film, a show, etc.). As another example, an individual may be unavailable for a given video content production, and the face of that individual m

Claims

1. A computer-implemented method for changing a face within an image, the method comprising: receiving a first image including a face associated with a first facial identity; generating, via a machine learning model, at least a first texture map and a first position map based on the first image, wherein the machine learning model comprises a plurality of decoders, wherein each decoder included in the plurality of decoders is trained on images corresponding to a different facial identity; and rendering a second image including a face associated with a second facial identity based on the first texture map and the first position map, wherein the second facial identity is different from the first facial identity. || 10. One or more computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: receiving a first image including a face associated with a first facial identity; generating, via a machine learning model, at least a first texture map and a first position map based on the first image, wherein the machine learning model comprises a plurality of decoders, wherein each decoder included in the plurality of decoders is trained on images corresponding to a different facial identity; and rendering a second image including a face associated with a second facial identity based on the first texture map and the first position map, wherein the second facial identity is different from the first facial identity. || 19. A computer system comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the one or more processors to: receive a first image including a face associated with a first facial identity; generate, via a machine learning model, at least a first texture map and a first position map based on the first image, wherein the machine learning model comprises a plurality of decoders, wherein each decoder included in the plurality of decoders is trained on images corresponding to a different facial identity; and render a second image including a face associated with a second facial identity based on the first texture map and the first position map, wherein the second facial identity is different from the first facial identity.