Method and system for deep learning based face swapping with multiple encoders

Number: 12452385
Published: 2025-10-21
Filed: 2022-03-29
Assignee: Disney Enterprises, Inc.
Inventors: Naruniec; Jacek Krzysztof et al.
CPC: G06V10/82; G06V40/166; G06V40/171; G06V40/172; G06V40/174; G06V40/176; G06V40/19; H04N5/272
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Deep-learning multi-encoder face-swapping VFX technique.

Abstract

A computer-implemented method of changing a face within an output image or video frame includes: receiving an input image that includes a face presenting a facial expression in a pose; separately encoding different portions of the image by, for each separately encoded portion, generating a latent space point of the portion, thereby generating a plurality of multi-dimensional vectors where each multi-dimensional vector is an encoded representation of a different portion of the input image; concatenating the plurality of multi-dimensional vectors into a combined latent space vector; and decoding the combined latent space vector to generate the output image in accordance with a desired facial identity but with the facial expression and pose of the face in the input image.

Background

CROSS REFERENCE TO RELATED APPLICATIONS (1) This application is related to concurrently filed U.S. patent application Ser. No. 17/707,782 for “METHOD AND SYSTEM FOR LATENT-SPACE FACIAL FEATURE EDITING IN DEEP LEARNING BASED FACE SWAPPING” filed on Mar. 29, 2022, the disclosure of which is hereby incorporated by reference in its entirety for all purposes. BACKGROUND OF THE INVENTION (2) Face swapping is the process of replacing an actor's face in a plate with another person's face. In visual effects, face swapping is desirable for many creative goals, like replacing the face of a stunt double with that of the main actor or to achieve de-aging by swapping the face of a present-day actor with a younger looking face learned on archival footage. In the recent past, face-swapping techniques based on deep learning have become popular and are starting to see adoption for high quality visual effects production. These techniques typically employ encoder-decoder neural networks where the encoder ingests images of the actor to be replaced (e.g. the stunt double) and outputs a “latent space point” (a lower-dimensional abstract representation of that input data). An identity-specific decoder can then transform this latent space point back into an image in which the stunt double's face is replaced with the main actor's face. (3) While some currently available deep-learning face swapping techniques can do a good job at translating the facial expression of a source actor (e.g., stunt double)

Claims

1. A computer-implemented method of changing a face within an output image or video frame, the method comprising: receiving a two-dimensional input image that includes a face presenting a facial expression in a pose; separately encoding different portions of the two-dimensional input image by, for each separately encoded portion, generating a latent space point of the portion, thereby generating a plurality of multi-dimensional vectors where each multi-dimensional vector is an encoded representation of a different portion of the two-dimensional input image; applying an adjustment vector to a multi-dimensional vector of the plurality of multi-dimensional vectors, thereby creating an adjusted vector; concatenating the plurality of multi-dimensional vectors, including the adjusted vector, into a combined latent space vector; and decoding the combined latent space vector to generate the output image in accordance with a desired facial identity but with the facial expression and pose of the face in the two-dimensional input image. || 2. The computer-implemented method of changing a face within an image or video frame set forth in claim 1 further comprising, after receiving the input image and prior to separately encoding, identifying different features within the image that correlate to the different portions of the image. || 3. The computer-implemented method of changing a face within an image or video frame set forth in claim 2 further comprising, for each of the different features identified, extracting, from the input image, an image segment that comprises the identified feature thereby generating a plurality of image segments. || 4. The computer-implemented method of changing a face within an image or video frame set forth in claim 3 wherein the plurality of image segments includes: a first image segment that contains a portion of the input image with a left eye of the face, a second image segment that contains a portion of the input image with a right eye of the face, a third image segment that contains a portion of the input image with a mouth of the face, and a fourth image segment that contains a remaining portion of the input image not included in the first, second or third image segments. || 5. The computer-implemented method of changing a face within an image or video frame set forth in claim 4 wherein each of the first, second, third and fourth image segments comprises a predetermined size. || 6. The computer-implemented method of changing a face within an image or video frame set forth in claim 1 wherein the separately encoding step comprises generating a first latent space point of a left eye of the face in the input image, generating a second latent space point of a right eye of the face in the input image, generating a third latent space point of a mouth of the face in the input image and generating a fourth latent space point of a remaining portion of the face in the input image. || 7. The computer-implemented method of changing a face within an image or video frame set forth in claim 6 wherein the concatenating comprises concatenating the first, second, third and fourth latent space points into the combined latent space vector. || 8. The computer-implemented method of changing a face within an image or video frame set forth in claim 1 wherein the input image is normalized prior to the receiving step. || 9. The computer-implemented method of changing a face within an image or video frame set forth in claim 8 wherein the input image is resized to a predetermined size prior to the receiving step. || 10. The computer-implemented method of changing a face within an image or video frame set forth in claim 1 further comprising incorporating the output image into one or more of a movie, a video, a video game or virtual or augmented reality content. || 11. The computer-implemented method of changing a face within an output image or video frame set forth in claim 1, wherein the adjustment vector is a latent space vector generated by a mathematical operation on a plurality of latent space vectors corresponding to various encoded facial expressions. || 12. The computer-implemented method of changing a face within an output image or video frame set forth in claim 1, wherein the adjustment vector is a latent space vector generated by averaging latent space vectors corresponding to various encoded facial expressions. || 13. A computer-implemented method of changing a face within an output image or video frame, the method comprising: receiving a two-dimensional input image that includes a face presenting a facial expression in a pose; dividing the two-dimensional input image into a plurality of image segments, the plurality of image segments including at least a first image segment that comprises a left eye region of the face, a second image segment that comprises a right eye region of the face, a third image segment that comprises a mouth region of the face, and a fourth image segment that comprises a remainder of the face not included in the first, second and third two-dimensional image segment; separately encoding each of the first, second, third and fourth image segments to generate first, second, third and fourth latent space points, wherein each latent space point is a multi-dimensional vector that is a lower-dimensional representation of its respective two-dimensional image segment; applying an adjustment vector to at least one of the first, second, third, and fourth latent space points, thereby adjusting at least one of the first, second, third, or fourth latent space points; concatenating the first, second, third and fourth latent space points into a combined latent space vector, wherein at least one of the first, second, third, or fourth latent space points is an adjusted latent space point; and decoding the combined latent space vector to generate the output image in accordance with a desired facial identity but with the facial expression and pose of the face in the two-dimensional input image. || 14. The computer-implemented method of changing a face within an image or video frame set forth in claim 13 wherein the input image is normalized prior to the receiving step. || 15. The computer-implemented method of changing a face within an image or video frame set forth in claim 14 wherein the input image is resized to a predetermined size prior to the receiving step. || 16. The computer-implemented method of changing a face within an image or video frame set forth in claim 15 wherein each of the first, second, third and fourth image segments comprises a predetermined size. || 17. A system for changing a face within an output image or video frame, the system comprising: a memory storing a plurality of computer-readable instructions; and one or more processors operable to execute the computer-readable instructions and cause the system to: receive a two-dimensional input image that includes a face presenting a facial expression in a pose; separately encode different portions of the two-dimensional input image by, for each separately encoded portion, generate a latent space point of the portion, thereby generating a plurality of multi-dimensional vectors where each multi-dimensional vector is an encoded representation of a different portion of the two-dimensional input image; applying an adjustment vector to a multi-dimensional vector of the plurality of multi-dimensional vectors, thereby creating an adjusted vector; concatenate the plurality of multi-dimensional vectors including the adjusted vector into a combined latent space vector; and decode the combined latent space vector to generate the output image in accordance with a desired facial identity but with the facial expression and pose of the face in the two-dimensional input image. || 18. The system set forth in claim 17 wherein the plurality of computer readable instructions further comprise instructions to cause the system to, after receiving the input image and prior to separately encoding the different portions of the image, identify different features within the image that correlate to the different portions of the image. || 19. The system set forth in claim 18 wherein the plurality of computer readable instructions further comprise instructions to cause the system to, for each of the different features identified, extract, from the input image, an image segment that comprises the identified feature thereby generating a plurality of image segments. || 20. The system set forth in claim 19 wherein the plurality of image segments includes: a first image segment that contains a portion of the input image with a left eye of the face, a second image segment that contains a portion of the input image with a right eye of the face, a third image segment that contains a portion of the input image with a mouth of the face, and a fourth image segment that contains a remaining portion of the input image not included in the first, second or third image segments. || 21. The system set forth in claim 20 wherein each of the first, second, third and fourth image segments comprises a predetermined size. || 22. The system set forth in claim 17 wherein the plurality of computer readable instructions further comprise instructions to cause the system to normalize and resize the input image prior to the receiving step.