- Number
- 20230319223
- Published
- 2023-10-05
- Filed
- 2022-03-29
- Assignee
- DISNEY ENTERPRISES, INC
- Inventors
- Naruniec; Jacek Krzysztof et al.
- CPC
- G06V10/82; G06V40/166; G06V40/171; G06V40/172; G06V40/174; G06V40/176; G06V40/19; H04N5/272
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Multi-encoder deep-learning face-swapping VFX technique.
Abstract
A computer-implemented method of changing a face within an output image or video frame includes: receiving an input image that includes a face presenting a facial expression in a pose; separately encoding different portions of the image by, for each separately encoded portion, generating a latent space point of the portion, thereby generating a plurality of multi-dimensional vectors where each multi-dimensional vector is an encoded representation of a different portion of the input image; concatenating the plurality of multi-dimensional vectors into a combined latent space vector; and decoding the combined latent space vector to generate the output image in accordance with a desired facial identity but with the facial expression and pose of the face in the input image
Background
BACKGROUND OF THE INVENTION
Face swapping is the process of replacing an actor's face in a plate with another person's face. In visual effects, face swapping is desirable for many creative goals, like replacing the face of a stunt double with that of the main actor or to achieve de-aging by swapping the face of a present-day actor with a younger looking face learned on archival footage. In the recent past, face-swapping techniques based on deep learning have become popular and are starting to see adoption for high quality visual effects production. These techniques typically employ encoder-decoder neural networks where the encoder ingests images of the actor to be replaced (e.g. the stunt double) and outputs a “latent space point” (a lower-dimensional abstract representation of that input data). An identity-specific decoder can then transform this latent space point back into an image in which the stunt double's face is replaced with the main actor's face.
While some currently available deep-learning face swapping techniques can do a good job at translating the facial expression of a source actor (e.g., stunt double) to target character in many instances, improvements in this regard are continuously being sought. In particular, one challenge with many deep learning techniques is the lack of control over the swapped image. For example, the eye gaze in the replaced face might be slightly off and there is no clear, easy way to correct the eye gaze direction. BRIEF
Claims
1. A computer-implemented method of changing a face within an output image or video frame, the method comprising: receiving an input image that includes a face presenting a facial expression in a pose; separately encoding different portions of the image by, for each separately encoded portion, generating a latent space point of the portion, thereby generating a plurality of multi-dimensional vectors where each multi-dimensional vector is an encoded representation of a different portion of the input image; concatenating the plurality of multi-dimensional vectors into a combined latent space vector; and decoding the combined latent space vector to generate the output image in accordance with a desired facial identity but with the facial expression and pose of the face in the input image. ||
2. The computer-implemented method of changing a face within an image or video frame set forth in claim 1 further comprising, after receiving the input image and prior to separately encoding, identifying different features within the image that correlate to the different portions of the image. ||
3. The computer-implemented method of changing a face within an image or video frame set forth in claim 2 further comprising, for each of the different features identified, extracting, from the input image, an image segment that comprises the identified feature thereby generating a plurality of image segments. ||
4. The computer-implemented method of changing a face within an image or video frame set forth in claim 3 wherein the plurality of image segments includes: a first image segment that contains a portion of the input image with a left eye of the face, a second image segment that contains a portion of the input image with a right eye of the face, a third image segment that contains a portion of the input image with a mouth of the face, and a fourth image segment that contains a remaining portion of the input image not included in the first, second or third image segments. ||
5. The computer-implemented method of changing a face within an image or video frame set forth in claim 5 wherein each of the first, second, third and fourth image segments comprises a predetermined size. ||
6. The computer-implemented method of changing a face within an image or video frame set forth in claim 1 wherein the separately encoding step comprises generating a first latent space point of a left eye of the face in the input image, generating a second latent space point of a right eye of the face in the input image, generating a third latent space point of a mouth of the face in the input image and generating a fourth latent space point of a remaining portion of the face in the input image. ||
7. The computer-implemented method of changing a face within an image or video frame set forth in claim 6 wherein the concatenating comprises concatenating the first, second, third and fourth latent space points into the combined latent space vector. ||
8. The computer-implemented method of changing a face within an image or video frame set forth in claim 1 wherein the input image is normalized prior to the receiving step. 9 The computer-implemented method of changing a face within an image or video frame set forth in claim 8 wherein the input image is resized to a predetermined size prior to the receiving step. ||
10. The computer-implemented method of changing a face within an image or video frame set forth in claim 1 further comprising incorporating the output image into one or more of a movie, a video, a video game or virtual or augmented reality content. ||
11. A computer-implemented method of changing a face within an output image or video frame, the method comprising: receiving an input image that includes a face presenting a facial expression in a pose; dividing the input image into a plurality of image segments, the plurality of image segments including at least a first image segment that comprises a left eye region of the face, a second image segment that comprises a right eye region of the face, a third image segment that comprises a mouth region of the face, and a fourth image segment that comprises a remainder of the face not included in the first, second and third image segment; separately encoding each of the first, second, third and fourth image segments to generate first, second, third and fourth latent space points, wherein each latent space point is a multi-dimensional vector that is a lower-dimensional representation of its respective image segment; concatenating the first, second, third and fourth latent space points into a combined latent space vector; and decoding the combined latent space vector to generate the output image in accordance with a desired facial identity but with the facial expression and pose of the face in the input image. ||
12. The computer-implemented method of changing a face within an image or video frame set forth in claim 11 wherein the input image is normalized prior to the receiving step. 13 The computer-implemented method of changing a face within an image or video frame set forth in claim 12 wherein the input image is resized to a predetermined size prior to the receiving step. ||
14. The computer-implemented method of changing a face within an image or video frame set forth in claim 13 wherein each of the first, second, third and fourth image segments comprises a predetermined size. ||
15. A system for changing a face within an output image or video frame, the system comprising: a memory storing a plurality of computer-readable instructions; and one or more processors operable to execute the computer-readable instructions and cause the system to: receive an input image that includes a face presenting a facial expression in a pose; separately encode different portions of the image by, for each separately encoded portion, generate a latent space point of the portion, thereby generating a plurality of multi-dimensional vectors where each multi-dimensional vector is an encoded representation of a different portion of the input image; concatenate the plurality of multi-dimensional vectors into a combined latent space vector; and decode the combined latent space vector to generate the output image in accordance with a desired facial identity but with the facial expression and pose of the face in the input image. ||
16. The system set forth in claim 15 wherein the plurality of computer readable instructions further comprise instructions to cause the system to, after receiving the input image and prior to separately encoding the different portions of the image, identify different features within the image that correlate to the different portions of the image. ||
17. The system set forth in claim 16 wherein the plurality of computer readable instructions further comprise instructions to cause the system to, for each of the different features identified, extract, from the input image, an image segment that comprises the identified feature thereby generating a plurality of image segments. ||
18. The system set forth in claim 17 wherein the plurality of image segments includes: a first image segment that contains a portion of the input image with a left eye of the face, a second image segment that contains a portion of the input image with a right eye of the face, a third image segment that contains a portion of the input image with a mouth of the face, and a fourth image segment that contains a remaining portion of the input image not included in the first, second or third image segments. ||
19. The system set forth in claim 18 wherein each of the first, second, third and fourth image segments comprises a predetermined size. ||
20. The system set forth in claim 15 wherein the plurality of computer readable instructions further comprise instructions to cause the system to normalize and resize the input image prior to the receiving step.