SUBJECT-AGNOSTIC FACE SWAPPING WITH LOW-RANK ADAPTATION

Number: 20250308117
Published: 2025-10-02
Filed: 2025-03-28
Assignee: Disney Enterprises, Inc.
Inventors: Kansy; Manuel Jakob et al.
CPC: G06N3/0455; G06T11/60
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Subject-agnostic face-swapping VFX technique.

Abstract

In some embodiments, a method generates a first representation of a first image including a first facial identity and generates an identity representation from a second image that describes a second facial identity of the second image. The identity representation is mapped to a set of low-rank adaptation weights. The method adapts the first representation to an adapted first representation using the set of low-rank adaptation weights that are applied to a layer in a model. Decoder input values are generated based on the adapted first representation. The method performs decoding using the decoder input values to generate an output image. The output image swaps the first facial identity of the first image with the second facial identity of the second image..

Background

BACKGROUND

Face swapping refers to the changing of the facial identity of an individual in standalone images or video frames while maintaining the performance of the individual within the standalone images or video frames. This facial identity may include aspects of a facial appearance that arise from differences in personal identities, ages, eye colors, or other factors. For example, two different facial identities may be attributed to two different individuals, the same individual under different lighting conditions, or the same individual at different ages. Further, the performance of an individual, which also is referred to as the dynamic behavior of an individual, includes the facial expressions and poses of the individual, as depicted in the video frames or in standalone images.

Face swapping can be conducted under various types of scenarios. For example, the facial identity of an actor within a given scene of video content (e.g., a film, a show, etc.) could be changed to a different facial identity of the same actor at a younger age or at an older age. In another example, a first actor could be unavailable for a video shoot because of scheduling conflicts, because the first actor is deceased, or for other reasons. To incorporate the likeness of the first actor into video content generated during the shoot, footage of a second actor could be captured during the shoot, and the face of the second actor in the footage could be replaced with the face of the fi

Claims

1. A method comprising: generating a first representation of a first image including a first facial identity; generating an identity representation from a second image that describes a second facial identity of the second image; mapping the identity representation to a set of low-rank adaptation weights; adapting the first representation to an adapted first representation using the set of low-rank adaptation weights that are applied to a layer in a model; generating, by the model, decoder input values based on the adapted first representation; and performing decoding using the decoder input values to generate an output image, wherein the output image swaps the first facial identity of the first image with the second facial identity of the second image. || 17. A non-transitory computer-readable storage medium having stored thereon computer executable instructions, which when executed by a computing device, cause the computing device to be operable for: generating a first representation of a first image including a first facial identity; generating an identity representation from a second image that describes a second facial identity of the second image; mapping the identity representation to a set of low-rank adaptation weights; adapting the first representation to an adapted first representation using the set of low-rank adaptation weights that are applied to a layer in a model; generating, by the model, decoder input values based on the adapted first representation; and performing decoding using the decoder input values to generate an output image, wherein the output image swaps the first facial identity of the first image with the second facial identity of the second image. || 20. An apparatus comprising: one or more computer processors; and a computer-readable storage medium comprising instructions for controlling the one or more computer processors to be operable for: generating a first representation of a first image including a first facial identity; generating an identity representation from a second image that describes a second facial identity of the second image; mapping the identity representation to a set of low-rank adaptation weights; adapting the first representation to an adapted first representation using the set of low-rank adaptation weights that are applied to a layer in a model; generating, by the model, decoder input values based on the adapted first representation; and performing decoding using the decoder input values to generate an output image, wherein the output image swaps the first facial identity of the first image with the second facial identity of the second image.