Tunable models for changing faces in images

Number: 11568524
Published: 2023-01-31
Filed: 2020-04-16
Assignee: DISNEY ENTERPRISES, INC.
Inventors: Helminger; Leonard Markus, Naruniec; Jacek Krzysztof, Weber; Romann Matthew, Schroers; Christopher Richard
CPC: G06V40/169; G06T5/20; G06T9/00; G06T9/002; G06T11/60; G06V10/32; G06V10/454; G06V10/82; G06V40/168
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Tunable face-editing VFX technique.

Abstract

Techniques are disclosed for changing the identities of faces in images. In embodiments, a tunable model for changing facial identities in images includes an encoder, a decoder, and dense layers that generate either adaptive instance normalization (AdaIN) coefficients that control the operation of convolution layers in the decoder or the values of weights within such convolution layers, allowing the model to change the identity of a face in an image based on a user selection. A separate set of dense layers may be trained to generate AdaIN coefficients for each of a number of facial identities, and the AdaIN coefficients output by different sets of dense layers can be combined to interpolate between facial identities. Alternatively, a single set of dense layers may be trained to take as input an identity vector and output AdaIN coefficients or values of weighs within convolution layers of the decoder.

Background

BACKGROUND Technical Field (1) Embodiments of the present disclosure relate generally to computer science and computer graphics and, more specifically, to tunable models for changing faces in images. Description of the Related Art (2) Oftentimes, the facial identity of an individual needs to be changed in the frames of a video or in a standalone image while maintaining a performance of the individual. As used herein, a facial identity refers to an appearance of a face that is considered distinct from other facial appearances due to differences in personal identity, age, lighting conditions, or the like. Two facial identities may be of different individuals or the same individual under different conditions, such as the same individual at different ages or under different lighting conditions. As used herein, the performance of an individual, which is also sometimes referred to as the dynamic “behavior” of the individual, includes the expressions and poses with which the individual appears in the frames of a video or a standalone image. (3) One example scenario that requires a facial identity to be changed while maintaining the performance of an individual is when the individual needs to be portrayed at a younger age in a particular scene within a film. As another example, an individual may be unavailable for a given film production, and the face of that individual may need to be inserted into a particular scene to replace the face of another individual who was available to perf

Claims

1. A computer-implemented method for changing a face within an image or video frame, the method comprising: encoding an input image that includes a first face to generate a latent representation of the input image; and decoding the latent representation based on one or more parameters that are determined via at least one dense layer of a machine learning model to generate an output image that includes a second face, wherein the one or more parameters are used to change a first facial identity of the first face to a second facial identity of the second face. || 10. A computer-implemented method for changing a face within an image or video frame, the method comprising: encoding an input image that includes a face to generate a latent representation of the input image; and decoding the latent representation based on one or more parameters that are determined via at least one dense layer of a machine learning model to generate an output image, wherein the latent representation is decoded using a decoder that includes a plurality of levels, and each level included in the plurality of levels comprises a convolution layer and an upscaling layer, and wherein the one or more parameters are determined by processing, for each level included in the plurality of levels, an output generated by the convolution layer included in the level using a corresponding dense layer included in the at least one dense layer. || 12. A non-transitory computer-readable storage medium including instructions that, when executed by a processing unit, cause the processing unit to perform steps for changing a face within an image or video frame, the steps comprising: encoding an input image that includes a first face to generate a latent representation of the input image; and decoding the latent representation based on one or more parameters that are determined via at least one dense layer of a machine learning model to generate an output image that includes a second face, wherein the one or more parameters are used to change a first facial identity of the first face to a second facial identity of the second face. || 20. A computing device comprising: a memory storing an application; and a processor coupled to the memory, wherein when executed by the processor, the application causes the processor to: encode an input image that includes a first face to generate a latent representation of the input image, and decode the latent representation based on one or more parameters that are determined via at least one dense layer of a machine learning model to generate an output image that includes a second face, wherein the one or more parameters are used to change a first facial identity of the first face to a second facial identity of the second face.