ADAPTIVE CONVOLUTIONS IN NEURAL NETWORKS

Number: 20220156987
Published: 2022-05-19
Filed: 2021-04-06
Assignee: DISNEY ENTERPRISES, INC.
Inventors: CHANDRAN; Prashanth, BRADLEY; Derek Edward, URNAU GOTARDO; Paulo Fabiano, ZOSS; Gaspard
CPC: G06N3/045; G06N3/0455; G06N3/0464; G06N3/047; G06N3/0475; G06N3/063; G06N3/08; G06N3/084; G06N3/09; G06N3/094; G06T11/10; G06T3/4046; G06V10/82
Verdict: Set aside generic neural-network architecture research
Source: Google Patents · FreePatentsOnline

Abstract

A technique for performing style transfer between a content sample and a style sample is disclosed. The technique includes applying one or more neural network layers to a first latent representation of the style sample to generate one or more convolutional kernels. The technique also includes generating convolutional output by convolving a second latent representation of the content sample with the one or more convolutional kernels. The technique further includes applying one or more decoder layers to the convolutional output to produce a style transfer result that comprises one or more content-based attributes of the content sample and one or more style-based attributes of the style sample.

Background

BACKGROUND Field of the Various Embodiments

Embodiments of the present disclosure relate generally to convolutional neural networks, and more specifically, to adaptive convolutions in neural networks. DESCRIPTION OF THE RELATED ART

Style transfer refers to a technique for transferring the “style” of a first image onto a second image without modifying the content of the second image. For example, colors, patterns, and/or other style-based attributes of the first image may be transferred onto one or more faces, buildings, bridges, and/or other objects in the second image without removing the objects from the second image or adding new objects to the second image.

Existing style transfer methods typically use convolutional neural networks to learn or characterize the “global” statistics of the style image and transfer the statistics to the content image. For example, an encoder network may be used to generate feature maps for both the content and style images. A mean and standard deviation may be calculated for one or more portions of the feature map for the style image, and the corresponding portion(s) of the feature map for the content image may be normalized to have the same mean and standard deviation. A decoder network may then be used to convert the normalized feature map into an output image that combines the style of the style image with the content of the content image.

On the other hand, existing style transfer techniques are unable to ident

Claims

1. A method for performing style transfer between a content sample and a style sample, comprising: applying one or more neural network layers to a first latent representation of the style sample to generate one or more convolutional kernels; generating convolutional output by convolving a second latent representation of the content sample with the one or more convolutional kernels; and applying one or more decoder layers to the convolutional output to produce a style transfer result that comprises one or more content-based attributes of the content sample and one or more style-based attributes of the style sample. || 12. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform the steps of: applying one or more neural network layers to a first latent representation of a style sample to generate one or more convolutional kernels; generating convolutional output by convolving a second latent representation of a content sample with the one or more convolutional kernels; and applying one or more decoder layers to the convolutional output to produce a style transfer result that comprises one or more content-based attributes of the content sample and one or more style-based attributes of the style sample. || 20. A system, comprising: a memory that stores instructions, and a processor that is coupled to the memory and, when executing the instructions, is configured to: apply an encoder network to a style image and a content image to generate a first latent representation of the style image and a second latent representation of the content image; apply one or more neural network layers to the first latent representation to generate one or more convolutional kernels; generate convolutional output by convolving the second latent representation with the one or more convolutional kernels; and apply one or more decoder layers to the convolutional output to produce a style transfer result that comprises one or more content-based attributes of the content sample and one or more style-based attributes of the style sample. || 21. A method for performing convolutions within a neural network, comprising: applying one or more neural network layers to a first input to generate one or more convolutional kernels; generating convolutional output by convolving a second input with the one or more convolutional kernels; and applying one or more decoder layers to the convolutional output to produce a decoding result, wherein the decoding result comprises one or more first attributes of the first input and one or more second attributes of the second input.