CONTROLLABLE 3D STYLE TRANSFER FOR RADIANCE FIELDS

Number: 20240428540
Published: 2024-12-26
Filed: 2024-06-21
Assignee: DISNEY ENTERPRISES, INC.
Inventors: Fernandez Labrador; Clara Maria et al.
CPC: G06T19/20; G06V10/44; G06V20/70
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Controllable 3D style-transfer technique for radiance-field (NeRF) rendering (related).

Abstract

The present invention sets forth a technique for performing style transfer. The technique includes converting a style sample into a first set of semantic features and a first set of visual features and determining a set of content samples corresponding to a plurality of views of a three-dimensional (3D) scene. The technique also includes, for each content sample included in the set of content samples, converting the content sample into an additional set of semantic features and an additional set of visual features and determining a set of matches between (i) the additional set of semantic features and the additional set of visual features and (ii) the first set of semantic features and the first set of visual features. The technique further includes generating a style transfer result, wherein the style transfer result comprises structural elements of the 3D scene and stylistic elements of the style sample.

Background

BACKGROUND Field of the Various Embodiments

Embodiments of the present disclosure relate generally to machine learning and image processing and, more specifically, to techniques for transferring styles in three-dimensional (3D) scenes. Description of the Related Art

Style transfer is a technique for generating stylized output by combining one or more structural elements included in one or more content samples with stylistic elements included in one or more style samples. Structural elements may include features such as objects, lines, edges, outlines, or surfaces. Stylistic elements may include one or more of colors, textures, patterns, or lighting characteristics included in the style samples. Style transfer is applicable to two-dimensional (2D) content samples, such as still images, or to 3D representations of the contents of a scene, such as neural radiance fields (NeRFs).

Existing techniques for performing style transfer in 3D representations of scenes are typically limited to transferring a style from a single style sample to the entirety of a content sample. Consequently, these techniques tend to lack fine-grained controllability, such as the ability to transfer a style to a specified element or object included in the content sample and/or transfer different styles to different regions within the content sample.

Other existing techniques may operate on 3D inputs, such as 3D point clouds or 3D mesh representations of content and/or style sampl

Claims

1. A computer-implemented method for performing style transfer, the method comprising: converting a style sample into a first set of semantic features and a first set of visual features; determining a set of content samples corresponding to a plurality of views of a three-dimensional (3D) scene; for each content sample included in the set of content samples: converting the content sample into an additional set of semantic features and an additional set of visual features; and determining a set of matches between (i) the additional set of semantic features and the additional set of visual features and (ii) the first set of semantic features and the first set of visual features; and generating a style transfer result that includes a representation of the 3D scene based on one or more losses associated with the sets of matches determined for the set of content samples, wherein the style transfer result comprises one or more structural elements of the 3D scene and one or more stylistic elements of the style sample. || 11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: converting a style sample into a first set of semantic features and a first set of visual features; determining a set of content samples corresponding to a plurality of views of a three-dimensional (3D) scene; for each content sample included in the set of content samples: converting the content sample into an additional set of semantic features and an additional set of visual features; and determining a set of matches between (i) the additional set of semantic features and the additional set of visual features and (ii) the first set of semantic features and the first set of visual features; and generating a style transfer result that includes a representation of the 3D scene based on one or more losses associated with the sets of matches determined for the set of content samples, wherein the style transfer result comprises one or more structural elements of the 3D scene and one or more stylistic elements of the style sample. || 18. A system comprising: one or more memories storing instructions; and one or more processors for executing the instructions to: convert a style sample into a first set of semantic features and a first set of visual features; determine a set of content samples corresponding to a plurality of views of a three-dimensional (3D) scene; for each content sample included in the set of content samples: convert the content sample into an additional set of semantic features and an additional set of visual features; and determine a set of matches between (i) the additional set of semantic features and the additional set of visual features and (ii) the first set of semantic features and the first set of visual features; and generate a style transfer result that includes a representation of the 3D scene based on one or more losses associated with the sets of matches determined for the set of content samples, wherein the style transfer result comprises one or more structural elements of the 3D scene and one or more stylistic elements of the style sample.