Efficient Neural Style Transfer For Fluid Simulations

Neural style transfer applied to fluid-simulation VFX rendering.

A system includes a hardware processor, and a system memory storing a software code and a machine learning (ML) model trained to apply a stylization to an image. The hardware processor executes the software code to receive a first sequence of images and style data describing a desired stylization of content depicted by the first sequence of images. The hardware processor further executes the software code to stylize the content, using the ML model, to provide a stylized content having the desired stylization, wherein stylizing includes applying an exponential moving average (EMA) temporal smoothing algorithm to sequential image pairs of the first sequence of images to generate a second sequence of images providing a depiction of the content having the desired stylization, and output the stylized content having the desired stylization.

BACKGROUND

Artistically controlling fluids is a challenging task. One approach to addressing this challenge is to use volumetric neural style transfer techniques to manipulate fluid simulation data. However, applying volumetric style transfer algorithms directly to production in their original formulation is impracticable, and several changes are needed to adapt the approach to production pipelines. Moreover, the energy minimization solved by conventional methods is camera dependent (hereinafter “view-dependent”). To avoid that view dependency, a computationally expensive iterative optimization must typically be performed for multiple views sampled around the original simulation, which can undesirably take up to several minutes per frame. Thus, there is a need in the art for a fluid simulation solution enabling stylizations that are significantly faster, simpler, more controllable, and less prone to artifacts than conventional approaches.