Machine Learning Model-Based Image Noise Synthesis

Number: 20240161252
Published: 2024-05-16
Filed: 2023-11-03
Assignee: Disney Enterprises, Inc.
Inventors: Zhang; Yang et al.
CPC: G06T5/50; G06T5/73
Verdict: Set aside generic image-noise synthesis
Source: Google Patents · FreePatentsOnline

Abstract

A system includes a hardware processor, a system memory storing a software code, and a machine learning (ML) model trained using style loss to predict image noise. The hardware processor is configured to execute the software code to receive a clean image and at least one noise setting of a camera used to capture a version of the clean image that includes noise, and provide the clean image and the at least one noise setting as a noise generation input to the ML model. The hardware processor is further configured to execute the software code to generate, using the ML model and based on the noise generation input, a synthesized noise map for renoising the clean image.

Background

BACKGROUND

Film grain and digital camera sensor noise are important characteristics of analog film and digital cameras. In modern digital movie production pipelines, noise often has to be removed for visual effects (VFX) compositing. Noise also has to be removed for optimized data compression. Then, in order to better portray the cinematographic aspect of a film, several post-processing operations are commonly applied to the digital content for adding the noise back to the content. A good noise modeling algorithm is necessary to retain the noise characteristics of the original noise distribution and to synthesize the desired quality the artists intend.

However, existing methods are capable of synthesizing only limited types of noise. For example, the existing method in the AV1 codec produces repetitive noise patterns and lack of randomness in color channels, which makes the synthesized noisy frame unrealistic and distracting. Furthermore, the existing methods cannot accurately model the distribution of the targeted noise source. In other words, the renoising process cannot produce the artistic intent in the video production. Also, the existing methods cannot provide artistic controls, such as specific types of camera sensor noise synthesis or noise synthesis for different camera settings (including ISO levels, shutter speed, and color temperature).

Claims

1. A system comprising: a hardware processor; a system memory storing a software code; and a machine learning (ML) model trained using style loss to predict image noise; the hardware processor configured to execute the software code to: receive a clean image and at least one noise setting of a camera used to capture a version of the clean image that includes noise; provide the clean image and the at least one noise setting as a noise generation input to the ML model; and generate, using the ML model and based on the noise generation input, a synthesized noise map for renoising the clean image. || 11. A method for use by a system including a hardware processor and a system memory storing a software code and a machine learning (ML) model trained, using style loss, to predict image noise, the method comprising: receiving, by the software code executed by the hardware processor, a clean image and at least one noise setting of a camera used to capture a version of the clean image that includes noise; providing, by the software code executed by the hardware processor, the clean image and the at least one noise setting as a noise generation input to the ML model; and generating, by the software code executed by the hardware processor, using the ML model and based on the noise generation input, a synthesized noise map for renoising the clean image.