Techniques for denoising videos

Number: 12664618
Published: 2026-06-23
Filed: 2022-04-14
Assignee: DISNEY ENTERPRISES, INC.
Inventors: Zhang; Yang, Aydin; Tunc Ozan, Schroers; Christopher Richard
CPC: G06N3/084; G06T5/70; G06N3/0455; G06N3/096; G06N20/00; G06T5/60; G06T5/73; G06N3/0464; G06T2207/20081
Verdict: Set aside dropped in weekly review
Source: Google Patents · FreePatentsOnline

The keeper's note

Techniques are disclosed for denoising videos.

Abstract

Techniques are disclosed for denoising videos. In some embodiments, video frames are denoised using a denoising model that includes an encoder-decoder architecture and attention modules. During training of the denoising model, the attention modules learn weightings to upweight certain dimensions of input features to help pixel registration, remove ghosting artifacts, and improve temporal consistency when the frames of a video are being denoised. The denoising model can also be used to train a student denoising model that has a same architecture as, but is smaller and faster than, the denoising model. After training, noisy video frames can be input into the denoising model and/or the student denoising model to generate corresponding denoised video frames.

Background

BACKGROUND Technical Field (1) Embodiments of the present disclosure relate generally to computer science and video processing and, more specifically, to techniques for denoising videos. Description of the Related Art (2) Image noise refers to random variation in the color and/or brightness within images. Examples of images include standalone images and the frames of a video. Images captured by cameras can contain noise due to the mechanics of camera sensors. For example, the quality of images captured in low-lighting conditions can be adversely affected by noise. Noise can also be a byproduct in images that are rendered using Monte Carlo rendering techniques. (3) Image denoising is the process of removing noise from an image. Conventional approaches for image denoising can introduce unwanted artifacts into denoised images. For example, when the frames of a video are denoised according to conventional approaches, the denoised frames can have temporally inconsistent denoised regions that vary from frame to frame, even when no such variations should occur. As another example, noise and texture can “drag” from one denoised frame to another denoised frame, producing a ghost-like effect. Such artifacts that are introduced during the denoising process greatly reduce the quality of the generated denoised images. (4) In addition, conventional approaches for image denoising require a user to have extensive knowledge of the noise characteristics within each noisy image in order to manu

Claims

1. A computer-implemented method for denoising videos, the method comprising: receiving a video that includes a plurality of frames; and processing one or more frames included in the plurality of frames using a denoising model that includes an encoder and a decoder to generate one or more denoised frames, wherein the encoder includes one or more encoder blocks and the decoder includes one or more decoder blocks, and each of the one or more encoder blocks and the one or more decoder blocks includes a three-dimensional (3D) convolutional layer and an attention module that calculates (i) a first set of weights for a plurality of feature channels associated with a plurality of features and (ii) a second set of weights for a spatial dimension associated with the plurality of features. || 10. One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processing units, cause the one or more processing units to perform steps for denoising videos, the steps comprising: receiving a video that includes a plurality of frames; and processing one or more frames included in the plurality of frames using a denoising model that includes an encoder and a decoder to generate one or more denoised frames, wherein the encoder includes one or more encoder blocks and the decoder includes one or more decoder blocks, and each of the one or more encoder blocks and the one or more decoder blocks includes a three-dimensional (3D) convolutional layer and an attention module that calculates (i) a first set of weights for a plurality of feature channels associated with a plurality of features and (ii) a second set of weights for a spatial dimension associated with the plurality of features. || 19. A system, comprising: one or more memories storing instructions; and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to: receive a video that includes a plurality of frames, and process one or more frames included in the plurality of frames using a denoising model that includes an encoder and a decoder to generate one or more denoised frames, wherein the encoder includes one or more encoder blocks and the decoder includes one or more decoder blocks, and each of the one or more encoder blocks and the one or more decoder blocks includes a three-dimensional (3D) convolutional layer and an attention module that calculates (i) a first set of weights for a plurality of feature channels associated with a plurality of features and (ii) a second set of weights for a spatial dimension associated with the plurality of features.