TECHNIQUES FOR PROCESSING VIDEOS USING TEMPORALLY-CONSISTENT TRANSFORMER MODEL

Number: 20230281757
Published: 2023-09-07
Filed: 2022-07-28
Assignee: DISNEY ENTERPRISES, INC.
Inventors: Zhang; Yang et al.
CPC: G06T5/70; G06T3/18; G06T3/4007; G06T5/10; G06T5/60
Verdict: Set aside generic video processing ML
Source: Google Patents · FreePatentsOnline

Abstract

Techniques are disclosed for enhancing videos using a machine learning model that is a temporally-consistent transformer model. The machine learning model processes blocks of frames of a video in which the temporally first input video frame of each block of frames is a temporally second to last output video frame of a previous block of frames. After the machine learning model is trained, blocks of video frames, or features extracted from the video frames, can be warped using an optical flow technique and transformed using a wavelet transform technique. The transformed video frames are concatenated along a channel dimension and input into the machine learning model that generates corresponding processed video frames.

Background

BACKGROUND Technical Field

Embodiments of the present disclosure relate generally to computer science and video processing and, more specifically, to techniques for processing videos using a temporally-consistent transformer model. Description of the Related Art

The frames of a video can include various degradations, such as noise and blurriness. For example, a video captured by a camera can contain noise due to the mechanics of camera sensors. As another example, a video and/or frames thereof can be blurry when the video and/or frames are captured by a camera that is out of focus.

Transformer models are artificial neural networks. Conventional transformer models have been applied to some computer vision tasks, such as detecting objects within images and classifying those objects. However, when transformer models are used to process the frames of a video, the processed frames can have temporally inconsistent regions that vary from frame to frame, even when no such variations should occur. The temporally inconsistent regions are, as a general matter, readily noticeable as flickering and other artifacts that can greatly reduce the quality of processed videos.

As the foregoing illustrates, what is needed in the art are more effective techniques for processing videos using transformer models. SUMMARY

One embodiment of the present disclosure sets forth a computer-implemented method for enhancing videos. The method includes processing a first plur

Claims

1. A computer-implemented method for enhancing videos, the method comprising: processing a first plurality of video frames using a machine learning model to generate a first plurality of processed video frames; and processing a second plurality of video frames using the machine learning model to generate a second plurality of processed video frames, wherein a temporally first video frame included in the second plurality of video frames is a temporally second to last video frame included in the first plurality of processed video frames. || 10. One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processing units, cause the one or more processing units to perform steps for enhancing videos, the steps comprising: processing a first plurality of video frames using a machine learning model to generate a first plurality of processed video frames; and processing a second plurality of video frames using the machine learning model to generate a second plurality of processed video frames, wherein a temporally first video frame included in the second plurality of video frames is a temporally second to last video frame included in the first plurality of processed video frames. || 19. A computer-implemented method for training a machine learning model, the method comprising: adding a plurality of amounts of degradation to a set of video frames to generate a plurality of sets of degraded video frames, wherein each set of degraded video frames includes a different amount of degradation; and performing one or more operations to train the machine learning model based on the plurality of sets of degraded video frames, wherein the one or more operations minimize a loss function that penalizes a difference between a temporally last frame of each plurality of processed video frames generated by the machine learning model and a temporally first frame of a subsequent plurality of processed video frames generated by the machine learning model.