- Number
- 20230153952
- Published
- 2023-05-18
- Filed
- 2022-02-11
- Assignee
- DISNEY ENTERPRISES, INC.
- Inventors
- Djelouah; Abdelaziz et al.
- CPC
- G06T5/70; G06T3/4053; G06T5/60; G06T5/73
- Verdict
- Set aside generic video remastering
- Source
- Google Patents · FreePatentsOnline
Abstract
Restoration methods and systems are disclosed for video remastering. Techniques disclosed include receiving a video sequence. For each frame of the video sequence, techniques disclosed include encoding, by a degradation encoder, a video content associated with the frame into a latent vector. The latent vector is a representation of the degradation present in the video content; the degradation present in the video content includes one or more degradation types. Based on the latent vector and the video content, techniques disclosed further include generating, by a backbone network, one or more feature maps, and, then, restoring the frame based on the one or more feature maps.
Background
BACKGROUND
Streaming services require expansive catalogs to be competitive. Old legacy films can enrich and supplement the content of such catalogs. However, the video content of legacy films is typically degraded – that is, video content, captured by low-resolution cameras, based on old sensor technologies, may be blurry, noisy, and scratched. To meet current expectations of quality and current streaming and display technologies, remastering (restoration) of these legacy films is required.
Current restoration techniques, based on deep learning technologies, provide tools that separately tackle video denoising or video upscaling. Such specialized tools can be applied sequentially to denoise and, then, to upscale a video into higher resolution, for example. However, applying independently optimized restoration tools, in a cascading manner, may lead to sub-optimal performance in terms of restoration quality and computational complexity. Thus, techniques that restore video content by jointly addressing different types of degradations are needed.
Claims
1. A method for video remastering by a restoration system, comprising: receiving, by the system, a video sequence; and for each frame of the video sequence, encoding, by a degradation encoder, a video content associated with the frame into a latent vector, the latent vector is a representation of a degradation present in the video content, the degradation includes one or more degradation types, generating, by a backbone network, based on the latent vector and the video content, one or more feature maps, and restoring, based on the one or more feature maps, the frame. ||
12. A restoration system for video remastering, comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the processor to: receive, by the system, a video sequence, and for each frame of the video sequence, encode, by a degradation encoder, a video content associated with the frame into a latent vector, the latent vector is a representation of a degradation present in the video content, the degradation includes one or more degradation types, generate, by a backbone network, based on the latent vector and the video content, one or more feature maps, and restore, based on the one or more feature maps, the frame. ||
20. A non-transitory computer-readable medium comprising instructions executable by at least one processor to perform a method for video remastering by a restoration system, the method comprising: receiving, by the system, a video sequence; and for each frame of the video sequence, encoding, by a degradation encoder, a video content associated with the frame into a latent vector, the latent vector is a representation of a degradation present in the video content, the degradation includes one or more degradation types, generating, by a backbone network, based on the latent vector and the video content, one or more feature maps, and restoring, based on the one or more feature maps, the frame. ||
21. A method for training a degradation encoder of a restoration system for video remastering, comprising: extracting samples of video content from a training video set; degrading the samples of video content according to respective pairs of samples of degradation parameters, extracted from a training degradation set, resulting in pairs of degraded samples of video content; and training the degradation encoder based on the pairs of degraded samples of video content, wherein the degradation encoder is trained to receive video content and to output a latent vector that represents a degradation present in the received video content. ||
23. A method for training a degradation parameter encoder of a restoration system for video remastering, comprising: extracting samples of video content from a training video set; degrading the samples of video content according to respective samples of degradation parameters, extracted from a training degradation set; encoding, by a degradation encoder, the degraded samples of video content, resulting in respective latent vectors, and training the degradation parameter encoder based on the latent vectors and the respective samples of degradation parameters, wherein the degradation parameter encoder is trained to estimate a degradation parameter based on a latent vector. ||
24. A method for training a mutator of a restoration system for video remastering, comprising: extracting samples of video content from a training video set; degrading the samples of video content according to respective pairs of samples of degradation parameters, extracted from a training degradation set, resulting in pairs of degraded samples of video content; encoding, by a degradation encoder, the pairs of degraded samples of video content, resulting in respective pairs of latent vectors, and training the mutator based on the pairs of latent vectors and the respective pairs of degradation parameters, wherein the mutator is trained to receive a first latent vector that corresponds to a first degradation parameter and a second degradation parameter, and to output a second latent vector that corresponds to the second degradation parameter.