Entropy-Constrained Neural Video Representations

Number: 20240163479
Published: 2024-05-16
Filed: 2023-11-03
Assignee: Disney Enterprises, Inc,
Inventors: Azevedo; Roberto Gerson de Albuquerque et al.
CPC: H04N19/42; H04N19/176; H04N19/13; H04N19/597; G06N3/0464; H04N19/503; G06N3/09; G06N3/0475; G06N3/045; H04N19/136
Verdict: Set aside video-representation compression research
Source: Google Patents · FreePatentsOnline

Abstract

A system includes a neural network (NN) having a matrix expansion block configured to construct a matrix representation of an input sequence, a component merging block configured to merge the matrix representation with a grid, an encoder configured to receive an output of the component merging block, a convolution stage configured to generate, using an output of the encoder, a multi-component representation of an output corresponding to the input sequence, and a convolutional upscaling stage configured to produce, using the multi-component representation of the output, an output sequence corresponding to the input sequence. A method for use by the system includes receiving an input sequence, modeling the input sequence to generate a neural network representation of the input sequence, compressing the neural network representation to generate a compressed neural network representation, and generating, from the compressed neural network representation, a compressed output sequence corresponding to the input sequence.

Background

BACKGROUND

Video compression is a long-standing and difficult problem that has inspired much research. The main goal of video compression is to represent a digital video with the minimum amount of storage requirements, while minimizing loss of quality. Although many advances have been made in the last decades in traditional video codecs, the advent of deep learning has inspired neural network-based approaches allowing new forms of video processing.

However, for the task of lossy video compression, existing neural video representation (NVR) methods typically continue to be outperformed by traditional techniques. That performance gap can be explained by the fact that current NVR methods: i) use architectures that do not efficiently obtain a compact representation of temporal and spatial input coordinates, and ii) minimize rate and distortion disjointly by first overfitting a network on a video and then using heuristic techniques such as post-training quantization or weight pruning to compress the model.

Claims

1. A system comprising: a matrix expansion block configured to construct a matrix representation of an input sequence; a component merging block configured to merge the matrix representation with a grid; an encoder configured to receive an output of the component merging block; a convolution stage configured to generate, using an output of the encoder, a multi-component representation of an output corresponding to the input sequence; and a convolutional upscaling stage configured to produce, using the multi-component representation of the output, an output sequence corresponding to the input sequence. || 11. A method for use by a system including a hardware processor and a neural network (NN), the method comprising: receiving, by the NN controlled by the hardware processor, an input sequence; modeling, by the NN controlled by the hardware processor, the input sequence to generate a neural network representation of the input sequence; compressing, by the NN controlled by the hardware processor, the neural network representation of the input sequence to generate a compressed neural network representation of the input sequence; and generating, from the compressed neural network representation, by the NN controlled by the hardware processor, a compressed output sequence corresponding to the input sequence. || 17. A method for use by a system including a hardware processor and a neural network (NN), the method comprising: receiving, by the NN controlled by the hardware processor, a frame index of a video sequence; constructing, by the NN controlled by the hardware processor, a matrix representation of the video sequence; merging, by the NN controlled by the hardware processor, the matrix representation with a fixed coordinate grid to provide a spatial-temporal data structure; generating, by the NN controlled by the hardware processor, using a first convolutional neural network (CNN) of the NN and the spatial-temporal data structure, a spatial-temporal representation of an output corresponding to the video sequence; and upscaling the spatial-temporal representation of the output, by the NN controlled by the hardware processor and using a second CNN of the NN, to produce an output sequence corresponding to the video sequence.