- Number
- 11409791
- Published
- 2022-08-09
- Filed
- 2017-06-12
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Torabi; Atousa, Sigal; Leonid
- CPC
- G06F16/638; G06F16/7844; G06N3/044; G06N3/0442; G06N3/045; G06N3/0464; G06N3/08; G06N3/084; G06N3/09; G06V10/82; G06V20/41; H04N21/8405
- Verdict
- Set aside video tagging/search ML, business
- Source
- Google Patents · FreePatentsOnline
Abstract
Systems, methods and articles of manufacture for modeling a joint language-visual space. A textual query to be evaluated relative to a video library is received from a requesting entity. The video library contains a plurality of instances of video content. One or more instances of video content from the video library that correspond to the textual query are determined, by analyzing the textual query using a data model that includes a soft-attention neural network module that is jointly trained with a language Long Short-term Memory (LSTM) neural network module and a video LSTM neural network module. At least an indication of the one or more instances of video content is returned to the requesting entity.
Background
BACKGROUND Field of the Invention (1) Embodiments described herein generally relate to classifying video content, and more particularly, to techniques for tagging video content using a joint language-visual embedding space. Description of the Related Art (2) A number of different techniques exist today for delivering video content to users. Generally speaking, existing systems for delivering video content, such as over-the-air broadcasts, cable television service, Internet television service, telephone network television service, satellite television service, satellite radio service, websites, etc., provide a relatively impersonalized, generic experience to all viewers. For example, with respect to broadcast television, all viewers of a given television network station receive essentially the same content in essentially the same order. (3) Moreover, the amount of digital video content that is available for users to consume is growing at an ever-increasing pace. For example, online platforms exist today that enables users to upload original content for distribution to other users. These platforms, together with the advent of inexpensive mobile recording devices (e.g., smart phones), have led to a significant increase in the amount of digital video content available for consumption. (4) Natural language-based video and image search has been a long standing topic of research among information retrieval, multimedia, and computer vision communities. Several existing on-line platfo
Claims
1. A computer-implemented method of querying video content, the computer-implemented method comprising: receiving, from a requesting entity, a textual query to be evaluated relative to a video library, the video library containing a plurality of instances of video content; training a data model based on a plurality of training samples, wherein the data model comprises a soft-attention neural network module, a language Long Short-term Memory (LSTM) neural network module, and a video LSTM neural network module, which are jointly trained, wherein each of the plurality of training samples includes (i) a respective instance of video content and (ii) a respective plurality of phrases describing the respective instance of video content, wherein training the data model comprises, for each of the plurality of training samples: encoding each of the plurality of phrases for the training sample as a matrix, wherein each word within the plurality of phrases is encoded as a vector using a trained model for word representation, wherein the trained model is separate from the data model; determining a weighted ranking between the plurality of phrases, based on a respective length of each phrase, such that lengthier phrases are ranked above less lengthy phrases; encoding the respective instance of video content for the training sample as a sequence of frames; extracting frame features from the sequence of frames; performing an object classification analysis on the extracted frame features; and generating a matrix representing the respective instance of video content, based on the extracted frame features and the object classification analysis, the matrix including feature vectors; processing the textual query using the trained data model to identify a ranking of the plurality of instances of video content responsive to the textual query, wherein the soft-attention neural network module aligns an output of a last state of the language LSTM neural network module with feature vectors of an instance of the plurality of instances of video content, wherein the ranking is identified by generating an attention-based representation that is fed to the video LSTM neural network module, wherein the attention-based representation is generated by calculating an attention-weighted average of frames of the instance of video content based on the aligned output and minimizing a ranking loss function having a penalty function that is asymmetric; and returning at least an indication of the ranking of the plurality of instances of video content to the requesting entity. ||
6. A computer-implemented method of querying video content, the computer-implemented method comprising: receiving, from a requesting entity, a textual query to be evaluated relative to a video library, the video library containing a plurality of instances of video content; training a data model based in part on a plurality of training samples, wherein the data model comprises a soft-attention neural network module that is jointly trained with a language Long Short-term Memory (LSTM) neural network module and a video LSTM neural network module, wherein each of the plurality of training samples includes (i) a respective instance of video content and (ii) a respective plurality of phrases describing the respective instance of video content, wherein a first one of the plurality of training samples comprises a single-frame instance of video content generated from an image file, and wherein training the data model comprises, for each of the plurality of training samples: encoding each of the plurality of phrases for the training sample as a matrix, wherein each word within the plurality of phrases is encoded as a vector using a trained model for word representation, wherein the trained model is separate from the data model; determining a weighted ranking between the plurality of phrases, based on a respective length of each phrase, such that lengthier phrases are ranked above less lengthy phrases; and generating a matrix representing the respective instance of video content, based at least in part on an object classification analysis performed on frame features extracted from the respective instance of video content, the matrix including feature vectors; and processing the textual query using the trained data model to identify a ranking of the plurality of instances of video content responsive to the textual query, wherein the soft-attention neural network module is aligns an output of a last state of the language LSTM neural network module with feature vectors of an instance of the plurality of instances of video content, wherein the ranking is identified by generating an attention-based representation that is fed to the video LSTM neural network module, wherein the attention-based representation is generated by calculating an attention-weighted average of video frames of the instance of video content based on the aligned output and minimizing a ranking loss function having a penalty function that is asymmetric; and returning at least an indication of the ranking of the plurality of instances of video content to the requesting entity. ||
1. ||
12. A system to query video content, the system comprising: one or more computer processors; a memory containing a program executable by the one or more computer processors to perform an operation comprising: receiving, from a requesting entity, a textual query to be evaluated relative to a video library, the video library containing a plurality of instances of video content; training a data model based on a plurality of training samples, wherein the data model comprises a soft-attention neural network module, a language Long Short-term Memory (LSTM) neural network module, and a video LSTM neural network module, which are jointly trained, wherein each of the plurality of training samples includes (i) a respective instance of video content and (ii) a respective plurality of phrases describing the respective instance of video content, wherein training the data model comprises, for each of the plurality of training samples: encoding each of the plurality of phrases for the training sample as a matrix, wherein each word within the plurality of phrases is encoded as a vector using a trained model for word representation, wherein the trained model is separate from the data model; determining a weighted ranking between the plurality of phrases, based on a respective length of each phrase, such that lengthier phrases are ranked above less lengthy phrases; encoding the respective instance of video content for the training sample as a sequence of frames; extracting frame features from the sequence of frames; performing an object classification analysis on the extracted frame features; and generating a matrix representing the respective instance of video content, based on the extracted frame features and the object classification analysis, the matrix including feature vectors; processing the textual query using the trained data model to identify a ranking of the plurality of instances of video content responsive to the textual query, wherein the soft-attention neural network module aligns an output of a last state of the language LSTM neural network module with feature vectors of an instance of the plurality of instances of video content, wherein the ranking is identified by generating an attention-based representation that is fed to the video LSTM neural network module, wherein the attention-based representation is generated by calculating an attention-weighted average of frames of the instance of video content based on the aligned output and minimizing a ranking loss function having a penalty function that is asymmetric; and returning at least an indication of the ranking of the plurality of instances of video content to the requesting entity.