- Number
- 20200012725
- Published
- 2020-01-09
- Filed
- 2018-07-05
- Assignee
- Disney Enterprises, Inc.
- Inventors
- LI; Boyang, SIGAL; Leonid, DOGAN; Pelin
- CPC
- G06V30/2276; G06V20/41; G06N3/09; G06V30/19173; G06N3/044; G06F40/45; G06N5/046; G06N3/0442; G06V10/811; G06N3/048; G06F18/256; G06V10/82; G06N3/045; G06N3/0464; G06V10/454; G06F18/214
- Verdict
- Set aside video/text metadata alignment analytics, business
- Source
- Google Patents · FreePatentsOnline
Abstract
Systems, methods and computer program products related to aligning heterogeneous sequential data. A first sequential data stream and a second sequential data stream are received. An action related to aligning the first sequential data stream and the second sequential data stream is determined using an alignment neural network. The alignment neural network includes a fully connected layer that receives as input: data from the first sequential data stream, data from the second sequential data stream, and data relating to a previously determined action by the alignment neural network related to aligning the first sequential data stream and the second sequential data stream.
Background
BACKGROUNDField of the Invention
The present invention relates to computerized neural networks, and more specifically, to a neural network for aligning heterogeneous sequential data.Description of the Related Art
Alignment of sequential data is a common problem in many different fields, including molecular biology, natural language processing, historic linguistics, and computer vision, among other fields. Aligning heterogeneous sequences of data, with complex correspondences, can be particularly complex. Heterogeneity refers to the lack of a readily apparent surface matching. For example, alignment of visual and textual content can be very complex. This is particularly true where one-to-many and one-to-none correspondences are possible, as in alignment of video from a film or television show with a script relating to the film or television show. One or more embodiments herein describe use of a computerized neural network to align sequential heterogeneous data, for example visual and textual data.SUMMARY
Embodiments described herein include a method for aligning heterogeneous sequential data. The method includes receiving a first sequential data stream and a second sequential data stream. The method further includes determining an action related to aligning the first sequential data stream and the second sequential data stream using an alignment neural network. The alignment neural network includes a fully connected layer that receives as input: data from the
Claims
1. A method of aligning heterogeneous sequential data, comprising: receiving a first sequential data stream comprising a first plurality of segments and a second sequential data stream comprising a second plurality of segments; determining a first alignment action related to aligning the first plurality of segments in the first sequential data stream with the second plurality of segments in the second sequential data stream using an alignment neural network, the alignment neural network comprising: a fully connected layer that receives as input: data from the first sequential data stream, data from the second sequential data stream, and data, retrieved from storage, relating to a plurality of previously determined alignment actions by the alignment neural network related to aligning the first sequential data stream and the second sequential data stream, wherein the alignment neural network is configured to determine the first alignment action based, at least in part, on the data related to the plurality of previously determined alignment actions; and aligning a first segment in the first plurality of segments with a second segment in the second plurality of segments by performing the determined first alignment action.
10. A computer program product for aligning heterogeneous sequential data, the computer program product comprising: a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code comprising computer-readable program code configured to perform an operation, the operation comprising: receiving a first sequential data stream comprising a first plurality of segments and a second sequential data stream comprising a second plurality of segments; determining a first alignment action related to aligning the first plurality of segments in the first sequential data stream with the second plurality of segments in the second sequential data stream using an alignment neural network, the alignment neural network comprising: a fully connected layer that receives as input: data from the first sequential data stream, data from the second sequential data stream, and data, retrieved from storage, relating to a plurality of previously determined alignment actions by the alignment neural network related to aligning the first sequential data stream and the second sequential data stream, wherein the alignment neural network is configured to determine the first alignment action based, at least in part, on the data related to the plurality of previously determined alignment actions; and aligning a first segment in the first plurality of segments with a second segment in the second plurality of segments by performing the determined first alignment action.
16. A system, comprising: a processor; and a memory containing a program that, when executed on the processor, performs an operation, the operation comprising: receiving a first sequential data stream comprising a first plurality of segments and a second sequential data stream comprising a second plurality of segments; determining a first alignment action related to aligning the first plurality of segments in the first sequential data stream with the second plurality of segments in the second sequential data stream using an alignment neural network, the alignment neural network comprising: a fully connected layer that receives as input: data from the first sequential data stream, data from the second sequential data stream, and data, retrieved from storage, relating to a plurality of previously determined alignment actions by the alignment neural network related to aligning the first sequential data stream and the second sequential data stream, wherein the alignment neural network is configured to determine a first alignment action based, at least in part, on the data related to the plurality of previously determined alignment actions; and aligning a first segment in the first plurality of segments with a second segment in the second plurality of segments by performing the determined first alignment action.