- Number
- 12014143
- Published
- 2024-06-18
- Filed
- 2019-02-25
- Assignee
- DISNEY ENTERPRISES, INC.
- Inventors
- Dogan; Pelin et al.
- CPC
- G06F40/169; G06F40/30; G06N3/044; G06N3/0442; G06N3/045; G06N3/0464; G06N3/08; G06N3/09
- Verdict
- Set aside generic NLP phrase-grounding research
- Source
- Google Patents · FreePatentsOnline
Abstract
In various embodiments, a phrase grounding model automatically performs phrase grounding for a source sentence and a source image. The phrase grounding model determines that a first phrase included in the source sentence matches a first region of the source image based on the first phrase and at least a second phrase included in the source sentence. The phrase grounding model then generates a matched pair that specifies the first phrase and the first region. Subsequently, one or more annotation operations are performed on the source image based on the matched pair. Advantageously, the accuracy of the phrase grounding model is increased relative to prior art solutions where the interrelationships between phrases are typically disregarded.
Background
BACKGROUND Field of the Various Embodiments (1) Embodiments of the present invention relate generally to natural language and image processing and, more specifically, to techniques for performing contextual phrase grounding. Description of the Related Art (2) Phrase grounding is the process of matching phrases included in a source sentence to corresponding regions in a source image described by the source sentence. For example, phrase grounding could match the phrases “a small kid,” “blond hair,” and “a cat” included in the source sentence “a small kid with blond hair is kissing a cat” to three different regions in an associated picture or source image. Some examples of high-level tasks that involve phrase grounding include image retrieval, image captioning, and visual question answering (i.e., generating natural language answers to natural language questions about an image). For many high-level tasks, manually performing phrase grounding is prohibitively time consuming. Consequently, phrase grounding techniques are oftentimes performed automatically. (3) In one approach to automatically performing phrase grounding, a noun-matching application breaks a source sentence into constituent noun phrases. The noun-matching application also performs object detection operations that generate bounding boxes corresponding to different objects in a source image. For each noun phrase, the noun-matching application then performs machine-learning operations that independently map each noun
Claims
1. A computer-implemented method for performing automated phrase grounding operations, the computer-implemented method comprising: extracting, from a source sentence, a first phrase and a second phrase; executing a first encoder neural network that generates, based on the first phrase, a first encoding of the first phrase and, based on the second phrase, a second encoding of the second phrase; executing one or more neural networks to convert a plurality of bounding boxes for a plurality of objects included in a source image into a plurality of box states corresponding to the plurality of bounding boxes, wherein each box state included in the plurality of box states encodes one or more interrelationships between a corresponding bounding box included in the plurality of bounding boxes and one or more additional bounding boxes included in the plurality of bounding boxes; executing a decision neural network to generate a first plurality of grounding decisions based on input that includes (i) a first box state that is included in the plurality of box states and corresponds to a first bounding box included in the plurality of bounding boxes, (ii) the first encoding, and (iii) the second encoding; performing one or more comparisons using the first plurality of grounding decisions and a decision threshold to determine that a first grounding decision included in the first plurality of grounding decisions indicates that the first phrase matches the first bounding box; generating a first matched pair that specifies the first phrase and the first bounding box; and causing one or more annotation operations to be performed on the source image based on the first matched pair. ||
11. One or more non-transitory computer readable media including instructions that, when executed by one or more processors, cause the one or more processors to perform automated phrase grounding operations by performing the steps of: extracting, from a source sentence, a first phrase and a second phrase; executing a first encoder neural network that generates, based on the first phrase, a first encoding of the first phrase and, based on the second phrase, a second encoding of the second phrase; executing one or more neural networks to convert a plurality of bounding boxes for a plurality of objects included in a source image into a plurality of box states corresponding to the plurality of bounding boxes, wherein each box state included in the plurality of box states encodes one or more interrelationships between a corresponding bounding box included in the plurality of bounding boxes and one or more additional bounding boxes included in the plurality of bounding boxes; executing a decision neural network to generate a first plurality of grounding decisions based on input that includes (i) a first box state that is included in the plurality of box states and corresponds to a first bounding box included in the plurality of bounding boxes, (ii) the first encoding, and (iii) the second encoding; performing one or more comparisons using the first plurality of grounding decisions and a decision threshold to determine that a first grounding decision included in the first plurality of grounding decisions indicates that the first phrase matches the first bounding box; generating a first matched pair that specifies the first phrase and the first bounding box; and causing one or more annotation operations to be performed on the source image based on the first matched pair. ||
20. A system, comprising: one or more memories storing instructions; and one or more processors that, when executing the instructions, are configured to: extract, from a source sentence, a first phrase and a second phrase; execute a first encoder neural network that generates, based on the first phrase, a first encoding of the first phrase and, based on the second phrase, a second encoding of the second phrase; execute one or more neural networks to convert a plurality of bounding boxes for a plurality of objects included in a source image into a plurality of box states corresponding to the plurality of bounding boxes, wherein each box state included in the plurality of box states encodes one or more interrelationships between a corresponding bounding box included in the plurality of bounding boxes and one or more additional bounding boxes included in the plurality of bounding boxes; execute a decision neural network to generate a first plurality of grounding decisions based on input that includes (i) a first box state that is included in the plurality of box states and corresponds to a first bounding box included in the plurality of bounding boxes, (ii) the first encoding, and (iii) the second encoding; perform one or more comparisons using the first plurality of grounding decisions and a decision threshold to determine that a first grounding decision included in the first plurality of grounding decisions indicates that the first phrase matches the first bounding box; generate a first matched pair that specifies the first phrase and the first bounding box; and cause one or more annotation operations to be performed on the source image based on the first matched pair.