VISUAL REFERENCE RESOLUTION USING ATTENTION MEMORY FOR VISUAL DIALOG

Number: 20190236136
Published: 2019-08-01
Filed: 2018-01-30
Assignee: Disney Enterprises, Inc.
Inventors: SIGAL; Leonid et al.
CPC: G06N3/08; G06N3/044; G06N3/0442; G06F40/216; G06N3/0464; G06F40/30; G06N3/045; G06N3/09; G06N5/041; G06N3/04
Verdict: Set aside visual reference resolution attention memory visual dialog, generic ML research
Source: Google Patents · FreePatentsOnline

Abstract

A dialog engine configured to answer a sequence of questions related to an image. An attention module residing in the dialog engine includes an attention retrieval process and a tentative attention process. The attention retrieval process retrieves a relevant attention map that may have previously been used to answer a previous query. The tentative attention process generates a tentative attention map based on an image and other input parameters. The attention module combines the relevant attention map and the tentative attention map to generate a fused attention map. Based on the fused attention map, the dialog engine generates a response to the query. Finally, the dialog engine stores the fused attention map in an attention memory for use in answering future queries.

Background

BACKGROUND OF THE INVENTIONField of the Invention

Embodiments of the present invention relate generally to artificial intelligence and, more specifically, to visual reference resolution using attention memory for visual dialog.Description of the Related Art

Artificial Neural Networks (ANNs) can be trained to perform a wide variety of tasks. One such task is referred to as “visual question answering” (VQA). Once trained to perform the VQA task, a previous ANN architecture may analyze an image and then answer simple questions related to that image. For example, a previous ANN could analyze an image of a woman wearing a blue dress. Then, when asked to provide the color of the woman's dress, the ANN would answer “blue.”

Previous ANNs trained to perform VQA tasks suffer from a significant drawback, however. Specifically, a previous ANN cannot resolve the meanings of semantic references included in questions. For instance, in the above example, if the previous ANN were asked the question, “what is the color of her hair,” the ANN would not be able to resolve the pronoun “her” as indicating the woman in the image. The ANN, therefore, would not be able to answer the question correctly. As a general matter, because previous ANNs cannot resolve semantic references, previous ANNs have difficulty correctly interpreting follow-up questions that refer back to previous portions of a given dialog. Not being able to resolve semantic references limits the usefulness of ANNs fo

Claims

1. A computer-implemented method for resolving semantic references via a neural network architecture, the method comprising: generating, based on a current input, a tentative attention map indicating a first portion of the current input associated with a first query; retrieving, based on a set of previous inputs, a relevant attention map indicating a second portion of the current input associated with the set of previous inputs; fusing the tentative attention map with the relevant attention map to generate a fused attention map; and generating a first response to the first query based on the fused attention map, wherein the fused attention map is associated with at least one referent corresponding to at least one semantic reference included in the first query. | 11. A non-transitory computer-readable medium including program instructions that, when executed by a processor, causes the processor to resolve semantic references via a neural network architecture by performing the steps of: generating, based on a current input, a tentative attention map indicating a first portion of the current input associated with a first query; retrieving, based on a set of previous inputs, a relevant attention map indicating a second portion of the current input associated with the set of previous inputs; fusing the tentative attention map with the relevant attention map to generate a fused attention map; and generating a first response to the first query based on the fused attention map, wherein the fused attention map is associated with at least one referent corresponding to at least one semantic reference included in the query. | 19. A system for resolving semantic references, comprising: a memory storing a dialog engine; and a processor that, upon executing the dialog engine, performs the steps of: generating, based on a current input, a tentative attention map indicating a first portion of the current input associated with a first query, retrieving, based on a set of previous inputs, a relevant attention map indicating a second portion of the current input associated with the set of previous inputs, fusing the tentative attention map with the relevant attention map to generate a fused attention map, and generating a first response to the first query based on the fused attention map, wherein the fused attention map is associated with at least one referent corresponding to at least one semantic reference included in the query.