Visual reference resolution using attention memory for visual dialog

Number: 10726206
Published: 2020-07-28
Filed: 2018-01-30
Assignee: Disney Enterprises, Inc.
Inventors: Sigal; Leonid, Lehrmann; Andreas, Seo; Paul Hongsuck
CPC: G06N5/041; G06N3/044; G06F40/216; G06N3/04; G06N3/0464; G06F40/30; G06N3/08; G06N3/0442; G06N3/09; G06N3/045
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

ML visual-dialog reference resolution research.

Abstract

A dialog engine configured to answer a sequence of questions related to an image. An attention module residing in the dialog engine includes an attention retrieval process and a tentative attention process. The attention retrieval process retrieves a relevant attention map that may have previously been used to answer a previous query. The tentative attention process generates a tentative attention map based on an image and other input parameters. The attention module combines the relevant attention map and the tentative attention map to generate a fused attention map. Based on the fused attention map, the dialog engine generates a response to the query. Finally, the dialog engine stores the fused attention map in an attention memory for use in answering future queries.

Background

BACKGROUND OF THE INVENTIONField of the Invention(1) Embodiments of the present invention relate generally to artificial intelligence and, more specifically, to visual reference resolution using attention memory for visual dialog.Description of the Related Art(2) Artificial Neural Networks (ANNs) can be trained to perform a wide variety of tasks. One such task is referred to as “visual question answering” (VQA). Once trained to perform the VQA task, a previous ANN architecture may analyze an image and then answer simple questions related to that image. For example, a previous ANN could analyze an image of a woman wearing a blue dress. Then, when asked to provide the color of the woman's dress, the ANN would answer “blue.”(3) Previous ANNs trained to perform VQA tasks suffer from a significant drawback, however. Specifically, a previous ANN cannot resolve the meanings of semantic references included in questions. For instance, in the above example, if the previous ANN were asked the question, “what is the color of her hair,” the ANN would not be able to resolve the pronoun “her” as indicating the woman in the image. The ANN, therefore, would not be able to answer the question correctly. As a general matter, because previous ANNs cannot resolve semantic references, previous ANNs have difficulty correctly interpreting follow-up questions that refer back to previous portions of a given dialog. Not being able to resolve semantic references limits the usefulness of ANNs for perform

Claims

1. A computer-implemented method for resolving semantic references via a neural network architecture, the method comprising: generating, based on a current input, a tentative attention map indicating a first portion of an image associated with the current input, wherein the first portion of the image corresponds to a first query; retrieving, based on a set of previous inputs, a relevant attention map indicating a second portion of the image associated with the current input, wherein the second portion of the image corresponds to the set of previous inputs; fusing the tentative attention map with the relevant attention map to generate a fused attention map; and generating a first response to the first query based on the fused attention map, wherein the fused attention map is associated with at least one referent corresponding to at least one semantic reference included in the first query. 11. A non-transitory computer-readable medium including program instructions that, when executed by a processor, causes the processor to resolve semantic references via a neural network architecture by performing the steps of: generating, based on a current input, a tentative attention map indicating a first portion of an image associated with the current input, wherein the first portion of the image corresponds to a first query; retrieving, based on a set of previous inputs, a relevant attention map indicating a second portion of the image associated with the current input, wherein the second portion of the image corresponds to the set of previous inputs; fusing the tentative attention map with the relevant attention map to generate a fused attention map; and generating a first response to the first query based on the fused attention map, wherein the fused attention map is associated with at least one referent corresponding to at least one semantic reference included in the first query. 19. A system for resolving semantic references, comprising: a memory storing a dialog engine; and a processor that, upon executing the dialog engine, performs the steps of: generating, based on a current input, a tentative attention map indicating a first portion of an image associated with the current input, wherein the first portion of the image corresponds to a first query, retrieving, based on a set of previous inputs, a relevant attention map indicating a second portion of the image associated with the current input, wherein the second portion of the image corresponds to the set of previous inputs, fusing the tentative attention map with the relevant attention map to generate a fused attention map, and generating a first response to the first query based on the fused attention map, wherein the fused attention map is associated with at least one referent corresponding to at least one semantic reference included in the first query.