- Number
- 20250284938
- Published
- 2025-09-11
- Filed
- 2025-03-06
- Assignee
- DISNEY ENTERPRISES, INC.
- Inventors
- AYDIN; Tunc Ozan et al.
- CPC
- G06N3/045; G06N3/08; G06N3/048; G06N3/0455
- Verdict
- Set aside generic ML architecture research
- Source
- Google Patents · FreePatentsOnline
Abstract
The present invention sets forth techniques for generating attention values via a modulated softmax attention mechanism. The techniques include calculating key, query, and value matrices associated with an input matrix including one or more input tokens, calculating a first vector including one or more per-token scaling values and a second vector including one or more per-token bias values. The techniques also include generating an attention prior matrix based at least on the first and second vectors, and calculating, for each of the one or more input tokens, a modulated attention score associated with the input token. The techniques further include calculating a matrix including one or more modulated attention values associated with the one or more input tokens, and transmitting the one or more modulated attention values to at least one stage included in a transformer network.
Background
BACKGROUND Field of the Various Embodiments
Embodiments of the present disclosure relate generally to deep neural networks and, more specifically, to techniques for improving attention mechanisms in deep neural networks, including all aspects of the related hardware, software, graphical user interfaces, and algorithms associated with implementing the contemplated systems, techniques, functions, and operations set forth herein, Description of the Related Art
In the field of machine learning, attention mechanisms include techniques used in deep learning models that allow the model to selectively focus on specific areas of the input data when performing inference operations. An attention mechanism is a function that maps a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key. The key-value attention mechanism separates the source-side content vector into two types of memory known as the key and the value. The key is used for calculating the attention distribution, and the value is used for encoding the context representation. Attention mechanisms are often incorporated into transformer networks, which are widely implemented in language modeling tasks. Transformer networks also achieve state-of-the-art performance in vision tasks, such as
Claims
1. A computer-implemented method for generating modulated attention values, the computer-implemented method comprising: calculating key, query, and value matrices associated with an input matrix including one or more input tokens, based on one or more learned linear transformation matrices; calculating a first vector including one or more per-token scaling values and a second vector including one or more per-token bias values, based at least on the input matrix and a learned weight matrix; generating an attention prior matrix based at least on the first and second vectors; calculating, for each of the one or more input tokens, a modulated attention score associated with the input token, based on at least on the attention prior matrix, the key matrix, and the query matrix; calculating a matrix including one or more modulated attention values associated with the one or more input tokens, based at least on the value matrix and the modulated attention scores associated with the input tokens; and transmitting the one or more modulated attention values to at least one stage included in a transformer network. ||
8. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: calculating key, query, and value matrices associated with an input matrix including one or more input tokens, based on one or more learned linear transformation matrices; calculating a first vector including one or more per-token scaling values and a second vector including one or more per-token bias values, based at least on the input matrix and a learned weight matrix; generating an attention prior matrix based at least on the first and second vectors; calculating, for each of the one or more input tokens, a modulated attention score associated with the input token, based on at least on the attention prior matrix, the key matrix, and the query matrix; calculating a matrix including one or more modulated attention values associated with the one or more input tokens, based at least on the value matrix and the modulated attention scores associated with the input tokens; and transmitting the one or more modulated attention values to at least one stage included in a transformer network. ||
15. A system comprising: one or more memories storing instructions; and one or more processors for executing the instructions to: calculate key, query, and value matrices associated with an input matrix including one or more input tokens, based on one or more learned linear transformation matrices; calculate a first vector including one or more per-token scaling values and a second vector including one or more per-token bias values, based at least on the input matrix and a learned weight matrix; generate an attention prior matrix based at least on the first and second vectors; calculate, for each of the one or more input tokens, a modulated attention score associated with the input token, based on at least on the attention prior matrix, the key matrix, and the query matrix; calculate a matrix including one or more modulated attention values associated with the one or more input tokens, based at least on the value matrix and the modulated attention scores associated with the input tokens; and transmit the one or more modulated attention values to at least one stage included in a transformer network.