- Number
- 11068658
- Published
- 2021-07-20
- Filed
- 2017-12-01
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Mandt; Stephan Marcel, Bamler; Robert
- CPC
- G06F40/284; G06F40/30; G06N20/00; G06N3/0499; G06N5/04; G06N7/01
- Verdict
- Set aside NLP word-embedding research, generic
- Source
- Google Patents · FreePatentsOnline
Abstract
Systems, methods, and articles of manufacture to perform an operation comprising deriving, based on a corpus of electronic text, a machine learning data model that associates words with corresponding usage contexts over a window of time, according to a diffusion process, wherein the machine learning data model comprises a plurality of skip-gram models, wherein each skip-gram model comprises a word embedding vector and a context embedding vector for a respective time step associated with the respective skip-gram model, generating a smoothed model by applying a variational inference operation over the machine learning data model, and identifying, based on the smoothed model and the corpus of electronic text, a change in a semantic use of a word over at least a portion of the window of time.
Background
BACKGROUND Field of the Invention (1) The present disclosure relates to word embeddings. More specifically, the present disclosure relates to machine learning techniques for determining dynamic word embeddings. Description of the Related Art (2) Word embeddings model the distribution of words based on their surrounding words, based on a mathematical embedding from a space with one dimension per word to a continuous vector space with lower dimensions. Geometric distances between word vectors in the vector space reflect the degree of semantic similarity between words, while difference vectors encode semantic and syntactic relations between words. Conventionally, word embeddings have been formulated as static models. These static models assume that the meaning of any given word is the same across the entire text corpus, regardless of the time any given element of text was written. However, language evolves over time, and words can change their meaning (e.g., due to cultural shifts, technological innovations, and/or other events). Therefore, conventional static word embeddings have been unable to detect shifts in the meaning and use of words over time. SUMMARY (3) In one embodiment, a method comprises deriving, based on a corpus of electronic text, a machine learning data model that associates words with corresponding usage contexts over a window of time, according to a diffusion process, wherein the machine learning data model comprises a plurality of skip-gram models, wherein e
Claims
1. A method, comprising: deriving, based on a corpus of electronic text, a machine learning data model that associates words with corresponding usage contexts over a window of time, wherein the machine learning data model comprises a plurality of skip-gram models, wherein each skip-gram model comprises a word embedding vector and a context embedding vector for a respective time step associated with the respective skip-gram model, wherein deriving the machine learning data model comprises applying a diffusion process to the word embedding vectors and the context embedding vectors of the plurality of skip-gram models such that the word embedding vectors and the context embedding vectors are aligned to a common frame of reference of time; generating a smoothed model by applying a variational inference operation; and identifying, based on the smoothed model and the corpus of electronic text, a change in a semantic use of a word over at least a portion of the window of time. ||
8. A non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable to perform an operation comprising: deriving, based on a corpus of electronic text, a machine learning data model that associates words with corresponding usage contexts over a window of time, wherein the machine learning data model comprises a plurality of skip-gram models, wherein each skip-gram model comprises a word embedding vector and a context embedding vector for a respective time step associated with the respective skip-gram model, wherein deriving the machine learning data model comprises applying a diffusion process to the word embedding vectors and the context embedding vectors of the plurality of skip-gram models such that the word embedding vectors and the context embedding vectors are aligned to a common frame of reference of time; generating a smoothed model by applying a variational inference operation over the machine learning data model; and identifying, based on the smoothed model and the corpus of electronic text, a change in a semantic use of a word over at least a portion of the window of time. ||
15. A system, comprising: a computer processor; and a memory containing a program which when executed by the computer processor performs an operation comprising: deriving, based on a corpus of electronic text, a machine learning data model that associates words with corresponding usage contexts over a window of time, wherein the machine learning data model comprises a plurality of skip-gram models, wherein each skip-gram model comprises a word embedding vector and a context embedding vector for a respective time step associated with the respective skip-gram model, wherein deriving the machine learning data model comprises applying a diffusion process to the word embedding vectors and the context embedding vectors of the plurality of skip-gram models such that the word embedding vectors and the context embedding vectors are aligned to a common frame of reference of time; generating a smoothed model by applying a variational inference operation over the machine learning data model; and identifying, based on the smoothed model and the corpus of electronic text, a change in a semantic use of a word over at least a portion of the window of time.