DYNAMIC WORD EMBEDDINGS

Number: 20180157644
Published: 2018-06-07
Filed: 2017-12-01
Assignee: Disney Enterprises, Inc.
Inventors: MANDT; Stephan Marcel et al.
CPC: G06F40/284; G06F40/30; G06N20/00; G06N3/0499; G06N5/04; G06N7/01
Verdict: Set aside generic NLP word embeddings, no creative hook
Source: Google Patents · FreePatentsOnline

Abstract

Systems, methods, and articles of manufacture to perform an operation comprising deriving, based on a corpus of electronic text, a machine learning data model that associates words with corresponding usage contexts over a window of time, according to a diffusion process, wherein the machine learning data model comprises a plurality of skip-gram models, wherein each skip-gram model comprises a word embedding vector and a context embedding vector for a respective time step associated with the respective skip-gram model, generating a smoothed model by applying a variational inference operation over the machine learning data model, and identifying, based on the smoothed model and the corpus of electronic text, a change in a semantic use of a word over at least a portion of the window of time.

Background

BACKGROUNDField of the Invention

The present disclosure relates to word embeddings. More specifically, the present disclosure relates to machine learning techniques for determining dynamic word embeddings.Description of the Related Art

Word embeddings model the distribution of words based on their surrounding words, based on a mathematical embedding from a space with one dimension per word to a continuous vector space with lower dimensions. Geometric distances between word vectors in the vector space reflect the degree of semantic similarity between words, while difference vectors encode semantic and syntactic relations between words. Conventionally, word embeddings have been formulated as static models. These static models assume that the meaning of any given word is the same across the entire text corpus, regardless of the time any given element of text was written. However, language evolves over time, and words can change their meaning (e.g., due to cultural shifts, technological innovations, and/or other events). Therefore, conventional static word embeddings have been unable to detect shifts in the meaning and use of words over time.SUMMARY

In one embodiment, a method comprises deriving, based on a corpus of electronic text, a machine learning data model that associates words with corresponding usage contexts over a window of time, according to a diffusion process, wherein the machine learning data model comprises a plurality of skip-gram models, wherei

Claims

1. A method, comprising: deriving, based on a corpus of electronic text, a machine learning data model that associates words with corresponding usage contexts over a window of time, according to a diffusion process, wherein the machine learning data model comprises a plurality of skip-gram models, wherein each skip-gram model comprises a word embedding vector and a context embedding vector for a respective time step associated with the respective skip-gram model; generating a smoothed model by applying a variational inference operation over the machine learning data model; and identifying, based on the smoothed model and the corpus of electronic text, a change in a semantic use of a word over at least a portion of the window of time. 8. A non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable to perform an operation comprising: deriving, based on a corpus of electronic text, a machine learning data model that associates words with corresponding usage contexts over a window of time, according to a diffusion process, wherein the machine learning data model comprises a plurality of skip-gram models, wherein each skip-gram model comprises a word embedding vector and a context embedding vector for a respective time step associated with the respective skip-gram model; generating a smoothed model by applying a variational inference operation over the machine learning data model; and identifying, based on the smoothed model and the corpus of electronic text, a change in a semantic use of a word over at least a portion of the window of time. 15. A system, comprising: a computer processor; and a memory containing a program which when executed by the processor performs an operation comprising: deriving, based on a corpus of electronic text, a machine learning data model that associates words with corresponding usage contexts over a window of time, according to a diffusion process, wherein the machine learning data model comprises a plurality of skip-gram models, wherein each skip-gram model comprises a word embedding vector and a context embedding vector for a respective time step associated with the respective skip-gram model; generating a smoothed model by applying a variational inference operation over the machine learning data model; and identifying, based on the smoothed model and the corpus of electronic text, a change in a semantic use of a word over at least a portion of the window of time.