20.2 Word Embeddings I: Word2Vec

One of the most significant breakthroughs in the neural era of NLP was the development of word embeddings—dense vector representations of words that capture their semantic meanings. Before embeddings, models treated words as discrete, atomic symbols (e.g., via one-hot encoding). This meant that the model had no inherent understanding that "cat" and "kitten" are more similar than "cat" and "car". Word embeddings solve this by mapping words into a low-dimensional continuous vector space where semantic similarity corresponds to proximity.

Word2Vec, introduced by a team at Google led by Tomas Mikolov in 2013, is a seminal framework for learning word embeddings from large text corpora. It is not a single algorithm but a pair of model architectures: Continuous Bag-of-Words (CBOW) and Skip-gram.

The Core Idea: Distributional Hypothesis

Word2Vec is based on the distributional hypothesis, which states that "a word is characterized by the company it keeps." In other words, words that appear in similar contexts are likely to have similar meanings. Word2Vec operationalizes this idea by training a simple neural network on a proxy task: predicting a word given its context, or predicting the context given a word.

1. Continuous Bag-of-Words (CBOW)

In the CBOW model, the goal is to predict a target word based on its surrounding context words. For example, given the sentence "The quick brown ___ jumps over the lazy dog," and a context window of size 2, the model would use the words {"The", "quick", "jumps", "over"} to predict the target word "fox". The "bag-of-words" part of the name comes from the fact that the order of the context words does not influence the prediction.

The network is simple: the one-hot encoded context words are mapped to their embedding vectors, which are then averaged. This average vector is fed into a dense layer that uses a softmax activation to produce a probability distribution over the entire vocabulary for the target word.

2. Skip-gram

The Skip-gram model flips the CBOW architecture on its head. Instead of predicting a word from its context, it uses a single input word to predict its surrounding context words. For the same example, given the input word "fox", the model would try to predict {"The", "quick", "jumps", "over"}. This is generally considered to be more effective for learning high-quality embeddings, especially for infrequent words, though it is computationally more expensive.

During training, the network's weights are adjusted via backpropagation. However, the true goal is not the prediction task itself, but the learned weights of the hidden layer. These weights form the embedding matrix, where each row corresponds to the dense vector representation for a word in the vocabulary.

Word2Vec revolutionized NLP by providing a way to feed rich, semantic information about words into downstream models. This dramatically improved performance on tasks like sentiment analysis, machine translation, and named entity recognition. In the next chapter, we will explore other popular embedding techniques like GloVe and FastText.