19.1 Graph Neural Networks (GNNs) I: Foundations

Interactive Graph Neural Network Visualizations

1. Graph Convolution / Message Passing

Visualizes one round of message passing: nodes aggregate neighbor representations.

Aggregation Layers 1

2. Embedding Space Evolution

Shows synthetic 2D projection of node embeddings as layers deepen.

Show Labels Communities

3. Contrastive Learning (Positive vs Negative Pairs)

Illustrates how positive pairs are pulled together and negative pairs pushed apart in representation space.

Epoch 0

4. Graph Laplacian & Spectral Properties

Visualizes eigenvalues (spectrum) and selected eigenvector components ordering nodes.

Eigenvector

5. Message Passing Equation

Standard form: h_i^{(k+1)} = \sigma( W_k * AGG_{j∈N(i)∪{i}} ( h_j^{(k)} / c_{ij} ) ) with normalization constant c_{ij}.

Adjust normalization and observe effect on updated embeddings.

Normalization

20.3 Word Embeddings II: GloVe and FastText

While Word2Vec provided a powerful new way of learning word representations, the research community continued to explore alternative methods. Two other highly influential models are GloVe (Global Vectors for Word Representation) and FastText. Each offers a different perspective on how to best capture word meaning from text corpora.

GloVe: Global Vectors for Word Representation

Developed at Stanford University by Pennington, Socher, and Manning, GloVe combines the strengths of two major families of embedding methods: global matrix factorization methods (like Latent Semantic Analysis, LSA) and local context window methods (like Word2Vec's Skip-gram).

The core idea behind GloVe is that the ratio of word-word co-occurrence probabilities has the potential to encode meaning. For example, let's consider the words "ice" and "steam".

The ratio of P(k | ice) / P(k | steam) will be large for words k related to ice, like "solid".
The ratio will be small for words k related to steam, like "gas".
The ratio will be close to 1 for words that are related to both (like "water") or related to neither (like "fashion").

GloVe is trained on a global word-word co-occurrence matrix, $X$, where each element $X_{ij}$ represents how many times word j appears in the context of word i. The model learns word vectors (embeddings) such that their dot product equals the logarithm of their co-occurrence probability.

The cost function for GloVe is:

$$ J = \sum_{i,j=1}^{V} f(X_{ij}) (w_i^T \tilde{w}_j + b_i + \tilde{b}_j - \log X_{ij})^2 $$

Where:

$w_i$ and $\tilde{w}_j$ are the word and context vectors for words i and j.
$b_i$ and $\tilde{b}_j$ are bias terms.
$f(X_{ij})$ is a weighting function that gives less weight to very frequent co-occurrences (like "the" and "is"), preventing them from dominating the training process.

By training directly on the global co-occurrence counts, GloVe can leverage statistical information more efficiently than Word2Vec, which only looks at local context windows. This often results in high-quality embeddings, even with smaller corpora.

FastText: Enriching Word Vectors with Subword Information

A major limitation of both Word2Vec and GloVe is their inability to handle out-of-vocabulary (OOV) words. If a word was not seen during training, the model has no embedding for it. Furthermore, these models ignore the internal structure of words. For example, they don't inherently understand that "helpful" and "unhelpful" are related.

FastText, developed by Facebook's AI Research (FAIR) lab, addresses this by treating each word as a composition of character n-grams. For example, the word "where" with n=3 would be represented by the n-grams: <wh, whe, her, ere, re> (plus the special sequence <where> for the whole word).

The model then learns embeddings for all of these character n-grams. The final vector for a word is simply the sum of the vectors of its constituent n-grams. This approach has two key advantages:

Handling OOV Words: If a word is not in the vocabulary, FastText can still construct a vector for it by summing the vectors of its n-grams. This is incredibly useful for morphologically rich languages (like German or Turkish) and for handling typos or rare words.
Capturing Subword Meaning: By learning vectors for subwords, the model can understand morphological similarities. It can infer that words like "baking", "baked", and "baker" are semantically related because they share common n-grams (e.g., "bak").

FastText uses an architecture very similar to Word2Vec's Skip-gram or CBOW, but instead of predicting a word, it's trained to predict the presence or absence of context words. This makes it highly efficient and scalable.

Summary: Word2Vec vs. GloVe vs. FastText

Model	Core Idea	Strengths	Weaknesses
Word2Vec	Predict context from word (Skip-gram) or word from context (CBOW).	Simple, efficient, captures complex semantic analogies.	Ignores global stats, cannot handle OOV words.
GloVe	Factorize a global word-word co-occurrence matrix.	Leverages global statistics effectively, often performs well on smaller datasets.	Cannot handle OOV words.
FastText	Represent words as a sum of character n-gram vectors.	Excellent at handling OOV words and morphological similarities.	Can be slower and require more memory due to storing n-gram vectors.

With these powerful embedding techniques, we have a solid foundation for representing text in a way that neural networks can understand. Our next step is to explore how we can improve upon the sequence processing capabilities of RNNs by introducing the concept of attention.