6.2 Feature Extraction and Preprocessing

Introduction

Feature extraction transforms raw data into a representation suitable for machine learning. For LLMs, this includes tokenization, embeddings, and positional encoding. Preprocessing ensures features are properly scaled and normalized.

Feature Scaling

Different features often have different scales. Scaling ensures all features contribute equally to the model.

1. Min-Max Normalization

Scales features to a fixed range [0, 1]: $$x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$$

2. Standardization (Z-score)

Transforms features to have mean 0 and standard deviation 1: $$x' = \frac{x - \mu}{\sigma}$$ where $ \mu $ is the mean and $ \sigma $ is the standard deviation.

3. Robust Scaling

Uses median and IQR, robust to outliers: $$x' = \frac{x - \text{median}(x)}{\text{IQR}(x)}$$

Interactive: Feature Scaling Comparison

See how different scaling methods affect the data:

Scaling Method: Include Outliers:

Original: Mean = {{originalStats.mean | number:2}}, Std = {{originalStats.std | number:2}}

Scaled: Mean = {{scaledStats.mean | number:2}}, Std = {{scaledStats.std | number:2}}

Range: [{{scaledStats.min | number:2}}, {{scaledStats.max | number:2}}] | Skew (orig → scaled): {{originalStats.skew | number:2}} → {{scaledStats.skew | number:2}}

IQR Compression: {{iqrCompression | number:2}}x (lower is more compressed)

Interactive: Box-Cox Transformation Animation

Box-Cox aims to stabilize variance and make data more normal. For λ ≠ 0: $ y(\lambda) = \frac{x^{\lambda}-1}{\lambda} $; for λ = 0: $ y = \ln(x) $.

Lambda (λ): {{boxCoxLambda | number:1}} Distribution:

Skewness: Original = {{boxCoxStats.origSkew | number:2}} → Transformed = {{boxCoxStats.transSkew | number:2}}

Shapiro Approx (proxy): {{boxCoxStats.normality | number:2}} (higher ~ more normal)

Variance: {{boxCoxStats.origVar | number:2}} → {{boxCoxStats.transVar | number:2}}

Move λ slowly to watch the histogram morph toward symmetry.
λ ≈ 0 often appropriate for multiplicative / log-normal processes.
Use transformed data in models needing normality / constant variance.

Dimensionality Reduction

Reducing the number of features while preserving information improves efficiency and reduces overfitting.

Principal Component Analysis (PCA)

PCA finds orthogonal directions of maximum variance. Given data matrix $ X \in \mathbb{R}^{n \times d} $:

Center the data: $ X_c = X - \mu $
Compute covariance matrix: $ \Sigma = \frac{1}{n}X_c^T X_c $
Find eigenvectors $ v_1, \ldots, v_d $ of $ \Sigma $
Project onto top $ k $ eigenvectors: $ Z = X_c V_k $

The explained variance ratio for component $ i $ is: $$\frac{\lambda_i}{\sum_{j=1}^{d} \lambda_j}$$ where $ \lambda_i $ are the eigenvalues.

Interactive: PCA Visualization

See how PCA reduces 2D data to 1D:

Data Spread: {{dataSpread | number:1}} Correlation: {{pcaCorr | number:1}}

PC1 Explained Variance: {{pc1Variance | number:1}}%

PC2 Explained Variance: {{pc2Variance | number:1}}%

Information Retained: {{pc1Variance | number:0}}% with 1 dimension

Feature Engineering

Creating new features from existing ones can dramatically improve model performance.

Polynomial Features

For features $ x_1, x_2 $, degree-2 polynomial features are: $$\{1, x_1, x_2, x_1^2, x_1 x_2, x_2^2\}$$ This allows linear models to capture non-linear relationships.

Interactive: Polynomial Feature Transform

See how polynomial features enable linear models to fit non-linear data:

Polynomial Degree: {{polyDegree}} True Function:

Number of Features: Original = 1, Polynomial = {{numPolyFeatures}}

Training MSE: {{polyMSE | number:4}}

Fit Quality: {{polyMSE < 0.5 ? 'Excellent' : polyMSE < 2 ? 'Good' : 'Poor'}}

Text Feature Extraction (for LLMs)

1. Tokenization

Converting text into discrete tokens:

Word-level: "machine learning" → ["machine", "learning"]
Subword (BPE): "learning" → ["learn", "##ing"]
Character-level: "cat" → ["c", "a", "t"]

2. Embedding

Mapping tokens to continuous vectors. For vocabulary size $ V $ and embedding dimension $ d $: $$\text{Embedding}: \{1, \ldots, V\} \rightarrow \mathbb{R}^d$$ Similar words have similar embeddings (cosine similarity): $$\text{sim}(w_1, w_2) = \frac{e_1 \cdot e_2}{\|e_1\| \|e_2\|}$$

Interactive: Word Embedding Space

Visualize how words cluster in embedding space:

Embedding Dimension: Actual: High-dimensional → Shown: 2D (via PCA)

Word Categories: Animals (red), Actions (blue), Objects (green)

Observation: Similar words cluster together in embedding space

3. Positional Encoding

Since transformers have no inherent sequence order, we add positional information: $$\text{PE}(pos, 2i) = \sin\left(\frac{pos}{10000^{2i/d}}\right)$$ $$\text{PE}(pos, 2i+1) = \cos\left(\frac{pos}{10000^{2i/d}}\right)$$ where $ pos $ is position and $ i $ is dimension index.

Interactive: Positional Encoding Patterns

Visualize how positional encodings vary across positions:

Sequence Length: {{seqLen}} Embedding Dim: {{embDim}}

Pattern: Each dimension oscillates at a different frequency

Property: Relative positions have consistent patterns

Best Practices

Technique	When to Use	Caution
Min-Max Scaling	Neural networks, known value ranges	Sensitive to outliers
Standardization	Linear models, gradient descent	Doesn't bound values
PCA	High dimensions, linear correlations	Assumes linear relationships
Polynomial Features	Non-linear patterns, small datasets	Exponential feature growth
Embeddings	Categorical data, text, sequences	Requires large training data

Summary

Effective preprocessing pipeline for LLMs:

Tokenize: Convert text to tokens
Embed: Map tokens to continuous vectors
Add Position: Inject sequence information
Normalize: Layer normalization for stability
Augment: Create variations for robustness