7.4 Feature Imputation Techniques

Introduction

Missing data is common in real-world datasets. For LLM training, this includes incomplete text, corrupted tokens, or missing metadata. Imputation fills in missing values to enable model training without losing data.

1. Types of Missing Data

Missing Completely at Random (MCAR)

Probability of missingness is independent of observed and unobserved data: $$P(\text{missing} | X_{\text{obs}}, X_{\text{miss}}) = P(\text{missing})$$ Example: Sensor randomly fails to record temperature.

Missing at Random (MAR)

Probability depends on observed data but not unobserved: $$P(\text{missing} | X_{\text{obs}}, X_{\text{miss}}) = P(\text{missing} | X_{\text{obs}})$$ Example: Older users less likely to provide phone numbers.

Missing Not at Random (MNAR)

Missingness depends on the missing value itself: $$P(\text{missing} | X_{\text{obs}}, X_{\text{miss}}) \text{ depends on } X_{\text{miss}}$$ Example: High earners don't report income.

Interactive: Missing Data Patterns

Visualize different missing data mechanisms:

Missing Data Type: Missing Percentage: {{missingPct}}%

Pattern: {{missingPattern}}

Bias Impact: {{biasImpact}}

2. Simple Imputation Methods

Mean Imputation

Replace missing values with feature mean: $$\hat{x}_i = \bar{x} = \frac{1}{n}\sum_{j=1}^{n} x_j$$ Pros: Simple, fast. Cons: Reduces variance, ignores relationships.

Median Imputation

Use median instead of mean. More robust to outliers: $$\hat{x}_i = \text{median}(x_1, \ldots, x_n)$$

Mode Imputation

For categorical data, use most frequent value: $$\hat{x}_i = \text{mode}(x_1, \ldots, x_n)$$

Interactive: Simple Imputation Comparison

Compare mean, median, and mode imputation:

Data Distribution: Mean Median

Mean Imputation

MSE

Median Imputation

MSE

Best Method

Lower MSE

3. K-Nearest Neighbors (KNN) Imputation

Use $ k $ nearest neighbors to impute missing values. For a missing value at position $ i $: $$\hat{x}_i = \frac{1}{k}\sum_{j \in N_k(i)} x_j$$ where $ N_k(i) $ are the $ k $ nearest neighbors based on other features.

Distance metric (Euclidean): $$d(i, j) = \sqrt{\sum_{f \in \text{observed}} (x_{if} - x_{jf})^2}$$

Interactive: KNN Imputation

See how KNN uses local information for imputation:

K (neighbors): {{kValue}}

KNN MSE: {{knnMSE | number:3}}

Advantage: Captures local patterns, better for non-linear relationships

4. Model-Based Imputation

Regression Imputation

Train a regression model to predict missing values from other features: $$\hat{x}_i = f(X_{\text{other features}})$$ Use linear regression, tree models, or neural networks.

Multiple Imputation

Create multiple imputed datasets, analyze each, combine results:

Generate $ m $ complete datasets by imputing missing values
Analyze each dataset separately
Pool results using Rubin's rules

Interactive: Model-Based vs Simple Imputation

Compare simple and model-based approaches:

Imputation Method: Missing %: {{modelMissingPct}}%

Original MSE

After Imputation

Recovery Rate

{{modelMetrics.recovery | number:1}}%

5. Forward Fill and Backward Fill

For time-series or sequential data (like text):

Forward Fill: $ \hat{x}_t = x_{t-1} $ (use previous value)
Backward Fill: $ \hat{x}_t = x_{t+1} $ (use next value)
Interpolation: $ \hat{x}_t = \frac{x_{t-1} + x_{t+1}}{2} $ (average neighbors)

Interactive: Sequential Data Imputation

See how sequential imputation works for time-series:

Method:

Application to LLMs: Fill missing tokens in sequences

Use Case: Corrupted text data, incomplete sentences

6. Indicator Variables

Create binary indicator for missingness to preserve information: $$I_i = \begin{cases} 1 & \text{if } x_i \text{ was missing} \\ 0 & \text{otherwise} \end{cases}$$ Then impute the original feature and include $ I_i $ as an additional feature.

Imputation Method Selection Guide

Method	Best For	Complexity	Preserves Variance?
Mean/Median	MCAR, quick baseline	Very Low	No
Mode	Categorical features	Very Low	No
KNN	Local patterns, MAR	Medium	Partially
Regression	Strong feature relationships	Medium-High	Partially
Multiple Imputation	High missingness, inference	High	Yes
Forward/Backward Fill	Time-series, sequential	Very Low	Depends

LLM-Specific Considerations

Token Masking: Similar to imputation - BERT masks tokens for prediction
Sentence Completion: Forward fill analogy for autoregressive models
Missing Context: Use surrounding text to infer missing information
Embedding Imputation: Impute missing embeddings using similar tokens