6.4 Applications of Data Preprocessing

Introduction

Data preprocessing is crucial for machine learning success. Poor quality data leads to poor models, regardless of algorithm sophistication. For LLMs, preprocessing includes tokenization, normalization, handling special characters, and creating proper training sequences. This page shows practical before-and-after comparisons demonstrating preprocessing impact.

1. Text Normalization for LLMs

Raw text contains inconsistencies that can confuse models. Normalization includes:

Case Normalization: Converting to lowercase reduces vocabulary size
Whitespace Handling: Removing extra spaces and special characters
Unicode Normalization: Handling different encodings of same character
Punctuation Handling: Separating or removing punctuation

Interactive: Text Normalization Impact

See how normalization affects token distribution:

Lowercase Remove Punctuation Normalize Whitespace

Before Normalization

Unique Tokens: {{beforeTokens}}

After Normalization

Unique Tokens: {{afterTokens}}

Reduction: {{tokenReduction}}% fewer unique tokens

Benefit: Smaller vocabulary, better generalization

2. Outlier Removal Impact

Outliers can drastically affect model training. For a dataset with values $ X = \{x_1, x_2, \ldots, x_n\} $, outliers are typically defined using the IQR method: $$\text{Outlier if } x < Q_1 - 1.5 \cdot IQR \text{ or } x > Q_3 + 1.5 \cdot IQR$$ where $ IQR = Q_3 - Q_1 $ is the interquartile range.

Interactive: Outlier Removal Effect

Compare model performance with and without outlier removal:

Outlier Percentage: {{outlierPercent}}%

With Outliers

R² Score: {{withOutliersR2 | number:3}}

MSE: {{withOutliersMSE | number:2}}

Without Outliers

R² Score: {{withoutOutliersR2 | number:3}}

MSE: {{withoutOutliersMSE | number:2}}

Improvement: {{r2Improvement}}% better R² score

3. Feature Scaling Impact

Many algorithms are sensitive to feature scales. Gradient descent converges faster when features have similar scales. For features $ x_1, x_2, \ldots, x_n $ with vastly different ranges, standardization helps: $$z = \frac{x - \mu}{\sigma}$$

Interactive: Scaling Effect on Convergence

Compare gradient descent convergence with and without scaling:

Scale Difference: {{scaleDiff}}x

Without Scaling

Iterations to Converge: {{noScaleIterations}}

With Scaling

Iterations to Converge: {{withScaleIterations}}

Speed-up: {{scaleSpeedup}}x faster convergence

4. Missing Data Imputation

Missing data can lead to biased or inefficient models. Common imputation strategies:

Mean Imputation: Replace with feature mean (for MCAR)
Median Imputation: More robust to outliers
KNN Imputation: Use k-nearest neighbors' values
Model-based: Predict missing values using other features

Interactive: Imputation Strategy Comparison

Compare different imputation methods:

Missing Data %: {{missingPercent}}% Imputation Method:

Original MSE: {{originalMSE | number:3}}

With Missing MSE: {{missingMSE | number:3}}

After Imputation MSE: {{imputedMSE | number:3}}

Interactive: Missing Data Pattern Visualization

Explore structured vs random missingness patterns. Black cells = observed, light cells = missing.

Pattern Type: Rows: {{missingRows}} Cols: {{missingCols}} Missing %: {{missingRate}}%

Overall Missing: {{missingStats.overall | number:1}}% | Most Missing Column: {{missingStats.maxCol.name}} ({{missingStats.maxCol.pct | number:1}}%) | Row Missing Variance: {{missingStats.rowVar | number:2}}

MCAR: Missing Completely At Random – uniform scatter.
MAR: Missing At Random – systematic gaps tied to another feature.
MNAR: Missing Not At Random – depends on unobserved / the value itself.
Block: Contiguous segments missing (e.g., sensor downtime).

5. Data Augmentation for Text

Data augmentation increases training data diversity without manual labeling. For text:

Synonym Replacement: Replace words with synonyms
Back Translation: Translate to another language and back
Random Insertion/Deletion: Add or remove words
Paraphrasing: Generate semantically similar sentences

Interactive: Augmentation Impact on Model

See how augmentation affects training data distribution:

Augmentation Factor: {{augmentFactor}}x Augmentation Type:

Original Samples: {{originalSamples}}

Validation Accuracy: {{originalAcc | number:1}}%

Augmented Samples: {{augmentedSamples}}

Validation Accuracy: {{augmentedAcc | number:1}}%

6. Complete Preprocessing Pipeline

A typical preprocessing pipeline combines multiple techniques: $$\text{Pipeline: } X_{\text{raw}} \xrightarrow{\text{clean}} X_{\text{clean}} \xrightarrow{\text{impute}} X_{\text{complete}} \xrightarrow{\text{scale}} X_{\text{normalized}} \xrightarrow{\text{select}} X_{\text{final}}$$

Interactive: Full Pipeline Comparison

Compare model performance with different pipeline stages:

Remove Outliers Impute Missing Feature Scaling Feature Selection

Raw Data Accuracy: {{rawAccuracy | number:1}}%

Processed Data Accuracy: {{processedAccuracy | number:1}}%

Overall Improvement: +{{accuracyImprovement | number:1}} percentage points

Best Practices Summary

Understand Your Data: Always explore before preprocessing
Document Steps: Keep track of all transformations for reproducibility
Use Training Data Only: Fit preprocessing on training set, apply to validation/test
Validate Choices: Compare different preprocessing strategies
Preserve Information: Only remove data when necessary
Consider Domain: Different domains need different preprocessing
Automate Pipelines: Use tools like scikit-learn Pipeline for consistency
Monitor Impact: Measure how each step affects model performance

Preprocessing Checklist for LLMs

Step	Action	Why It Matters
1. Text Cleaning	Remove HTML, special chars, normalize unicode	Reduces noise, prevents encoding issues
2. Tokenization	Split into subwords (BPE, WordPiece)	Balances vocabulary size and coverage
3. Length Handling	Truncate or split long sequences	Fits model context window
4. Special Tokens	Add [CLS], [SEP], [PAD], [MASK]	Provides structure for model
5. Deduplication	Remove duplicate sequences	Prevents memorization, improves generalization
6. Quality Filtering	Remove low-quality, toxic content	Improves model behavior and safety

Conclusion

Effective preprocessing is the foundation of successful machine learning. For LLMs, proper data preparation can mean the difference between a model that generalizes well and one that overfits or fails to learn. Always measure the impact of preprocessing decisions on your specific task and dataset.