5.3 Data Quality Challenges

Introduction

Data quality is critical for machine learning success. Poor quality data leads to models that learn incorrect patterns, generalize poorly, and fail in production. For LLMs, data quality affects language understanding, generation accuracy, and the presence of biases.

Types of Data Quality Issues

Missing Data: Incomplete observations in the dataset
Noise: Random errors in measurements or labels
Outliers: Extreme values that deviate significantly from other observations
Inconsistency: Contradictory or conflicting data
Duplication: Repeated instances that can bias the model

Missing Data

Missing data can occur in three patterns:

MCAR (Missing Completely At Random): Missingness is independent of observed and unobserved data
MAR (Missing At Random): Missingness depends only on observed data
MNAR (Missing Not At Random): Missingness depends on unobserved data

Mathematical Impact

Let $ \mathcal{D} = \{(x_i, y_i)\}_{i=1}^n $ be our dataset. With missing data, we observe: $$\tilde{x}_i = \begin{cases} x_i & \text{with probability } (1-p) \\ \text{missing} & \text{with probability } p \end{cases}$$ This reduces effective sample size and can introduce bias if not handled properly.

Interactive: Missing Data Impact

See how missing data affects model training:

Missing Data Rate: {{missingRate * 100 | number:0}}% Missing Pattern:

Complete Data MSE: {{completeMse | number:3}}

With Missing Data MSE: {{missingMse | number:3}}

Error Increase: {{((missingMse - completeMse) / completeMse * 100) | number:1}}%

Noise in Training Data

Noise represents random errors in features or labels. For a true relationship $ y = f(x) $, we observe: $$\tilde{y} = f(x) + \epsilon$$ where $ \epsilon \sim \mathcal{N}(0, \sigma^2) $ is Gaussian noise.

Impact on Model Performance

Noise increases the Bayes error rate (minimum achievable error) and can cause:

Reduced model accuracy
Increased variance in predictions
Difficulty in learning true patterns
Potential overfitting to noise

Interactive: Noise Analysis

Adjust noise levels to see their effect on model fitting:

Feature Noise (σₓ): {{featureNoise}} Label Noise (σᵧ): {{labelNoise}}

Signal-to-Noise Ratio: {{snr | number:2}} dB

R² Score: {{rSquared | number:3}} (closer to 1 is better)

Outliers

Outliers are data points that deviate significantly from other observations. Mathematically, a point $ x_i $ is an outlier if: $$|x_i - \mu| > k\sigma$$ where $ \mu $ is the mean, $ \sigma $ is standard deviation, and $ k $ is typically 2-3.

Detection Methods

Z-score: $ z_i = \frac{x_i - \mu}{\sigma} $, flag if $ |z_i| > 3 $
IQR Method: Flag if $ x_i < Q_1 - 1.5 \cdot IQR $ or $ x_i > Q_3 + 1.5 \cdot IQR $
Isolation Forest: Points that are easily isolated are likely outliers

Interactive: Outlier Detection

Visualize different outlier detection methods:

Outlier Percentage: {{outlierPct}}% Detection Method:

True Outliers: {{trueOutliers}}

Detected Outliers: {{detectedOutliers}}

Precision: {{outlierPrecision | number:3}} | Recall: {{outlierRecall | number:3}}

Handling Strategies

Issue	Strategy	When to Use
Missing Data	Mean/Median Imputation	MCAR, numerical features
	Model-Based Imputation	MAR, complex relationships
	Deletion	Low missing rate (<5%)
Noise	Regularization (L1, L2)	High noise, prevent overfitting
	Smoothing/Filtering	Time series, signal data
Outliers	Removal	Clear errors, large dataset
	Capping/Winsorization	Legitimate but extreme values
	Robust Methods	When outliers are informative

Implications for LLMs

In language model training, data quality challenges manifest as:

Text Corruption: Encoding errors, formatting issues (analogous to noise)
Incomplete Documents: Truncated texts (missing data)
Toxic Content: Harmful language that should be filtered (outliers)
Factual Errors: Incorrect information in training data (label noise)
Duplicate Content: Repeated texts that skew model behavior

Best Practices

Exploratory Data Analysis: Always visualize and understand data before modeling
Document Decisions: Record how you handle each quality issue
Test Sensitivity: Evaluate model performance with different handling strategies
Validate Assumptions: Check if missingness patterns match your assumptions
Monitor in Production: Data quality can drift over time

Sample Data Statistics

Metric	Clean Data	With Issues	Impact
{{stat.metric}}	{{stat.clean \| number:3}}	{{stat.issues \| number:3}}	{{stat.impact > 0 ? '+' : ''}}{{stat.impact \| number:1}}%