5.4 Imbalanced Data and Discretization

Part 1: Imbalanced Data

Introduction

Imbalanced data occurs when classes are not represented equally in the dataset. This is common in real-world applications like fraud detection (99% legitimate, 1% fraud) or medical diagnosis (95% healthy, 5% disease). Standard ML algorithms optimize for overall accuracy, which can lead to poor performance on minority classes.

Mathematical Problem

Given dataset $ \mathcal{D} = \{(x_i, y_i)\}_{i=1}^n $ where $ y_i \in \{0, 1\} $, let: $$\text{Imbalance Ratio} = \frac{|\{i : y_i = 0\}|}{|\{i : y_i = 1\}|}$$ When this ratio is large (e.g., 99:1), a naive classifier that always predicts class 0 achieves 99% accuracy but has 0% recall for class 1.

Interactive: Imbalance Impact

Adjust the class imbalance ratio and see how it affects model performance:

Minority Class %: {{minorityPct}}% Model Type:

Accuracy: {{accuracy | number:3}}

Precision (Minority): {{precision | number:3}}

Recall (Minority): {{recall | number:3}}

F1 Score: {{f1Score | number:3}}

Handling Strategies

1. Resampling

Oversampling: Duplicate minority class samples
SMOTE: Synthetic Minority Over-sampling Technique $$x_{\text{new}} = x_i + \lambda (x_j - x_i)$$ where $ x_j $ is a k-nearest neighbor of $ x_i $ and $ \lambda \in [0,1] $
Undersampling: Reduce majority class samples

Interactive: SMOTE Visualization

See how SMOTE generates synthetic samples:

Oversampling Factor: {{smoteFactor}}x k-Neighbors: {{smoteK}}

Original Minority Samples: {{originalMinority}}

Synthetic Samples Generated: {{syntheticGenerated}}

New Class Balance: {{newBalance | number:1}}%

2. Cost-Sensitive Learning

Modify the loss function to penalize misclassification of minority class more: $$L_{\text{weighted}} = \sum_{i=1}^{n} w_{y_i} \cdot L(y_i, \hat{y}_i)$$ where $ w_1 = \frac{n}{2 \cdot n_1} $ and $ w_0 = \frac{n}{2 \cdot n_0} $ balance the classes.

3. Evaluation Metrics

For imbalanced data, use metrics beyond accuracy:

Precision: $ P = \frac{TP}{TP + FP} $ - What fraction of predicted positives are correct?
Recall: $ R = \frac{TP}{TP + FN} $ - What fraction of actual positives are found?
F1 Score: $ F_1 = 2 \cdot \frac{P \cdot R}{P + R} $ - Harmonic mean of precision and recall
AUC-ROC: Area under the Receiver Operating Characteristic curve

Part 2: Discretization

Introduction

Discretization (binning) converts continuous features into discrete intervals. This can:

Reduce noise and overfitting
Handle non-linear relationships
Make models more interpretable
Improve computational efficiency

Mathematical Formulation

Given continuous feature $ x \in \mathbb{R} $, define bins $ B_1, B_2, \ldots, B_k $ where: $$B_j = [t_{j-1}, t_j), \quad j = 1, \ldots, k$$ The discretization function is: $$\phi(x) = j \quad \text{if } x \in B_j$$

Discretization Methods

1. Equal-Width Binning

Divide the range into $ k $ equal intervals: $$t_j = x_{\min} + j \cdot \frac{x_{\max} - x_{\min}}{k}, \quad j = 0, \ldots, k$$

2. Equal-Frequency Binning

Each bin contains approximately $ n/k $ samples. Bins are determined by quantiles.

3. Clustering-Based

Use k-means or other clustering to determine bin boundaries based on data distribution.

Interactive: Discretization Comparison

Compare different binning strategies:

Number of Bins: {{numBins}} Method: Distribution:

Information Loss: {{infoLoss | number:3}} (lower is better)

Bin Sizes: {{binSizes}}

Impact on Model Performance

Interactive: Discretization Effect on Prediction

See how discretization affects model fitting:

Bins for Feature: {{featureBins === 0 ? 'Continuous' : featureBins}}

Continuous MSE: {{continuousMse | number:3}}

Discretized MSE: {{discretizedMse | number:3}}

Model Interpretability: {{featureBins === 0 ? 'Low' : 'High'}}

Applications in LLMs

Token Vocabularies: Text discretization into finite token sets
Positional Encodings: Discretizing position information
Attention Bucketing: Grouping sequence lengths for efficiency
Quantization: Discretizing continuous model weights for deployment

Best Practices Summary

Problem	Solution	When to Use
Severe Imbalance (99:1)	SMOTE + Cost-Sensitive	Fraud detection, anomaly detection
Moderate Imbalance (80:20)	Weighted Loss	Most classification tasks
Skewed Distribution	Equal-Frequency Binning	Income, age, time features
Uniform Distribution	Equal-Width Binning	Measurement data
Complex Patterns	Keep Continuous	Neural networks, ensemble methods