6.1 Data Exploration and Introduction to Working with Data

Introduction

Data exploration is the critical first step in any machine learning project. Before building models, we must understand the data's structure, distribution, relationships, and quality. For LLMs, this means analyzing text corpora, token distributions, and linguistic patterns.

Descriptive Statistics

Key statistical measures help summarize data:

Central Tendency:
- Mean: $ \mu = \frac{1}{n}\sum_{i=1}^{n} x_i $
- Median: Middle value when data is sorted
- Mode: Most frequent value
Spread:
- Variance: $ \sigma^2 = \frac{1}{n}\sum_{i=1}^{n} (x_i - \mu)^2 $
- Standard Deviation: $ \sigma = \sqrt{\sigma^2} $
- Range: $ \max(x) - \min(x) $
Shape:
- Skewness: $ \frac{1}{n}\sum_{i=1}^{n} \left(\frac{x_i - \mu}{\sigma}\right)^3 $
- Kurtosis: Measure of tail heaviness

Interactive: Distribution Explorer

Generate different distributions and explore their statistics:

Distribution Type: Sample Size: {{sampleSize}}

Descriptive Statistics

Count:	{{stats.count}}
Mean (μ):	{{stats.mean \| number:3}}
Median:	{{stats.median \| number:3}}
Std Dev (σ):	{{stats.std \| number:3}}
Min:	{{stats.min \| number:3}}
Max:	{{stats.max \| number:3}}
Range:	{{stats.range \| number:3}}
Skewness:	{{stats.skewness \| number:3}}

Correlation Analysis

Understanding relationships between features is essential. The Pearson correlation coefficient measures linear relationships: $$\rho_{X,Y} = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y} = \frac{\sum_{i=1}^{n}(x_i - \mu_X)(y_i - \mu_Y)}{\sqrt{\sum_{i=1}^{n}(x_i - \mu_X)^2}\sqrt{\sum_{i=1}^{n}(y_i - \mu_Y)^2}}$$ where $ \rho \in [-1, 1] $. $ \rho = 1 $ indicates perfect positive correlation, $ \rho = -1 $ perfect negative correlation, and $ \rho = 0 $ no linear correlation.

Interactive: Correlation Explorer

Adjust correlation and see the scatter plot:

Correlation (ρ): {{correlation | number:1}} Noise Level: {{noiseLevel | number:1}}

Pearson Correlation: {{actualCorrelation | number:3}}

Covariance: {{covariance | number:3}}

R² (explained variance): {{rSquared | number:3}}

Multivariate Analysis

When dealing with multiple features, we use:

Correlation Matrix: Pairwise correlations between all features
Covariance Matrix: $ \Sigma_{ij} = \text{Cov}(X_i, X_j) $
Principal Components: Directions of maximum variance

Interactive: Correlation Matrix Heatmap

Explore relationships between multiple features:

Number of Features: {{numFeatures}}

Strong Correlations: {{strongCorrelations.length}} pairs with |ρ| > 0.7

Selected Pair: F{{selectedPair.i+1}} vs F{{selectedPair.j+1}} (ρ = {{selectedPair.corr | number:2}})

Data Quality Assessment

During exploration, identify:

Missing Values: $ \frac{\text{# missing}}{\text{# total}} \times 100\% $
Outliers: Values beyond $ \mu \pm 3\sigma $
Duplicates: Identical or near-identical records
Inconsistencies: Contradictory values

Interactive: Box Plot Analysis

Box plots reveal distribution shape and outliers:

Add Outliers: Number of Groups: {{numGroups}}

Explanation:

Box: Interquartile range (IQR) containing middle 50% of data
Line in box: Median (Q2)
Whiskers: Extend to Q1 - 1.5×IQR and Q3 + 1.5×IQR
Dots: Outliers beyond whiskers

Interactive: Data Quality Explorer (Drag & Z-score Outliers)

Drag points to create outliers and adjust the Z-score threshold to see automatic detection. Threshold lines animate as you change the slider.

Points: {{qualityPointsCount}} Z-Score Threshold: {{zThreshold | number:1}} Show Distance Labels

Mean (X,Y): {{qualityStats.meanX | number:2}}, {{qualityStats.meanY | number:2}} | Std (X,Y): {{qualityStats.stdX | number:2}}, {{qualityStats.stdY | number:2}}

Outliers Detected: {{qualityStats.outliers}} / {{qualityStats.count}} ({{(qualityStats.outliers/qualityStats.count*100) | number:1}}%)

Skewness (X,Y): {{qualityStats.skewX | number:2}}, {{qualityStats.skewY | number:2}}

Outliers highlighted in red when |Z| > threshold on either axis.
Drag any point to see real-time recomputation of mean, std and outlier flags.
Use lower thresholds (e.g., 2) to be more sensitive; higher thresholds (e.g., 3) for classical 3σ rule.

Time Series Patterns

For sequential data (important for LLM training):

Trend: Long-term increase or decrease
Seasonality: Regular periodic fluctuations
Autocorrelation: Correlation with lagged versions of itself $$\rho_k = \frac{\sum_{t=k+1}^{n}(x_t - \mu)(x_{t-k} - \mu)}{\sum_{t=1}^{n}(x_t - \mu)^2}$$

Interactive: Time Series Components

Decompose time series into trend, seasonality, and noise:

Trend Strength: {{trendStrength | number:1}} Seasonality: {{seasonality | number:1}} Noise: {{tsNoise | number:2}}

Best Practices for Data Exploration

Start with Summary Statistics: Get an overview before diving deep
Visualize Everything: Plots reveal patterns statistics might miss
Check Assumptions: Verify normality, independence, homoscedasticity
Look for Anomalies: Outliers, missing patterns, data entry errors
Understand Domain: Context matters for interpretation
Document Findings: Record insights and decisions

Sample Dataset Overview

Example: Text Document Statistics (for LLM Training)

Metric	Value	Interpretation
Total Documents	1,000,000	Training corpus size
Avg Document Length	512 tokens	Typical context window
Vocabulary Size	50,000 tokens	Unique tokens in corpus
Token Frequency (Zipf α)	1.2	Power law distribution
Missing/Corrupted	0.5%	Data quality indicator
Language Diversity	15 languages	Multilingual coverage