10.1 Classification Problem Setup

Up Next: In Sections 10.2–10.4 you'll interactively explore: (a) Logistic vs Linear output and sigmoid derivatives, (b) Maximum Likelihood and gradient descent paths, and (c) Softmax probability surfaces plus One-vs-Rest vs One-vs-One strategies. Use the navigation below to continue.

Introduction

Classification is supervised learning task where goal is to predict discrete class labels. Unlike regression which predicts continuous values, classification assigns inputs to predefined categories. This is foundational for many AI applications including language understanding, where we classify text into sentiment, topics, or intent.

1. Classification vs Regression

Aspect	Regression	Classification
Output	Continuous values (ℝ)	Discrete labels {1, 2, ..., K}
Example	House price, temperature	Spam/not spam, digit 0-9
Loss function	MSE, MAE	Cross-entropy, 0-1 loss
Prediction	Single number	Class probabilities

2. Binary vs Multi-class Classification

Binary Classification

Two classes only: $y \in \{0, 1\}$ or $y \in \{-1, +1\}$

Spam detection (spam/not spam)
Medical diagnosis (disease/healthy)
Sentiment analysis (positive/negative)

Multi-class Classification

More than two classes: $y \in \{1, 2, ..., K\}$

Digit recognition (0-9)
Language detection (English, Spanish, French, ...)
Image classification (cat, dog, bird, ...)

Interactive: Binary vs Multi-class

Visualize decision boundaries:

Classification Type: Data Separation: {{separation}}/10

Binary: Single decision boundary separates two classes

Multi-class: Multiple boundaries partition space into regions

3. Mathematical Formulation

Given training data $\{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$ where:

$\mathbf{x}_i \in \mathbb{R}^d$: Feature vector (d-dimensional input)
$y_i \in \{1, 2, ..., K\}$: Class label

Goal: Learn function $f: \mathbb{R}^d \to \{1, 2, ..., K\}$ that predicts class for new input.

Probabilistic Formulation

Instead of deterministic prediction, learn probability distribution: $$P(y = k | \mathbf{x}, \boldsymbol{\theta})$$ Predict class with highest probability: $$\hat{y} = \arg\max_{k} P(y = k | \mathbf{x}, \boldsymbol{\theta})$$

4. Decision Boundaries

Decision boundary is hypersurface separating different classes. For binary classification with classes 0 and 1: $$\text{Decision boundary: } \{\mathbf{x} : P(y=1|\mathbf{x}) = P(y=0|\mathbf{x})\} = \{\mathbf{x} : P(y=1|\mathbf{x}) = 0.5\}$$

Types of Decision Boundaries

Linear: Straight line/hyperplane (logistic regression, linear SVM)
Non-linear: Curved boundaries (decision trees, kernel SVM, neural networks)

Interactive: Decision Boundaries

Explore different boundary shapes:

Boundary Type:

Linear boundaries: Simple, fast, interpretable but limited expressiveness

Non-linear boundaries: More flexible but risk overfitting

5. Performance Metrics

Accuracy

Fraction of correct predictions: $$\text{Accuracy} = \frac{\text{Number of correct predictions}}{n}$$

Problem: Misleading with imbalanced classes (e.g., 99% spam, predicting all spam gives 99% accuracy).

Confusion Matrix

For binary classification:

	Predicted Negative	Predicted Positive
Actual Negative	True Negative (TN)	False Positive (FP)
Actual Positive	False Negative (FN)	True Positive (TP)

Precision and Recall

$$\text{Precision} = \frac{TP}{TP + FP} \quad \text{(Of predicted positives, how many correct?)}$$ $$\text{Recall} = \frac{TP}{TP + FN} \quad \text{(Of actual positives, how many found?)}$$ $$\text{F1 Score} = 2 \cdot \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

Interactive: Confusion Matrix

Adjust classification threshold to see tradeoffs:

Classification Threshold: {{threshold | number:2}}

Accuracy

Precision

Recall

F1 Score

6. Class Imbalance

Real-world datasets often have unequal class frequencies. Example: fraud detection (0.1% fraudulent transactions).

Handling Imbalance

Resampling: Oversample minority class or undersample majority
Class weights: Penalize misclassifying minority class more heavily
Different metrics: Use precision/recall/F1 instead of accuracy
Generate synthetic data: SMOTE and similar techniques

Interactive: Class Imbalance Effect

See how imbalance affects metrics:

Class Ratio (Positive/Negative): {{classRatio | number:2}}

Imbalanced data: Accuracy can be misleading - focus on precision/recall

7. Application to LLMs

Text classification: Sentiment analysis, topic classification, spam detection
Token classification: Named entity recognition (NER), part-of-speech tagging
Next token prediction: Multi-class classification over vocabulary (softmax)
Intent classification: Chatbot intent recognition
Safety classification: Toxic content detection, prompt injection detection

Interactive: Token Classification

Simulate next-token prediction distribution:

Temperature: {{temperature}}

Vocabulary size: Modern LLMs classify over 50K+ tokens

Temperature: Controls confidence - low temperature = peaked distribution

Key Takeaways

Classification predicts discrete labels, regression predicts continuous values
Binary classification: 2 classes; Multi-class: K > 2 classes
Decision boundaries separate class regions in feature space
Accuracy alone can be misleading - use precision, recall, F1
Class imbalance requires special handling
Probabilistic formulation provides confidence estimates
LLMs use classification for tokens, sentiment, entities, safety