10.2 Logistic Regression Model

Introduction

Logistic regression is fundamental algorithm for binary classification. Despite its name, it is classification method, not regression. It models probability that input belongs to particular class using sigmoid function, providing interpretable probabilistic predictions crucial for decision-making in LLMs and other AI systems.

1. From Linear to Logistic Regression

Linear regression predicts continuous output: $$y = \mathbf{w}^T \mathbf{x} + b$$ For classification, we need output between 0 and 1 (probability). Direct linear model fails because:

Output can be any real number, not just [0, 1]
No probabilistic interpretation
Sensitive to outliers

Solution: Apply sigmoid function to linear combination.

2. The Sigmoid Function

Sigmoid (logistic) function maps any real number to (0, 1): $$\sigma(z) = \frac{1}{1 + e^{-z}}$$

Properties

Output range: (0, 1) - perfect for probabilities
Monotonic: preserves ordering
Smooth and differentiable everywhere
Symmetry: $\sigma(-z) = 1 - \sigma(z)$
Derivative: $\sigma'(z) = \sigma(z)(1 - \sigma(z))$ - convenient for optimization

Interactive: Sigmoid Function

Explore how sigmoid transforms linear inputs:

Scale factor: {{scale}} Shift: {{shift}}

Scale: Controls steepness of sigmoid curve

Shift: Moves decision boundary (threshold where σ = 0.5)

3. Logistic Regression Model

For binary classification with features $\mathbf{x} \in \mathbb{R}^d$, logistic regression models probability: $$P(y = 1 | \mathbf{x}) = \sigma(\mathbf{w}^T \mathbf{x} + b) = \frac{1}{1 + e^{-(\mathbf{w}^T \mathbf{x} + b)}}$$

Where:

$\mathbf{w} \in \mathbb{R}^d$: Weight vector (model parameters)
$b \in \mathbb{R}$: Bias term (intercept)
$\mathbf{w}^T \mathbf{x} + b$: Linear combination (logit)

And complement probability: $$P(y = 0 | \mathbf{x}) = 1 - P(y = 1 | \mathbf{x}) = \frac{e^{-(\mathbf{w}^T \mathbf{x} + b)}}{1 + e^{-(\mathbf{w}^T \mathbf{x} + b)}}$$

4. Decision Boundary

Classification decision: predict class 1 if $P(y=1|\mathbf{x}) > 0.5$

Decision boundary is where $P(y=1|\mathbf{x}) = 0.5$: $$\sigma(\mathbf{w}^T \mathbf{x} + b) = 0.5 \implies \mathbf{w}^T \mathbf{x} + b = 0$$

This is linear boundary (hyperplane in d dimensions). Logistic regression creates linear decision boundaries.

Interactive: Decision Boundary

Adjust weights to see how decision boundary changes:

Weight w₁: {{w1}} Weight w₂: {{w2}} Bias b: {{bias}}

Decision line: Points on line have P(y=1) = 0.5

Gradient direction: Points perpendicular to boundary where probability changes fastest

Interactive: Enhanced Decision Boundary & Probability Contours

Adjust parameters and observe how the 0.5 decision line and surrounding probability field change. Contour lines show equal-probability (iso-probability) sets.

w₁: {{w1 | number:2}} w₂: {{w2 | number:2}} Bias b: {{bias | number:2}} Show probability heat Sample points

Decision line: w₁ x + w₂ y + b = 0 (P = 0.5). Light to dark blue indicates larger P(y=1). White contour lines every 0.1 probability.

5. Log-Odds and Logit Function

Log-odds (logit) is logarithm of odds ratio: $$\text{logit}(p) = \log\left(\frac{p}{1-p}\right) = \log\left(\frac{P(y=1|\mathbf{x})}{P(y=0|\mathbf{x})}\right) = \mathbf{w}^T \mathbf{x} + b$$

This shows logistic regression models log-odds as linear function of features. Key insights:

Odds ratio: $\frac{P(y=1|\mathbf{x})}{P(y=0|\mathbf{x})} = e^{\mathbf{w}^T \mathbf{x} + b}$
Each weight $w_j$ represents change in log-odds per unit increase in $x_j$
Logit function is inverse of sigmoid: $\text{logit}(\sigma(z)) = z$

Interactive: Probability vs Logit

See relationship between probability and log-odds:

Logit value: {{logitValue}}

Probability

Odds

Log-Odds (Logit)

6. Interpretation of Coefficients

For feature $x_j$ with coefficient $w_j$:

$w_j > 0$: Increasing $x_j$ increases probability of class 1
$w_j < 0$: Increasing $x_j$ decreases probability of class 1
$|w_j|$: Magnitude indicates strength of effect
One unit increase in $x_j$ multiplies odds by $e^{w_j}$

Interactive: Feature Impact

See how individual features affect prediction:

Feature 1: {{feat1}} Feature 2: {{feat2}} Feature 3: {{feat3}}

Prediction

Probability of Class 1: {{featPrediction | number:3}}

Predicted Class: {{featPrediction > 0.5 ? 'Class 1' : 'Class 0'}}

7. Model Assumptions

Logistic regression assumes:

Binary outcome: Response variable is binary (0/1)
Independence: Observations are independent
Linearity in logit: Log-odds is linear function of features
No perfect multicollinearity: Features are not perfectly correlated
Large sample size: More features require more data

8. Logistic Regression in LLMs

Logistic regression appears in many LLM contexts:

Binary token prediction: Special tokens (yes/no, start/end)
Sentiment classification: Positive/negative text classification
Safety classifiers: Toxic/safe content detection
Attention mechanisms: Sigmoid gates in LSTM/GRU variants
Final layer (binary): Binary classification heads on transformer models

Interactive: Text Sentiment Model

Simulate simple sentiment classifier with feature scores:

Positive words: {{posWords}} Negative words: {{negWords}} Intensity (caps): {{intensity}}

Sentiment Analysis

Positive Probability: {{sentimentProb | number:3}}

Prediction: {{sentimentProb > 0.5 ? 'POSITIVE' : 'NEGATIVE'}}

Confidence: {{Math.abs(sentimentProb - 0.5) * 2 | number:3}}

9. Comparison with Other Models

Model	Decision Boundary	Probabilistic	Interpretability	Training Speed
Logistic Regression	Linear	Yes	High	Fast
Linear SVM	Linear	No (can add)	Medium	Fast
Decision Tree	Axis-aligned	Yes	High	Fast
Neural Network	Non-linear	Yes	Low	Slow
Naive Bayes	Non-linear	Yes	High	Very fast

10. Advantages and Limitations

Advantages

Simple and efficient to train
Provides probability estimates (calibrated with proper training)
Highly interpretable coefficients
Works well with limited data
No hyperparameter tuning required
Resistant to overfitting with regularization

Limitations

Assumes linear decision boundary
Cannot capture complex patterns without feature engineering
Sensitive to outliers
Assumes independence of features
Requires large sample for many features

Interactive: Linear Limitation

See where logistic regression struggles:

Data pattern:

Linear separable: Logistic regression performs well

Non-linear patterns: Need feature engineering or non-linear models

Key Takeaways

Sigmoid function transforms linear combination to probability [0, 1]
Logistic regression models log-odds as linear function of features
Decision boundary is linear hyperplane in feature space
Coefficients have clear interpretation in terms of odds ratios
Provides calibrated probability estimates for classification
Simple, fast, interpretable but limited to linear boundaries
Foundation for more complex models including neural networks
Widely used in LLMs for binary classification tasks