10.3 Parameter Learning for Logistic Regression

Introduction

Learning optimal parameters for logistic regression involves finding weights and bias that best fit training data. We use Maximum Likelihood Estimation (MLE) and gradient-based optimization methods. Understanding these optimization techniques is crucial as they form the foundation for training neural networks and LLMs.

1. Maximum Likelihood Estimation

Given training data \(\{(\mathbf{x}_i, y_i)\}_{i=1}^{n}\) where \(y_i \in \{0, 1\}\), we want to find parameters \(\boldsymbol{\theta} = (\mathbf{w}, b)\) that maximize probability of observed labels.

Likelihood Function

For single observation: $$P(y_i | \mathbf{x}_i, \boldsymbol{\theta}) = \begin{cases} \sigma(\mathbf{w}^T \mathbf{x}_i + b) & \text{if } y_i = 1 \\ 1 - \sigma(\mathbf{w}^T \mathbf{x}_i + b) & \text{if } y_i = 0 \end{cases}$$

Compact form: $$P(y_i | \mathbf{x}_i, \boldsymbol{\theta}) = \sigma(\mathbf{w}^T \mathbf{x}_i + b)^{y_i} \cdot (1 - \sigma(\mathbf{w}^T \mathbf{x}_i + b))^{1-y_i}$$

Assuming independence, likelihood for all data: $$L(\boldsymbol{\theta}) = \prod_{i=1}^{n} P(y_i | \mathbf{x}_i, \boldsymbol{\theta})$$

Log-Likelihood

Taking logarithm (monotonic transformation preserves maximum): $$\ell(\boldsymbol{\theta}) = \log L(\boldsymbol{\theta}) = \sum_{i=1}^{n} \left[ y_i \log \sigma(z_i) + (1-y_i) \log(1-\sigma(z_i)) \right]$$ where \(z_i = \mathbf{w}^T \mathbf{x}_i + b\)

2. Cross-Entropy Loss

We typically minimize negative log-likelihood (loss function): $$J(\boldsymbol{\theta}) = -\frac{1}{n}\ell(\boldsymbol{\theta}) = -\frac{1}{n}\sum_{i=1}^{n} \left[ y_i \log \hat{y}_i + (1-y_i) \log(1-\hat{y}_i) \right]$$ where \(\hat{y}_i = \sigma(\mathbf{w}^T \mathbf{x}_i + b)\) is predicted probability.

This is binary cross-entropy loss, fundamental in training classification models.

3. Gradient Computation

To minimize loss, we need gradients with respect to parameters.

Gradient with respect to weights

Using chain rule: $$\frac{\partial J}{\partial w_j} = \frac{1}{n}\sum_{i=1}^{n} (\hat{y}_i - y_i) x_{ij}$$

Vector form: $$\nabla_{\mathbf{w}} J = \frac{1}{n}\mathbf{X}^T (\hat{\mathbf{y}} - \mathbf{y})$$ where \(\mathbf{X} \in \mathbb{R}^{n \times d}\) is design matrix.

Gradient with respect to bias

$$\frac{\partial J}{\partial b} = \frac{1}{n}\sum_{i=1}^{n} (\hat{y}_i - y_i)$$

Key Property

Gradient has elegant form: \((\text{prediction} - \text{truth}) \times \text{feature}\)

This is same form as linear regression! Sigmoid's derivative property makes this work.

4. Gradient Descent Algorithm

Iterative optimization algorithm that updates parameters in direction of negative gradient:

Update Rule

$$\mathbf{w}^{(t+1)} = \mathbf{w}^{(t)} - \alpha \nabla_{\mathbf{w}} J(\boldsymbol{\theta}^{(t)})$$ $$b^{(t+1)} = b^{(t)} - \alpha \frac{\partial J}{\partial b}(\boldsymbol{\theta}^{(t)})$$ where \(\alpha > 0\) is learning rate (step size).

Algorithm Steps

1. Initialize parameters (often to zero or small random values)
2. Compute predictions \(\hat{\mathbf{y}}\) using current parameters
3. Compute loss \(J(\boldsymbol{\theta})\)
4. Compute gradients \(\nabla_{\mathbf{w}} J, \frac{\partial J}{\partial b}\)
5. Update parameters using gradients and learning rate
6. Repeat steps 2-5 until convergence

5. Learning Rate Selection

Learning rate \(\alpha\) critically affects convergence:

• Too small: Slow convergence, many iterations needed
• Too large: Overshooting, oscillation, divergence
• Just right: Fast, stable convergence

Common Strategies

• Fixed: Constant learning rate (simple, may not be optimal)
• Decay: Decrease over time (e.g., \(\alpha_t = \alpha_0 / (1 + kt)\))
• Adaptive: Different rates per parameter (Adam, RMSprop)
• Line search: Optimize step size each iteration

6. Stochastic Gradient Descent (SGD)

Standard gradient descent computes gradient using all data (batch gradient descent). For large datasets, this is expensive.

SGD Variants

• Stochastic (online): Use one sample at a time $$\mathbf{w} \leftarrow \mathbf{w} - \alpha (\hat{y}_i - y_i) \mathbf{x}_i$$
• Mini-batch: Use small batch of samples (most common) $$\mathbf{w} \leftarrow \mathbf{w} - \alpha \frac{1}{|B|} \sum_{i \in B} (\hat{y}_i - y_i) \mathbf{x}_i$$

Advantages of SGD

• Much faster per iteration (fewer computations)
• Can escape local minima (noisy updates)
• Enables online learning (update as data arrives)
• Works with massive datasets that don't fit in memory

7. Convergence Criteria

When to stop training? Common criteria:

• Max iterations: Stop after fixed number of steps
• Loss threshold: Stop when \(J(\boldsymbol{\theta}) < \epsilon\)
• Gradient norm: Stop when \(\|\nabla J\| < \epsilon\)
• Parameter change: Stop when \(\|\boldsymbol{\theta}^{(t+1)} - \boldsymbol{\theta}^{(t)}\| < \epsilon\)
• Validation performance: Stop when validation loss stops improving (early stopping)

8. Regularization

To prevent overfitting, add penalty term to loss function:

L2 Regularization (Ridge)

$$J_{\text{reg}}(\boldsymbol{\theta}) = J(\boldsymbol{\theta}) + \frac{\lambda}{2} \|\mathbf{w}\|^2$$ Gradient becomes: $$\nabla_{\mathbf{w}} J_{\text{reg}} = \nabla_{\mathbf{w}} J + \lambda \mathbf{w}$$

L1 Regularization (Lasso)

$$J_{\text{reg}}(\boldsymbol{\theta}) = J(\boldsymbol{\theta}) + \lambda \|\mathbf{w}\|_1$$ Encourages sparse solutions (many weights become exactly zero).

9. Newton's Method

Second-order optimization using Hessian (matrix of second derivatives):

$$\boldsymbol{\theta}^{(t+1)} = \boldsymbol{\theta}^{(t)} - H^{-1} \nabla J(\boldsymbol{\theta}^{(t)})$$ where \(H\) is Hessian matrix.

Advantages

• Faster convergence (quadratic near optimum)
• No learning rate needed

Disadvantages

• Expensive: requires computing and inverting \(d \times d\) Hessian
• Not practical for high-dimensional problems (modern LLMs have billions of parameters)

10. Application to LLMs

Training logistic regression teaches fundamentals used in LLMs:

• Backpropagation: Chain rule for computing gradients (same principle)
• SGD variants: Adam, AdamW used in transformer training
• Cross-entropy loss: Same loss for language modeling (over vocabulary)
• Learning rate schedules: Warmup + decay critical for LLM training
• Gradient clipping: Prevent exploding gradients
• Batch size: Trade-off between speed and gradient noise

Key Takeaways

• Maximum Likelihood Estimation gives principled approach to parameter learning
• Cross-entropy loss is negative log-likelihood
• Gradient has elegant form: (prediction - truth) × feature
• Gradient descent iteratively moves toward minimum
• Learning rate critically affects convergence speed and stability
• Stochastic/mini-batch gradient descent enables large-scale training
• Regularization prevents overfitting
• These techniques generalize to neural networks and LLMs
• Modern optimizers (Adam) build on these foundations