Chapter 5: Logistic Regression

Bloom's Taxonomy Map for This Chapter

Learning Objectives

By the end of this chapter, you will be able to:

• Define logistic regression as a linear model composed with a sigmoid activation for binary classification
• Derive the sigmoid function σ(z) = 1/(1+e^−z) and prove its derivative σ′(z) = σ(z)(1 − σ(z))
• Derive the Binary Cross-Entropy loss from first principles using Maximum Likelihood Estimation
• Construct the computation graph for logistic regression and perform forward & backward passes
• Derive the gradient descent update rules ∂L/∂w and ∂L/∂b step by step
• Implement a complete LogisticRegression class from scratch using only NumPy
• Train the model on a synthetic Indian bank loan dataset and visualize the decision boundary
• Compare vectorized (NumPy) vs loop-based implementations for speed and clarity
• Execute a full forward + backward pass by hand on a worked example with 2 features and 3 samples
• Evaluate logistic regression in a real-world case study: CIBIL score prediction at SBI

Opening Hook — The ₹2 Lakh Crore Question

Core Concepts — The Mathematics of Binary Classification

Logistic regression answers one question: given input features x, what is the probability that the output belongs to class 1? It does this in three steps: (1) compute a linear combination z = w·x + b, (2) squash it through the sigmoid function to get a probability, and (3) compare that probability to the true label using cross-entropy loss. Let's derive each piece rigorously.

3a. The Sigmoid Function — From Linear to Probability

The Problem: Linear Outputs Are Unbounded

In Chapter 4, we saw that a neuron computes z = w₁x₁ + w₂x₂ + ... + wₙxₙ + b. This output z ∈ (−∞, +∞). But for binary classification, we need a probability — a number in [0, 1]. We need a function that maps ℝ → (0, 1).

Definition: The Sigmoid (Logistic) Function

Key Properties of Sigmoid

Sigmoid Value Table

3b. Binary Cross-Entropy Loss — Derived from Maximum Likelihood

We have a model that outputs ŷ = σ(w·x + b) ∈ (0, 1). We need a loss function that tells us how wrong the model is. For classification, we derive this from first principles using Maximum Likelihood Estimation (MLE).

Step 1: Define the Probabilistic Model

Our model outputs ŷ = P(y=1|x). Since y is binary (0 or 1), this is a Bernoulli distribution:

Verification:

• If y = 1: P(y=1|x) = ŷ¹ · (1−ŷ)⁰ = ŷ ✓ (we want this to be high)
• If y = 0: P(y=0|x) = ŷ⁰ · (1−ŷ)¹ = 1−ŷ ✓ (we want this to be high)

Step 2: Likelihood of the Entire Dataset

For m independent training samples {(x⁽¹⁾, y⁽¹⁾), ..., (x⁽ᵐ⁾, y⁽ᵐ⁾)}, the likelihood of observing all labels is:

Step 3: Log-Likelihood (Convert Product to Sum)

Products are numerically unstable and hard to differentiate. Take the natural log:

Step 4: From Maximizing Likelihood to Minimizing Loss

MLE says: find parameters w, b that maximize the log-likelihood. Since gradient descent minimizes, we negate and take the average:

Why This Loss Works: Intuition

3c. Gradient Descent — The Learning Algorithm

Now we have a model (sigmoid) and a loss (BCE). We need to find the best w and b that minimize J(w, b). We do this using gradient descent: repeatedly adjust parameters in the direction that reduces the loss.

The Update Rule

Deriving ∂J/∂w — The Full Chain

We need to differentiate J with respect to w. Let's use the chain rule through the computation graph:

Forward pass variables:

• z = w·x + b (linear combination)
• ŷ = a = σ(z) (activation / prediction)
• L = −[y·log(a) + (1−y)·log(1−a)] (loss for one sample)

Step 1: ∂L/∂a

Step 2: ∂a/∂z (sigmoid derivative)

Step 3: Combine using chain rule → ∂L/∂z

Step 4: ∂z/∂w and ∂z/∂b

Since z = w·x + b:

Step 5: Final gradients (chain rule all the way)

Step 6: Average over m samples for the cost gradient

3d. Computation Graph — Visualizing Forward and Backward Pass

A computation graph breaks complex operations into elementary steps, making it easy to apply the chain rule systematically. This is exactly how deep learning frameworks (PyTorch, TensorFlow) compute gradients automatically.

Vectorized Form (m samples, n features)

For the full training set where X is (n × m), y is (1 × m):

From-Scratch Implementation — Building It Yourself

Let's build a complete LogisticRegression class from scratch. This is the heart of the chapter — every line maps directly to the math we just derived.

4a. The LogisticRegression Class

Pythonimport numpy as np

class LogisticRegression:
    """
    Logistic Regression from scratch using NumPy.
    Binary classifier: predicts P(y=1|x) using sigmoid activation.
    
    Parameters
    ----------
    learning_rate : float, default=0.01
        Step size for gradient descent.
    n_iterations : int, default=1000
        Number of gradient descent iterations.
    """
    
    def __init__(self, learning_rate=0.01, n_iterations=1000):
        self.learning_rate = learning_rate
        self.n_iterations = n_iterations
        self.weights = None      # w: shape (n_features,)
        self.bias = None         # b: scalar
        self.loss_history = []    # Track loss per iteration
    
    def _sigmoid(self, z):
        """Numerically stable sigmoid function."""
        # Clip z to avoid overflow in exp
        z = np.clip(z, -500, 500)
        return np.where(
            z >= 0,
            1 / (1 + np.exp(-z)),          # For z >= 0: standard formula
            np.exp(z) / (1 + np.exp(z))     # For z < 0: equivalent, avoids overflow
        )
    
    def _compute_loss(self, y, y_hat):
        """
        Binary Cross-Entropy Loss.
        J = -(1/m) * Σ [y*log(ŷ) + (1-y)*log(1-ŷ)]
        """
        m = len(y)
        # Clip predictions to avoid log(0)
        eps = 1e-15
        y_hat = np.clip(y_hat, eps, 1 - eps)
        loss = -(1 / m) * np.sum(
            y * np.log(y_hat) + (1 - y) * np.log(1 - y_hat)
        )
        return loss
    
    def _forward(self, X):
        """
        Forward pass: X → z = Xw + b → a = σ(z)
        X shape: (m, n)  →  z shape: (m,)  →  a shape: (m,)
        """
        z = np.dot(X, self.weights) + self.bias   # Linear
        a = self._sigmoid(z)                       # Activation
        return a
    
    def _backward(self, X, y, y_hat):
        """
        Backward pass: compute gradients.
        dw = (1/m) * X^T · (ŷ - y)
        db = (1/m) * Σ(ŷ - y)
        """
        m = len(y)
        dz = y_hat - y                            # (m,) — prediction error
        dw = (1 / m) * np.dot(X.T, dz)             # (n,) — weight gradient
        db = (1 / m) * np.sum(dz)                 # scalar — bias gradient
        return dw, db
    
    def _update_parameters(self, dw, db):
        """Gradient descent update step."""
        self.weights -= self.learning_rate * dw
        self.bias -= self.learning_rate * db
    
    def fit(self, X, y):
        """
        Train the model using gradient descent.
        
        Parameters
        ----------
        X : np.ndarray of shape (m, n)
            Training features (m samples, n features).
        y : np.ndarray of shape (m,)
            Binary labels (0 or 1).
        """
        m, n = X.shape
        
        # Initialize parameters to zeros
        self.weights = np.zeros(n)
        self.bias = 0.0
        self.loss_history = []
        
        for i in range(self.n_iterations):
            # 1. Forward pass
            y_hat = self._forward(X)
            
            # 2. Compute loss (for tracking)
            loss = self._compute_loss(y, y_hat)
            self.loss_history.append(loss)
            
            # 3. Backward pass
            dw, db = self._backward(X, y, y_hat)
            
            # 4. Update parameters
            self._update_parameters(dw, db)
            
            # Print every 100 iterations
            if (i + 1) % 100 == 0:
                print(f"Iteration {i+1}/{self.n_iterations} — Loss: {loss:.6f}")
        
        return self
    
    def predict_proba(self, X):
        """Return probability predictions P(y=1|x)."""
        return self._forward(X)
    
    def predict(self, X, threshold=0.5):
        """Return binary predictions (0 or 1)."""
        return (self.predict_proba(X) >= threshold).astype(int)
    
    def accuracy(self, X, y):
        """Compute classification accuracy."""
        predictions = self.predict(X)
        return np.mean(predictions == y)

4b. Training on a Synthetic Indian Bank Loan Dataset

Pythonimport numpy as np
import matplotlib.pyplot as plt

# ─── Generate synthetic loan dataset ───
np.random.seed(42)

# Feature 1: Monthly income (₹ in thousands), normalized
# Feature 2: CIBIL score (300-900), normalized
m = 200  # 200 loan applicants

# Class 0: Defaulters (lower income, lower CIBIL)
X_default = np.random.randn(100, 2) * 0.8 + np.array([-1.0, -1.0])

# Class 1: Non-defaulters (higher income, higher CIBIL)
X_repaid = np.random.randn(100, 2) * 0.8 + np.array([1.0, 1.0])

# Combine
X = np.vstack([X_default, X_repaid])
y = np.array([0] * 100 + [1] * 100)

# Shuffle
shuffle_idx = np.random.permutation(m)
X, y = X[shuffle_idx], y[shuffle_idx]

print(f"Dataset: {X.shape[0]} samples, {X.shape[1]} features")
print(f"Class distribution: {np.sum(y==0)} defaulters, {np.sum(y==1)} non-defaulters")

# ─── Train the model ───
model = LogisticRegression(learning_rate=0.1, n_iterations=1000)
model.fit(X, y)

# ─── Evaluate ───
train_acc = model.accuracy(X, y)
print(f"\nFinal Training Accuracy: {train_acc:.2%}")
print(f"Learned weights: w₁={model.weights[0]:.4f}, w₂={model.weights[1]:.4f}")
print(f"Learned bias: b={model.bias:.4f}")

4c. Plotting the Loss Curve

Python# ─── Plot 1: Loss Curve ───
plt.figure(figsize=(10, 4))

plt.subplot(1, 2, 1)
plt.plot(model.loss_history, color='#7c3aed', linewidth=2)
plt.xlabel('Iteration')
plt.ylabel('Binary Cross-Entropy Loss')
plt.title('Training Loss Curve')
plt.grid(True, alpha=0.3)

# ─── Plot 2: Decision Boundary ───
plt.subplot(1, 2, 2)

# Create mesh grid
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                      np.linspace(y_min, y_max, 200))
grid = np.c_[xx.ravel(), yy.ravel()]
probs = model.predict_proba(grid).reshape(xx.shape)

# Plot decision regions
plt.contourf(xx, yy, probs, levels=50, cmap='RdYlGn', alpha=0.6)
plt.contour(xx, yy, probs, levels=[0.5], colors='#7c3aed', linewidths=2)

# Plot data points
plt.scatter(X[y==0, 0], X[y==0, 1], c='#ef4444', label='Default',
            edgecolors='white', s=40)
plt.scatter(X[y==1, 0], X[y==1, 1], c='#22c55e', label='Repaid',
            edgecolors='white', s=40)
plt.xlabel('Monthly Income (normalized)')
plt.ylabel('CIBIL Score (normalized)')
plt.title('Decision Boundary — Loan Default Prediction')
plt.legend()
plt.tight_layout()
plt.savefig('loan_logistic_regression.png', dpi=150)
plt.show()

4d. Vectorized vs Non-Vectorized: Speed Comparison

Pythonimport time

# ─── Non-vectorized (loop-based) gradient computation ───
def compute_gradients_loop(X, y, w, b):
    """Compute gradients using explicit Python loops — SLOW."""
    m, n = X.shape
    dw = np.zeros(n)
    db = 0.0
    
    for i in range(m):
        # Forward pass for sample i
        z_i = 0.0
        for j in range(n):
            z_i += w[j] * X[i, j]
        z_i += b
        a_i = 1 / (1 + np.exp(-z_i))
        
        # Backward pass for sample i
        dz_i = a_i - y[i]
        for j in range(n):
            dw[j] += X[i, j] * dz_i
        db += dz_i
    
    dw /= m
    db /= m
    return dw, db

# ─── Vectorized gradient computation ───
def compute_gradients_vectorized(X, y, w, b):
    """Compute gradients using NumPy vectorization — FAST."""
    m = X.shape[0]
    z = np.dot(X, w) + b
    a = 1 / (1 + np.exp(-z))
    dz = a - y
    dw = (1 / m) * np.dot(X.T, dz)
    db = (1 / m) * np.sum(dz)
    return dw, db

# ─── Benchmark ───
X_big = np.random.randn(10000, 20)  # 10K samples, 20 features
y_big = np.random.randint(0, 2, 10000)
w_test = np.random.randn(20)
b_test = 0.0

# Time the loop version
start = time.time()
dw_loop, db_loop = compute_gradients_loop(X_big, y_big, w_test, b_test)
time_loop = time.time() - start

# Time the vectorized version
start = time.time()
for _ in range(100):  # Run 100x since it's too fast for 1 run
    dw_vec, db_vec = compute_gradients_vectorized(X_big, y_big, w_test, b_test)
time_vec = (time.time() - start) / 100

print(f"Loop version:       {time_loop:.4f}s")
print(f"Vectorized version: {time_vec:.6f}s")
print(f"Speedup:            {time_loop/time_vec:.0f}x faster!")
print(f"\nResults match: {np.allclose(dw_loop, dw_vec)}")

Industry Code — Scikit-Learn Implementation

In production, you'd use scikit-learn's highly optimized LogisticRegression. Let's compare our from-scratch version with the industry standard.

Pythonfrom sklearn.linear_model import LogisticRegression as SklearnLR
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report
from sklearn.preprocessing import StandardScaler

# ─── Prepare data (same synthetic dataset from Section 4) ───
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Scale features (critical for convergence!)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# ─── Scikit-learn model ───
sk_model = SklearnLR(
    solver='lbfgs',        # Quasi-Newton optimizer (faster than GD)
    max_iter=1000,
    C=1.0,               # Inverse regularization strength
    random_state=42
)
sk_model.fit(X_train_scaled, y_train)

# ─── Evaluate ───
y_pred = sk_model.predict(X_test_scaled)
print("=== Scikit-Learn LogisticRegression ===")
print(f"Test Accuracy: {accuracy_score(y_test, y_pred):.2%}")
print(f"Weights: {sk_model.coef_[0]}")
print(f"Bias: {sk_model.intercept_[0]:.4f}")
print()
print(classification_report(y_test, y_pred, 
      target_names=['Default', 'Repaid']))

Visual Diagrams

6a. The Sigmoid Function — Shape and Key Points

6b. Loss Landscape — Why Cross-Entropy Is Convex

6c. Full Logistic Regression Pipeline

Worked Example — Full Forward & Backward Pass by Hand

Let's trace through one complete iteration of logistic regression with 2 features and 3 samples. No calculator shortcuts — we compute everything step by step.

Step 1: Forward Pass — Compute Predictions

Sample 1: x = [0.5, 0.8], y = 1

z⁽¹⁾ = w₁·x₁ + w₂·x₂ + b = (0.0)(0.5) + (0.0)(0.8) + 0.0 = 0.0

a⁽¹⁾ = σ(0.0) = 1/(1 + e⁰) = 1/2 = 0.5

Sample 2: x = [−0.3, −0.5], y = 0

z⁽²⁾ = (0.0)(−0.3) + (0.0)(−0.5) + 0.0 = 0.0

a⁽²⁾ = σ(0.0) = 0.5

Sample 3: x = [0.2, 0.1], y = 1

z⁽³⁾ = (0.0)(0.2) + (0.0)(0.1) + 0.0 = 0.0

a⁽³⁾ = σ(0.0) = 0.5

Step 2: Compute Loss

J = −(1/3) × [y⁽¹⁾ log(a⁽¹⁾) + (1−y⁽¹⁾) log(1−a⁽¹⁾) + y⁽²⁾ log(a⁽²⁾) + (1−y⁽²⁾) log(1−a⁽²⁾) + y⁽³⁾ log(a⁽³⁾) + (1−y⁽³⁾) log(1−a⁽³⁾)]

= −(1/3) × [(1)log(0.5) + (0)log(0.5) + (0)log(0.5) + (1)log(0.5) + (1)log(0.5) + (0)log(0.5)]

= −(1/3) × [log(0.5) + log(0.5) + log(0.5)]

= −(1/3) × 3 × (−0.6931) = 0.6931

Step 3: Backward Pass — Compute Gradients

Compute dz for each sample: dz⁽ⁱ⁾ = a⁽ⁱ⁾ − y⁽ⁱ⁾

dz⁽¹⁾ = 0.5 − 1 = −0.5  (model under-predicted for this positive sample)

dz⁽²⁾ = 0.5 − 0 = +0.5  (model over-predicted for this negative sample)

dz⁽³⁾ = 0.5 − 1 = −0.5  (model under-predicted for this positive sample)

Compute dw₁ = (1/3) Σ dz⁽ⁱ⁾ · x₁⁽ⁱ⁾

dw₁ = (1/3) × [(−0.5)(0.5) + (0.5)(−0.3) + (−0.5)(0.2)]

= (1/3) × [−0.25 + (−0.15) + (−0.10)]

= (1/3) × (−0.50) = −0.1667

Compute dw₂ = (1/3) Σ dz⁽ⁱ⁾ · x₂⁽ⁱ⁾

dw₂ = (1/3) × [(−0.5)(0.8) + (0.5)(−0.5) + (−0.5)(0.1)]

= (1/3) × [−0.40 + (−0.25) + (−0.05)]

= (1/3) × (−0.70) = −0.2333

Compute db = (1/3) Σ dz⁽ⁱ⁾

db = (1/3) × [(−0.5) + (0.5) + (−0.5)]

= (1/3) × (−0.5) = −0.1667

Step 4: Update Parameters

w₁ ← w₁ − α · dw₁ = 0.0 − 0.1 × (−0.1667) = +0.0167

w₂ ← w₂ − α · dw₂ = 0.0 − 0.1 × (−0.2333) = +0.0233

b  ← b  − α · db  = 0.0 − 0.1 × (−0.1667) = +0.0167

Step 5: Verify — Forward Pass with Updated Parameters

Sample 1: z = 0.0167(0.5) + 0.0233(0.8) + 0.0167 = 0.0437 → a = σ(0.0437) ≈ 0.5109 (↑ from 0.5, closer to y=1 ✓)

Sample 2: z = 0.0167(−0.3) + 0.0233(−0.5) + 0.0167 = −0.0049 → a = σ(−0.0049) ≈ 0.4988 (↓ from 0.5, closer to y=0 ✓)

Sample 3: z = 0.0167(0.2) + 0.0233(0.1) + 0.0167 = 0.0224 → a = σ(0.0224) ≈ 0.5056 (↑ from 0.5, closer to y=1 ✓)

Case Study — CIBIL Score-Based Loan Approval at SBI

Common Misconceptions — What Students Get Wrong

Comparison Table — Logistic Regression in Context

10a. Logistic Regression vs Other Classifiers

10b. Loss Functions for Classification

10c. Linear Regression vs Logistic Regression

Exercises

Section A — Multiple Choice Questions (10)

Hover over each question to reveal the answer.

Section B — Short Answer Questions (5)

Section C — Long Answer Questions (3)

Section D — Programming Exercises (3)

Section E — Mini-Project

Chapter Summary

What's Next?

In Chapter 6, we'll extend logistic regression to handle multiple classes (Softmax Regression) and then stack multiple logistic units into layers — building our first true neural network. The sigmoid, cross-entropy, computation graph, and gradient descent you learned here will be the foundation for everything that follows.

References & Further Reading

Textbooks

1. Bishop, C. M. (2006). Pattern Recognition and Machine Learning, Chapter 4.3: Logistic Regression. Springer.
2. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning, Chapter 5.5: Maximum Likelihood Estimation; Chapter 6.2.2: Sigmoid Units. MIT Press. deeplearningbook.org
3. Murphy, K. P. (2022). Probabilistic Machine Learning: An Introduction, Chapter 10: Logistic Regression. MIT Press.
4. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning, Chapter 4.4. Springer. Free PDF

Online Courses

1. Ng, A. (2017). Neural Networks and Deep Learning (Course 1, Week 2: Logistic Regression as a Neural Network). Coursera / deeplearning.ai. Link
2. Stanford CS229: Machine Learning — Lecture Notes on GLMs and Logistic Regression. PDF

Indian Industry & Regulatory

1. TransUnion CIBIL (2024). CIBIL Score: How It's Calculated. cibil.com
2. Reserve Bank of India (2023). Guidelines on Digital Lending. RBI Circular. rbi.org.in
3. Bajaj Finance Annual Report 2023–24. Risk Management: Model Governance Framework.
4. SBI Annual Report 2023–24. Credit Risk Management Using Statistical Models.

Research Papers

1. Cox, D. R. (1958). "The regression analysis of binary sequences." Journal of the Royal Statistical Society, Series B, 20(2), 215–242. [The original logistic regression paper]
2. Platt, J. (1999). "Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods." Advances in Large Margin Classifiers. [Platt Scaling for calibration]

Implementation References

1. NumPy Documentation: numpy.org
2. Scikit-learn LogisticRegression: API Docs