Chapter 6: Backpropagation

Bloom's Taxonomy Map for This Chapter

Learning Objectives

By the end of this chapter, you will be able to:

• Draw the computation graph for logistic regression and a 2-layer neural network, labeling every node and edge
• Execute a complete forward pass through a multi-layer network, computing activations layer by layer
• Apply the chain rule of calculus to compute gradients node-by-node in a backward pass
• Derive the four key backpropagation equations for a 2-layer network, step by numbered step
• Generalize the backpropagation equations to an L-layer network with arbitrary depth
• Implement the general backprop algorithm: cache forward values, then propagate gradients backward
• Verify analytical gradients using numerical gradient checking with finite differences
• Analyze the computational complexity of backprop and explain why it costs O(W) — the same order as the forward pass
• Debug common backprop implementation bugs: shape mismatches, missing cache, wrong derivatives
• Connect backpropagation to automatic differentiation engines in PyTorch and TensorFlow

Opening Hook — The 4-Page Paper That Changed Everything

The Intuition First — Blame Assignment in a Factory

The Factory Analogy

Imagine a chocolate factory with three departments arranged in a pipeline:

1. Department A (Raw Materials): Mixes cocoa, sugar, and milk
2. Department B (Processing): Heats, tempers, and molds the mixture
3. Department C (Quality Control): Tests and packages the final product

A customer complains: "The chocolate tastes too bitter!" The factory manager asks: "Who's responsible?"

The answer is: everyone in the chain. But how much blame goes to each department?

• If Department C didn't mess up the packaging → 100% blame passes back to B
• If Department B properly heated what it received → blame passes further back to A
• If Department A used too little sugar → that's the root cause

But the manager doesn't need to check every possible root cause independently. She traces the blame backward through the chain: final output → C → B → A. At each step, she asks: "How sensitive was your output to your input?" and multiplies the blame. This is exactly the chain rule applied backward through a computation graph.

The "Aha!" Question

Here's the question that should be bugging you: if a network has 100 million parameters, doesn't computing 100 million partial derivatives require 100 million passes through the network?

The answer — and this is the magic of backpropagation — is no. You need exactly one forward pass + one backward pass. Total cost: 2× the forward pass. This is the single most important algorithmic insight in deep learning, and we'll prove it in this chapter.

Mathematical Foundation I — Computation Graphs

4.1 What Is a Computation Graph?

A computation graph is a directed acyclic graph (DAG) where:

• Leaf nodes = inputs (features x, parameters w, b)
• Interior nodes = operations (multiply, add, sigmoid, log, etc.)
• Edges = data flowing from one operation to the next
• Final node = the loss value L

The computation graph makes two things explicit: (1) the order of operations in the forward pass, and (2) the path along which gradients flow in the backward pass.

4.2 Computation Graph for Logistic Regression

Let's draw the computation graph for a single training example with 2 features. The logistic regression model computes: ŷ = σ(w₁x₁ + w₂x₂ + b), then L = −[y log(ŷ) + (1−y) log(1−ŷ)].

4.3 Computation Graph for a 2-Layer Neural Network

Now let's scale up. A 2-layer network with n_h hidden units computes:

1. Layer 1: z⁽¹⁾ = W⁽¹⁾x + b⁽¹⁾,   a⁽¹⁾ = g(z⁽¹⁾)   [hidden layer]
2. Layer 2: z⁽²⁾ = W⁽²⁾a⁽¹⁾ + b⁽²⁾,   a⁽²⁾ = σ(z⁽²⁾)   [output layer]
3. Loss: L = −[y log(a⁽²⁾) + (1−y) log(1−a⁽²⁾)]

Mathematical Foundation II — The Forward Pass

5.1 Forward Pass: From Inputs to Loss

The forward pass is a sequence of elementary operations that transforms inputs into a scalar loss value. For a 2-layer network with a single training example:

5.2 Vectorized Forward Pass (m Samples)

For m training examples stacked as columns X ∈ ℝ^n_x×m:

The cost function over all m examples:

Mathematical Foundation III — The Backward Pass (Chain Rule)

6.1 The Chain Rule — Quick Review

If y = f(u) and u = g(x), then:

For a composition of n functions, y = f₁(f₂(...fₙ(x)...)):

6.2 Chain Rule on the Computation Graph

Here's the key idea: at each node in the computation graph, you only need to know two things:

1. The upstream gradient: how much the loss changes when this node's output changes (this comes from the node's children in the backward direction)
2. The local gradient: how much this node's output changes when its input changes (this is computed from the operation at this node)

Example: A Simple 3-Node Chain

Let's trace gradients through: x → [f] → u → [g] → y → [h] → L

6.3 Local Gradients for Common Operations

Before we tackle a full network, let's catalog the local gradients you'll need:

The Main Event — Backprop for a 2-Layer Network (Full Derivation)

This is the most important section in this chapter. We will derive every gradient, step by numbered step, for a 2-layer neural network. No steps will be skipped. If you're confused at any point, you're thinking correctly — go through it slowly.

7.1 Setup and Notation

7.2 The Full Derivation (Vectorized over m Samples)

7.3 The Four Backprop Equations (Summary)

Generalizing to L Layers — The General Backpropagation Formulas

8.1 L-Layer Forward Pass

For an L-layer network, the forward pass at layer ℓ (for ℓ = 1, 2, ..., L):

8.2 L-Layer Backward Pass

The backward pass starts at layer L and goes backward to layer 1:

8.3 Shape Verification Table

One of the most common bugs in backprop is shape mismatch. Use this table to verify:

Numerical Gradient Checking — Trust, But Verify

9.1 Why Gradient Checking?

Backpropagation is an algorithm that involves dozens of matrix operations with specific shapes, transposes, and element-wise multiplications. A single bug — a misplaced transpose, a forgotten factor of (1/m), a wrong activation derivative — will make your gradients wrong, and your network will silently fail to learn. Gradient checking uses an independent method (numerical differentiation) to verify your analytical gradients are correct.

9.2 The Two-Sided Finite Difference

For any function f(θ), we can approximate the derivative numerically:

9.3 The Gradient Checking Algorithm

Computational Complexity — Why Backprop is a Miracle

10.1 The Naive Approach: O(W²)

Without backpropagation, computing ∂J/∂wᵢ for each of W weights would require a separate forward pass (for numerical differentiation). That's O(W) work per gradient × W parameters = O(W²) total work.

10.2 Backpropagation: O(W)

Backpropagation computes ALL gradients in a single backward pass. Let's count the operations:

10.3 Memory Complexity

The memory cost of backpropagation is the cost of caching all forward pass activations:

Worked Examples

Example 1: By-Hand Chain Rule (Simple Graph)

Example 2: Backprop Through a Neuron (By Hand)

Example 3: Industry — Paytm Fraud Detection (2-Layer Network)

Example 4: Industry — Tesla Autopilot CNN

Python Implementation — From Scratch (NumPy)

12.1 Complete 2-Layer Neural Network with Backpropagation

Python (NumPy)
import numpy as np

# ─── Helper Functions ───
def sigmoid(z):
    """Numerically stable sigmoid."""
    return np.where(z >= 0,
                     1 / (1 + np.exp(-z)),
                     np.exp(z) / (1 + np.exp(z)))

def relu(z):
    return np.maximum(0, z)

def relu_derivative(z):
    return (z > 0).astype(np.float64)

# ─── Initialize Parameters ───
def initialize_parameters(n_x, n_h, n_y):
    """He initialization for ReLU layers, Xavier for output."""
    np.random.seed(42)
    W1 = np.random.randn(n_h, n_x) * np.sqrt(2.0 / n_x)   # He init
    b1 = np.zeros((n_h, 1))
    W2 = np.random.randn(n_y, n_h) * np.sqrt(1.0 / n_h)   # Xavier init
    b2 = np.zeros((n_y, 1))
    return {'W1': W1, 'b1': b1, 'W2': W2, 'b2': b2}

# ─── Forward Pass ───
def forward_pass(X, params):
    """
    Forward propagation for 2-layer network.
    Returns: A2 (predictions), cache (for backward pass)
    """
    W1, b1 = params['W1'], params['b1']
    W2, b2 = params['W2'], params['b2']

    # Layer 1: Linear → ReLU
    Z1 = W1 @ X + b1           # (n_h, m)
    A1 = relu(Z1)              # (n_h, m)

    # Layer 2: Linear → Sigmoid
    Z2 = W2 @ A1 + b2          # (n_y, m)
    A2 = sigmoid(Z2)           # (n_y, m)

    cache = {'Z1': Z1, 'A1': A1, 'Z2': Z2, 'A2': A2}
    return A2, cache

# ─── Compute Cost ───
def compute_cost(A2, Y):
    """Binary cross-entropy loss."""
    m = Y.shape[1]
    # Clip to avoid log(0)
    A2_clipped = np.clip(A2, 1e-8, 1 - 1e-8)
    cost = -(1/m) * np.sum(Y * np.log(A2_clipped) +
                            (1 - Y) * np.log(1 - A2_clipped))
    return float(np.squeeze(cost))

# ─── Backward Pass ───
def backward_pass(X, Y, params, cache):
    """
    Backpropagation for 2-layer network.
    Returns: gradients dict {dW1, db1, dW2, db2}
    """
    m = X.shape[1]
    W2 = params['W2']
    A1, A2, Z1 = cache['A1'], cache['A2'], cache['Z1']

    # ── Output Layer (Layer 2) ──
    dZ2 = A2 - Y                              # (n_y, m) — Eq 1
    dW2 = (1/m) * dZ2 @ A1.T                  # (n_y, n_h) — Eq 2
    db2 = (1/m) * np.sum(dZ2, axis=1, keepdims=True)  # (n_y, 1)

    # ── Hidden Layer (Layer 1) ──
    dA1 = W2.T @ dZ2                          # (n_h, m)
    dZ1 = dA1 * relu_derivative(Z1)           # (n_h, m) — Eq 3
    dW1 = (1/m) * dZ1 @ X.T                   # (n_h, n_x) — Eq 4
    db1 = (1/m) * np.sum(dZ1, axis=1, keepdims=True)  # (n_h, 1)

    return {'dW1': dW1, 'db1': db1, 'dW2': dW2, 'db2': db2}

# ─── Update Parameters ───
def update_parameters(params, grads, learning_rate):
    for key in ['W1', 'b1', 'W2', 'b2']:
        params[key] -= learning_rate * grads['d' + key]
    return params

# ─── Training Loop ───
def train(X, Y, n_h=4, learning_rate=0.01, epochs=10000, print_every=1000):
    n_x = X.shape[0]
    n_y = Y.shape[0]
    params = initialize_parameters(n_x, n_h, n_y)

    for i in range(epochs):
        # Forward
        A2, cache = forward_pass(X, params)
        cost = compute_cost(A2, Y)

        # Backward
        grads = backward_pass(X, Y, params, cache)

        # Update
        params = update_parameters(params, grads, learning_rate)

        if i % print_every == 0:
            print(f"Epoch {i:5d} | Cost: {cost:.6f}")

    return params

# ─── Demo: XOR Problem ───
X = np.array([[0,0,1,1],
              [0,1,0,1]])   # (2, 4)
Y = np.array([[0,1,1,0]])       # (1, 4) — XOR labels

params = train(X, Y, n_h=4, learning_rate=1.0, epochs=10000)
A2, _ = forward_pass(X, params)
print(f"\nPredictions: {np.round(A2, 3)}")
print(f"Truth:       {Y}")

12.2 Numerical Gradient Checking

Python (NumPy)
def gradient_check(X, Y, params, epsilon=1e-7):
    """
    Verify backprop gradients using two-sided numerical differences.
    """
    # Get analytical gradients
    A2, cache = forward_pass(X, params)
    grads = backward_pass(X, Y, params, cache)

    # Check each parameter
    for key in ['W1', 'b1', 'W2', 'b2']:
        param = params[key]
        dparam = grads['d' + key]
        numerical_grad = np.zeros_like(param)

        # Iterate over every element
        it = np.nditer(param, flags=['multi_index'])
        while not it.finished:
            idx = it.multi_index
            old_val = param[idx]

            # f(θ + ε)
            param[idx] = old_val + epsilon
            A2_plus, _ = forward_pass(X, params)
            cost_plus = compute_cost(A2_plus, Y)

            # f(θ - ε)
            param[idx] = old_val - epsilon
            A2_minus, _ = forward_pass(X, params)
            cost_minus = compute_cost(A2_minus, Y)

            # Numerical gradient
            numerical_grad[idx] = (cost_plus - cost_minus) / (2 * epsilon)

            # Restore
            param[idx] = old_val
            it.iternext()

        # Relative difference
        diff = np.linalg.norm(dparam - numerical_grad) / \
               (np.linalg.norm(dparam) + np.linalg.norm(numerical_grad) + 1e-8)
        status = "✅ PASS" if diff < 1e-5 else "❌ FAIL"
        print(f"  {key:3s}: relative diff = {diff:.2e}  {status}")

# Run gradient check on a small network
params_test = initialize_parameters(2, 3, 1)
print("Gradient Check:")
gradient_check(X, Y, params_test)

Python Implementation — PyTorch (Library Version)

13.1 Same Network in PyTorch (autograd does backprop for you)

Python (PyTorch)
import torch
import torch.nn as nn

# ─── Define the 2-Layer Network ───
class TwoLayerNet(nn.Module):
    def __init__(self, n_x, n_h, n_y):
        super().__init__()
        self.layer1 = nn.Linear(n_x, n_h)   # W1, b1
        self.layer2 = nn.Linear(n_h, n_y)   # W2, b2
        self.relu = nn.ReLU()
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        z1 = self.layer1(x)        # Linear: W1·x + b1
        a1 = self.relu(z1)         # ReLU activation
        z2 = self.layer2(a1)       # Linear: W2·a1 + b2
        a2 = self.sigmoid(z2)      # Sigmoid activation
        return a2                  # PyTorch caches everything for backward!

# ─── Prepare Data (XOR) ───
X = torch.tensor([[0,0],[0,1],[1,0],[1,1]], dtype=torch.float32)
Y = torch.tensor([[0],[1],[1],[0]], dtype=torch.float32)

# ─── Train ───
model = TwoLayerNet(n_x=2, n_h=4, n_y=1)
criterion = nn.BCELoss()                   # Binary Cross-Entropy
optimizer = torch.optim.SGD(model.parameters(), lr=1.0)

for epoch in range(10000):
    # Forward pass
    predictions = model(X)                 # calls model.forward(X)
    loss = criterion(predictions, Y)       # compute loss

    # Backward pass — ONE LINE!
    optimizer.zero_grad()                  # clear old gradients
    loss.backward()                        # ← THIS IS BACKPROPAGATION!
    optimizer.step()                       # update parameters

    if epoch % 2000 == 0:
        print(f"Epoch {epoch:5d} | Loss: {loss.item():.6f}")

# ─── Test ───
with torch.no_grad():
    preds = model(X)
    print(f"\nPredictions: {preds.T.numpy().round(3)}")
    print(f"Truth:       {Y.T.numpy()}")

13.2 Inspecting PyTorch's Autograd (What loss.backward() Actually Does)

Python (PyTorch)
# After calling loss.backward(), gradients are stored in .grad attributes
for name, param in model.named_parameters():
    print(f"{name:15s} | param shape: {str(param.shape):12s} | "
          f"grad shape: {str(param.grad.shape):12s} | "
          f"grad norm: {param.grad.norm().item():.4f}")

# Output:
# layer1.weight  | param shape: torch.Size([4, 2]) | grad shape: torch.Size([4, 2]) | grad norm: 0.0021
# layer1.bias    | param shape: torch.Size([4])     | grad shape: torch.Size([4])     | grad norm: 0.0012
# layer2.weight  | param shape: torch.Size([1, 4]) | grad shape: torch.Size([1, 4]) | grad norm: 0.0008
# layer2.bias    | param shape: torch.Size([1])     | grad shape: torch.Size([1])     | grad norm: 0.0004

# KEY INSIGHT: param.shape == param.grad.shape — ALWAYS!

Visual Aids — Seeing the Gradient Flow

14.1 Complete Forward + Backward Flow (2-Layer Network)

14.2 Gradient Flow Through Different Activations

14.3 Computation Graph: Node-by-Node Gradient Propagation

Common Misconceptions

GATE / Exam Corner

Key Formulas for Quick Revision

GATE-Style Problems

Interview Prep

17.1 Conceptual Questions

17.2 Coding Question

def softmax(z):
    exp_z = np.exp(z - np.max(z, axis=0, keepdims=True))
    return exp_z / np.sum(exp_z, axis=0, keepdims=True)

def softmax_cross_entropy_backward(z, y_onehot):
    """
    Combined backward pass for softmax + CE loss.
    z: (C, m) logits, y_onehot: (C, m) one-hot labels
    Returns: dz (C, m) — gradient w.r.t. logits
    """
    m = z.shape[1]
    a = softmax(z)          # (C, m)
    dz = (a - y_onehot) / m # The beautiful simplification!
    return dz

17.3 Case Study Question

Hands-On Lab / Mini-Project

Lab: Build a Modular Backpropagation Framework

def linear_forward(A_prev, W, b):
    Z = W @ A_prev + b
    cache = (A_prev, W, b)
    return Z, cache

def linear_backward(dZ, cache):
    A_prev, W, b = cache
    m = A_prev.shape[1]
    dW = (1/m) * dZ @ A_prev.T
    db = (1/m) * np.sum(dZ, axis=1, keepdims=True)
    dA_prev = W.T @ dZ
    return dA_prev, dW, db

Debug This!

def buggy_backward(X, Y, params, cache):
    m = X.shape[0]                              # 🐛 Bug 1: ???
    W2 = params['W2']
    A1, A2, Z1 = cache['A1'], cache['A2'], cache['Z1']

    # Output layer
    dZ2 = A2 - Y
    dW2 = (1/m) * dZ2 @ A1                     # 🐛 Bug 2: ???
    db2 = (1/m) * np.sum(dZ2, axis=1, keepdims=True)

    # Hidden layer
    dA1 = W2.T @ dZ2
    dZ1 = dA1 * (Z1 > 0)                       # Assumes ReLU — this is correct
    dW1 = (1/m) * dZ1 @ X.T
    db1 = (1/m) * np.sum(dZ1, axis=0, keepdims=True)  # 🐛 Bug 3: ???

    return {'dW1': dW1, 'db1': db1, 'dW2': dW2, 'db2': db2}

Exercises (22 Questions)

def leaky_relu_backward(dA, Z, alpha=0.01):
    dZ = np.where(Z > 0, dA, alpha * dA)
    return dZ

def check_shapes(params, grads):
    for key in params:
        grad_key = 'd' + key
        assert grads[grad_key].shape == params[key].shape, \
            f"Shape mismatch: {grad_key} has shape {grads[grad_key].shape} " \
            f"but {key} has shape {params[key].shape}"
    print("All gradient shapes verified! ✅")

Connections

Chapter Summary

Further Reading

🇮🇳 Indian Resources

• NPTEL: "Deep Learning" by Prof. Mitesh Khapra (IIT Madras) — Weeks 3-4 cover backpropagation with excellent visualizations
• NPTEL: "Machine Learning" by Prof. Sudeshna Sarkar (IIT Kharagpur) — Lecture 18-20 on neural network training
• GATE PYQs: GATE CS 2018-2024 papers, ML section — chain rule and gradient problems appear every year
• Book: "Deep Learning" by S. Haykin — comprehensive treatment of backprop with signal processing perspective

🌍 Global Resources

• Original paper: Rumelhart, Hinton, Williams — "Learning representations by back-propagating errors" (Nature, 1986)
• 3Blue1Brown: "Backpropagation calculus" — the most intuitive visual explanation (YouTube, 2017)
• Andrej Karpathy: "micrograd" — a tiny autograd engine in Python, perfect for understanding backprop (GitHub)
• CS231n (Stanford): Lecture 4 — "Backpropagation and Neural Networks" with fantastic computation graph visualizations
• Distill.pub: "Automatic Differentiation" (not published yet but planned) — interactive article
• Survey: Baydin et al. — "Automatic Differentiation in Machine Learning: a Survey" (JMLR 2018)
• Book: Goodfellow, Bengio, Courville — "Deep Learning" Chapter 6.5: Back-Propagation and Other Differentiation Algorithms
• Blog: Chris Olah — "Calculus on Computational Graphs: Backpropagation" (colah.github.io, 2015)