Chapter 7: One Hidden Layer Neural Networks

Bloom's Taxonomy Map for This Chapter

Learning Objectives

By the end of this chapter, you will be able to:

• Draw the complete architecture of a 2-layer neural network with proper layer notation: input layer (layer 0), hidden layer (layer 1), output layer (layer 2)
• Write the four forward propagation equations — Z^[1], A^[1], Z^[2], A^[2] — for a single example, then vectorize them over m training examples
• State the Universal Approximation Theorem and explain intuitively why step functions can approximate any continuous function
• Prove that using linear activations in hidden layers collapses the entire network to a single linear transformation, making depth useless
• Derive the complete backpropagation equations for a 2-layer network from first principles using chain rule, connecting to the gradient descent foundations from Chapter 6
• Demonstrate the symmetry-breaking problem: prove that if W = 0, all hidden neurons remain identical forever
• Implement a full NeuralNetwork class from scratch in NumPy with configurable hidden units, activation functions, and learning rate
• Solve XOR with your network and visualize how the decision boundary evolves during training
• Apply shallow networks to real problems: Zomato food classification (India) and Netflix genre prediction (US/Global)

Opening Hook — The XOR Resurrection

The Intuition First — Think Before You Compute

The Kitchen Analogy

Imagine you're running a dabbawala kitchen in Mumbai. You have raw ingredients coming in (rice, dal, sabzi, roti) — these are your inputs. A single chef (single neuron) can make exactly one dish — say, a linear combination of ingredients. That chef can distinguish "veg thali" from "non-veg thali" if the distinction is simple. But what if you need to classify dishes as "North Indian," "South Indian," or "fusion"? One chef can't handle that — the categories aren't linearly separable.

Now add a team of specialized sous-chefs (hidden neurons). One detects spice levels. Another detects coconut-based gravies. A third detects wheat-vs-rice base. Each sous-chef transforms the raw ingredients into a higher-level feature. The head chef (output neuron) then looks at these high-level features — "spicy + coconut + rice" → South Indian! — and makes the final call.

That team of sous-chefs is your hidden layer. They don't output the final answer. They create an intermediate representation — a new, more useful way to describe the data.

The "Aha" Question

Mathematical Foundation — Neural Network Representation

4.1 Architecture: Input → Hidden → Output

A one-hidden-layer neural network (also called a "2-layer neural network" because we count layers with learnable weights) has three layers of nodes but two layers of parameters:

4.2 Notation — The Complete System

Let's be precise about every symbol. This notation from Andrew Ng's framework carries through the entire book:

4.3 What Each Layer Learns

Forward Propagation — Computing the Prediction

5.1 For a Single Training Example

Given one input vector x ∈ ℝ^n_x×1, forward propagation computes the output in exactly four steps:

Let's trace through this with a concrete example. Suppose n_x=2, n^[1]=2, n^[2]=1, using tanh for the hidden layer and sigmoid for the output:

5.2 Vectorized Over m Examples

In practice, you never loop over training examples one by one. You stack all m examples as columns of a matrix X ∈ ℝ^n_x×m:

Key insight: The equations look identical to the single-example case! The only difference is that lowercase vectors become uppercase matrices. Each column of Z^[1] is z^[1](i) for the i-th example. NumPy's broadcasting handles b^[1] addition automatically.

Why Non-Linear Activations? — The Collapse Proof

6.1 The Question

You might ask: "Why bother with tanh, sigmoid, or ReLU? Why not just use g(z) = z (a linear activation)? It's simpler!"

Let's prove why that destroys your network.

6.2 What Non-Linearity Gives You

Non-linear activations like tanh or ReLU break this collapse. After applying tanh to z^[1], you can no longer "factor out" the hidden layer. The composition of a linear transformation followed by a non-linearity followed by another linear transformation is a genuinely richer function class.

Geometrically, the hidden layer's non-linear activation warps the input space. It bends, folds, and stretches it so that previously non-separable patterns become separable. This is exactly what we need for XOR.

6.3 The One Exception

There's exactly one place where a linear activation is acceptable: the output layer of a regression problem. If you're predicting a continuous value (house price, temperature), the output layer should use g^[2](z) = z. But the hidden layers must still be non-linear.

The Universal Approximation Theorem — Why Neural Networks Work

7.1 Statement of the Theorem

In plain language: Give me any continuous curve, surface, or hyper-surface. I can build a one-hidden-layer network that gets arbitrarily close to it.

7.2 The Intuition — Step Functions Approximate Anything

This is best understood visually. Here's the core idea in three steps:

The key insight: Each pair of hidden neurons creates one "bump." With n^[1] hidden neurons, you get ~n^[1]/2 bumps. More bumps means finer approximation. This is why the theorem says "finite number" — it's finite, but might be very large for complex functions.

7.3 The Caveats You Must Know

• Existence ≠ Learnability: The theorem says a solution exists but doesn't say gradient descent will find it.
• Width can be exponential: Approximating a function of d variables to accuracy ε might require O(ε^-d) neurons — the curse of dimensionality.
• Deep networks are more efficient: Functions that require exponentially many neurons in a shallow network can often be represented with polynomially many neurons in a deep network. This is why depth matters.

Backpropagation for a 2-Layer Network — Full Derivation

This is the heart of the chapter. We will derive every gradient from scratch using the chain rule. If you understood the single-neuron backpropagation from Chapter 6, you already have 70% of this. The new piece is propagating gradients backwards through the hidden layer.

8.1 The Cost Function

For binary classification with m examples:

8.2 The Computation Graph

8.3 Step-by-Step Derivation

8.4 The Pattern You Should See

Notice the beautiful symmetry between the two layers:

The general pattern is: dW[l] = (1/m) dZ[l] · A[l-1]T. This pattern extends to L layers — you'll see this in Chapter 8.

# Buggy backpropagation
dZ2 = A2 - Y
dW2 = dZ2 @ A1.T          # Bug 1 is here
db2 = np.sum(dZ2, axis=0, keepdims=True)  # Bug 2 is here
dZ1 = W2.T @ dZ2 * (1 - np.power(A1, 2))
dW1 = (1/m) * dZ1 @ X.T
db1 = (1/m) * np.sum(dZ1, axis=1, keepdims=True)

Random Initialization — Breaking Symmetry

9.1 The Problem: What Happens If W = 0?

In logistic regression, initializing weights to zero works fine because there's only one neuron. But in a neural network with multiple hidden neurons, zero initialization is catastrophic. Let's prove it.

9.2 The Solution: Random Initialization

Break symmetry by initializing weights randomly:

# Correct initialization
W1 = np.random.randn(n_h, n_x) * 0.01   # Small random values
b1 = np.zeros((n_h, 1))                   # Biases CAN be zero
W2 = np.random.randn(n_y, n_h) * 0.01   # Small random values
b2 = np.zeros((n_y, 1))                   # Biases CAN be zero

9.3 Why Multiply by 0.01?

If you use tanh or sigmoid activations, large initial weights push z into the saturated region of the activation function where the gradient is nearly zero. Training stalls — this is the "vanishing gradient" problem at initialization.

With small weights (×0.01), z stays near zero where tanh and sigmoid have their steepest gradients, so learning happens quickly in early iterations.

XOR Solved! — The Hidden Layer's Triumph

10.1 XOR: The Problem That Killed the Perceptron

Plot these points: (0,0)→0, (1,1)→0 (same class), (0,1)→1, (1,0)→1 (other class). No single straight line can separate them.

10.2 How the Hidden Layer Solves It

The key insight: the hidden layer transforms the input space into a new representation where XOR becomes linearly separable.

10.3 Decision Boundary Evolution During Training

When you train a neural network on XOR, the decision boundary evolves dramatically:

The network starts with a random decision boundary (basically guessing), then gradually learns to carve out the correct non-linear boundary that separates the XOR classes. With 2 hidden neurons, it creates a boundary with exactly 2 "bends" — just enough for XOR.

Worked Examples

Example 1: By-Hand Forward and Backward Pass (XOR)

Example 2: 🇮🇳 Zomato Food Classifier — Hidden Layer Wins

# Zomato food classifier architecture
# Input: [calories, sugar, protein, fiber, fat] → normalized
# Hidden: 4 neurons with tanh activation
# Output: 1 sigmoid neuron → P(healthy)
n_x = 5   # nutrition features
n_h = 4   # hidden neurons
n_y = 1   # binary output

Example 3: 🇺🇸 Netflix Genre Prediction — Latent Preferences

# Netflix genre predictor architecture
# Input: 7 viewing behavior features
# Hidden: 8 neurons with ReLU activation
# Output: sigmoid → P(enjoys_genre)
n_x = 7    # viewing features
n_h = 8    # latent taste dimensions
n_y = 1    # binary preference

Python Implementation — From Scratch + Library

12.1 From-Scratch NumPy: Complete NeuralNetwork Class

Python — NumPy from scratch
import numpy as np

class ShallowNeuralNetwork:
    """One hidden layer neural network for binary classification.
    
    Architecture: n_x → n_h → 1
    Hidden activation: tanh (configurable)
    Output activation: sigmoid
    Loss: binary cross-entropy
    """
    
    def __init__(self, n_x, n_h, learning_rate=0.01):
        """Initialize the network with random weights."""
        self.n_x = n_x    # input features
        self.n_h = n_h    # hidden units
        self.n_y = 1      # output units (binary)
        self.lr = learning_rate
        
        # ═══ Random initialization (symmetry breaking!) ═══
        self.W1 = np.random.randn(n_h, n_x) * 0.01
        self.b1 = np.zeros((n_h, 1))
        self.W2 = np.random.randn(1, n_h) * 0.01
        self.b2 = np.zeros((1, 1))
        
        # Store costs for plotting
        self.costs = []
    
    def sigmoid(self, z):
        """Sigmoid activation: σ(z) = 1/(1+e^(-z))"""
        return 1 / (1 + np.exp(-np.clip(z, -500, 500)))
    
    def tanh(self, z):
        """Tanh activation: already in NumPy"""
        return np.tanh(z)
    
    def forward(self, X):
        """Forward propagation: X → Z1 → A1 → Z2 → A2
        
        Args:
            X: input data, shape (n_x, m)
        Returns:
            A2: predictions, shape (1, m)
        """
        # Layer 1 (hidden)
        self.Z1 = self.W1 @ X + self.b1        # (n_h, m)
        self.A1 = self.tanh(self.Z1)            # (n_h, m)
        
        # Layer 2 (output)
        self.Z2 = self.W2 @ self.A1 + self.b2   # (1, m)
        self.A2 = self.sigmoid(self.Z2)          # (1, m)
        
        return self.A2
    
    def compute_cost(self, A2, Y):
        """Binary cross-entropy cost."""
        m = Y.shape[1]
        cost = -(1/m) * np.sum(
            Y * np.log(A2 + 1e-8) + 
            (1 - Y) * np.log(1 - A2 + 1e-8)
        )
        return np.squeeze(cost)
    
    def backward(self, X, Y):
        """Backpropagation: compute gradients for all parameters.
        
        THE 6 BACKPROP EQUATIONS:
        dZ2 = A2 - Y
        dW2 = (1/m) * dZ2 @ A1.T
        db2 = (1/m) * sum(dZ2)
        dZ1 = W2.T @ dZ2 * (1 - A1^2)   ← tanh derivative
        dW1 = (1/m) * dZ1 @ X.T
        db1 = (1/m) * sum(dZ1)
        """
        m = X.shape[1]
        
        # Output layer gradients
        dZ2 = self.A2 - Y                            # (1, m)
        self.dW2 = (1/m) * dZ2 @ self.A1.T              # (1, n_h)
        self.db2 = (1/m) * np.sum(dZ2, axis=1, keepdims=True)  # (1,1)
        
        # Hidden layer gradients (THE NEW PART!)
        dZ1 = (self.W2.T @ dZ2) * (1 - np.power(self.A1, 2))  # (n_h, m)
        self.dW1 = (1/m) * dZ1 @ X.T                   # (n_h, n_x)
        self.db1 = (1/m) * np.sum(dZ1, axis=1, keepdims=True)  # (n_h,1)
    
    def update_parameters(self):
        """Gradient descent update."""
        self.W1 -= self.lr * self.dW1
        self.b1 -= self.lr * self.db1
        self.W2 -= self.lr * self.dW2
        self.b2 -= self.lr * self.db2
    
    def fit(self, X, Y, iterations=10000, print_cost=True):
        """Train the network using gradient descent.
        
        Args:
            X: training data, shape (n_x, m)
            Y: labels, shape (1, m)
            iterations: number of gradient descent steps
        """
        for i in range(iterations):
            # Forward propagation
            A2 = self.forward(X)
            
            # Compute cost
            cost = self.compute_cost(A2, Y)
            
            # Backward propagation
            self.backward(X, Y)
            
            # Update weights
            self.update_parameters()
            
            # Record cost
            if i % 100 == 0:
                self.costs.append(cost)
                if print_cost and i % 1000 == 0:
                    print(f"Iteration {i}: cost = {cost:.6f}")
    
    def predict(self, X):
        """Make predictions (0 or 1)."""
        A2 = self.forward(X)
        return (A2 > 0.5).astype(int)
    
    def accuracy(self, X, Y):
        """Compute classification accuracy."""
        predictions = self.predict(X)
        return np.mean(predictions == Y) * 100

# ═══════════════════════════════════════════════
# TRAIN ON XOR — THE MOMENT OF TRUTH!
# ═══════════════════════════════════════════════

# XOR dataset
X = np.array([[0, 0, 1, 1],     # x1
              [0, 1, 0, 1]])    # x2
Y = np.array([[0, 1, 1, 0]])    # XOR output

# Create and train the network
nn = ShallowNeuralNetwork(n_x=2, n_h=4, learning_rate=1.0)
nn.fit(X, Y, iterations=10000)

# Test predictions
predictions = nn.predict(X)
print(f"\nPredictions: {predictions}")
print(f"True labels: {Y}")
print(f"Accuracy:    {nn.accuracy(X, Y):.1f}%")

XOR is solved! 🎉 Four hidden neurons with tanh activation, trained with gradient descent for 10,000 iterations, learn the XOR function perfectly.

12.2 Decision Boundary Visualization

Python — Visualization
import matplotlib.pyplot as plt

def plot_decision_boundary(model, X, Y):
    """Plot the decision boundary of a trained model."""
    # Create a mesh grid
    x_min, x_max = X[0, :].min() - 0.5, X[0, :].max() + 0.5
    y_min, y_max = X[1, :].min() - 0.5, X[1, :].max() + 0.5
    xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.01),
                         np.arange(y_min, y_max, 0.01))
    
    # Predict on every point in the grid
    grid_input = np.c_[xx.ravel(), yy.ravel()].T  # (2, n_points)
    Z = model.predict(grid_input)
    Z = Z.reshape(xx.shape)
    
    # Plot
    plt.contourf(xx, yy, Z, alpha=0.4, cmap='RdYlBu')
    plt.scatter(X[0, :], X[1, :], c=Y.ravel(), 
                cmap='RdYlBu', edgecolors='black', s=100)
    plt.xlabel('x₁'); plt.ylabel('x₂')
    plt.title('XOR Decision Boundary')
    plt.show()

plot_decision_boundary(nn, X, Y)

# Plot cost curve
plt.plot(nn.costs)
plt.xlabel('Iterations (×100)'); plt.ylabel('Cost')
plt.title('Training Cost Curve')
plt.show()

12.3 PyTorch Library Version

Python — PyTorch
import torch
import torch.nn as nn

# XOR data as PyTorch tensors
X_torch = torch.tensor([[0,0],[0,1],[1,0],[1,1]], dtype=torch.float32)
Y_torch = torch.tensor([[0],[1],[1],[0]], dtype=torch.float32)

# Define the network
class XORNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(2, 4)   # 2 inputs → 4 hidden
        self.layer2 = nn.Linear(4, 1)   # 4 hidden → 1 output
        self.tanh = nn.Tanh()
        self.sigmoid = nn.Sigmoid()
    
    def forward(self, x):
        x = self.tanh(self.layer1(x))     # hidden layer
        x = self.sigmoid(self.layer2(x))   # output layer
        return x

# Train
model = XORNet()
criterion = nn.BCELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1.0)

for epoch in range(10000):
    output = model(X_torch)
    loss = criterion(output, Y_torch)
    
    optimizer.zero_grad()
    loss.backward()        # PyTorch does backprop for you!
    optimizer.step()
    
    if epoch % 2000 == 0:
        print(f"Epoch {epoch}: loss = {loss.item():.6f}")

# Test
with torch.no_grad():
    preds = (model(X_torch) > 0.5).int()
    print(f"Predictions: {preds.T}")
    print(f"Accuracy: {(preds == Y_torch.int()).float().mean():.0%}")

Visual Aids & Diagrams

13.1 Complete Network Data Flow

13.2 Shape Cheat Sheet

13.3 XOR Feature Space Transformation

Common Misconceptions

GATE / Exam Corner

15.1 Formula Sheet — Pin This to Your Wall

15.2 Previous Year Questions & Predictions

15.3 Exam Prediction Table

Interview Prep

16.1 Conceptual Questions

16.2 Coding Questions

def forward_propagation(X, W1, b1, W2, b2):
    Z1 = W1 @ X + b1
    A1 = np.tanh(Z1)
    Z2 = W2 @ A1 + b2
    A2 = 1 / (1 + np.exp(-Z2))
    cache = (Z1, A1, Z2, A2)
    return A2, cache

# Buggy code
dZ2 = Y - A2                           # Bug 1
dW2 = (1/m) * A1 @ dZ2.T               # Bug 2
db2 = (1/m) * np.sum(dZ2, axis=1, keepdims=True)
dZ1 = W2.T @ dZ2 * (1 + np.power(A1, 2))  # Bug 3
dW1 = (1/m) * dZ1 @ X.T
db1 = (1/m) * np.sum(dZ1, axis=1, keepdims=True)

16.3 Case Study Question

Hands-On Lab / Mini-Project

# Numerical gradient checking
def gradient_check(model, X, Y, epsilon=1e-7):
    """Compare analytical gradients to numerical gradients."""
    # Get analytical gradients
    model.forward(X)
    model.backward(X, Y)
    
    # Check dW1
    for i in range(model.W1.shape[0]):
        for j in range(model.W1.shape[1]):
            # Compute J(W1[i,j] + ε)
            model.W1[i, j] += epsilon
            J_plus = model.compute_cost(model.forward(X), Y)
            
            # Compute J(W1[i,j] - ε)
            model.W1[i, j] -= 2 * epsilon
            J_minus = model.compute_cost(model.forward(X), Y)
            
            # Restore
            model.W1[i, j] += epsilon
            
            # Numerical gradient
            grad_numerical = (J_plus - J_minus) / (2 * epsilon)
            grad_analytical = model.dW1[i, j]
            
            # Compare
            diff = abs(grad_numerical - grad_analytical)
            if diff > 1e-5:
                print(f"❌ W1[{i},{j}]: analytical={grad_analytical:.8f}, "
                      f"numerical={grad_numerical:.8f}, diff={diff:.2e}")
            else:
                print(f"✓ W1[{i},{j}]: diff = {diff:.2e}")

Exercises

Section A — Conceptual Questions (5)

Section B — Mathematical Questions (8)

Section C — Coding Questions (4)

Section D — Critical Thinking (3)

Section ★ — Starred Research Questions (2)

Connections — Where This Chapter Fits

Chapter Summary

Further Reading

Primary Textbooks

• Goodfellow, Bengio & Courville (2016). Deep Learning. MIT Press. Chapter 6 (Deep Feedforward Networks) — the definitive reference for network architecture and universal approximation. Free at deeplearningbook.org.
• Andrew Ng — Coursera Deep Learning Specialization. Course 1, Week 3 — Shallow Neural Networks. The notation in this chapter follows Ng's conventions exactly.
• Michael Nielsen (2015). Neural Networks and Deep Learning. Free online book (neuralnetworksanddeeplearning.com). Chapter 4 has the best visual proof of the Universal Approximation Theorem.

Landmark Papers

• Cybenko, G. (1989). "Approximation by Superpositions of a Sigmoidal Function." Mathematics of Control, Signals, and Systems. — The original Universal Approximation Theorem.
• Hornik, K. (1991). "Approximation Capabilities of Multilayer Feedforward Networks." Neural Networks. — Extended the theorem to arbitrary non-polynomial activations.
• Lu, Z. et al. (2017). "The Expressive Power of Neural Networks: A View from the Width." NeurIPS. — Width-bounded universal approximation for ReLU networks.
• Jacot, A. et al. (2018). "Neural Tangent Kernel." NeurIPS. — Infinite-width networks as kernel methods.
• Minsky, M. & Papert, S. (1969). Perceptrons. MIT Press. — The book that proved XOR is impossible for single-layer networks.

🇮🇳 Indian Resources

• NPTEL Deep Learning (IIT Madras): Prof. Mitesh Khapra's course covers shallow networks in Weeks 4-5 with Hindi/English explanations. Free at nptel.ac.in.
• GATE Previous Year Papers: gate.iitk.ac.in — Search for "neural network" in CS and DA papers from 2018-2024.
• IndiaAI Portal: indiaai.gov.in — Government AI resources, datasets from Indian census, agriculture, and healthcare.
• Padhai (One Fourth Labs): padhai.onefourthlabs.in — Indian deep learning course with affordable pricing, covers shallow networks with Indian industry examples.

🌐 Global Resources

• TensorFlow Playground: playground.tensorflow.org — Build shallow networks interactively. Start with the XOR dataset and 2 hidden neurons!
• 3Blue1Brown — Neural Networks series (YouTube): Grant Sanderson's visual explanations of forward/backward propagation are unmatched.
• Distill.pub: "A Visual Proof That Neural Nets Can Compute Any Function" — the best online visual explanation of the UAT.
• CS231n (Stanford): Lecture 4 covers backpropagation with computation graphs — free on YouTube.