Chapter 6: Shallow Neural Networks

Bloom's Taxonomy Map for This Chapter

Learning Objectives

By the end of this chapter, you will be able to:

• Draw the architecture of a shallow neural network (1 hidden layer) with proper notation for layers, weights, biases, and activations
• Derive the forward propagation equations for a single training example and extend them to vectorized form over the entire dataset
• Prove mathematically that linear hidden activations collapse the entire network into a single linear transformation, making hidden layers useless
• Compare five activation functions — sigmoid, tanh, ReLU, Leaky ReLU, and ELU — with their formulas, derivatives, ranges, and trade-offs
• Derive the complete backpropagation equations for a 2-layer network using chain rule on the computation graph
• Explain the symmetry-breaking problem and why random initialization of weights is essential
• Implement a full NeuralNetwork class from scratch in NumPy that trains on planar data and visualizes non-linear decision boundaries
• Apply shallow neural networks to real-world classification problems in the Indian industry context

Opening Hook — When a Single Neuron Isn't Enough

Core Concepts — Building the Network Layer by Layer

3a. Neural Network Representation & Notation

A shallow neural network (also called a 2-layer neural network) has exactly three layers of nodes, but we count only layers with learnable parameters:

Superscript Notation Convention

3b. Forward Propagation — Single Example & Vectorized

Single Training Example (x⁽ⁱ⁾)

For a single input vector x with shape (n^[0], 1), the forward pass through our 2-layer network computes:

Step-by-step walkthrough:

1. Multiply: W^[1] (4×2) · x (2×1) = z_partial (4×1) — each hidden neuron computes its weighted sum of inputs
2. Add bias: z_partial + b^[1] (4×1) = z^[1] (4×1)
3. Activate: Apply g^[1] (e.g., tanh) element-wise → a^[1] (4×1) — now we have 4 hidden unit outputs
4. Multiply: W^[2] (1×4) · a^[1] (4×1) = z_partial (1×1) — output neuron combines hidden outputs
5. Add bias: z_partial + b^[2] (1×1) = z^[2] (1×1)
6. Activate: Apply g^[2] (sigmoid for binary classification) → a^[2] (1×1) = ŷ ∈ [0, 1]

Vectorized Over m Examples

Instead of looping over m training examples, we stack all examples as columns of a matrix X (shape: n^[0] × m). The vectorized forward propagation becomes:

Here, b[1] (shape: n^[1] × 1) is broadcast across all m columns automatically by NumPy. Each column of A^[2] is the prediction for one training example.

3c. Activation Functions — The Complete Deep Dive

The activation function g(z) is what gives neural networks their power. Without it, stacking layers is pointless (we'll prove this next). Here are the five most important activation functions:

Master Comparison Table

3d. Proof: Linear Activations Collapse the Network

Here's a critical question: what if we use a linear activation function g(z) = z (the identity function) for all layers? Let's prove that the network becomes equivalent to plain linear regression, making hidden layers useless.

3e. Backpropagation for a 2-Layer Network — Full Derivation

Backpropagation is the algorithm that computes gradients of the loss function with respect to every parameter. It uses the chain rule of calculus, working backwards from the output layer to the input layer.

The Cost Function

For binary classification with m examples:

Step 1: Output Layer Gradients (Layer 2)

Starting from the loss and working backwards through the sigmoid output:

Derivation of dZ^[2]:

∂J/∂a^[2] = −y/a^[2] + (1−y)/(1−a^[2])    (derivative of cross-entropy)
∂a^[2]/∂z^[2] = a^[2](1−a^[2])    (derivative of sigmoid)
dz^[2] = ∂J/∂z^[2] = ∂J/∂a^[2] · ∂a^[2]/∂z^[2]
= [−y/a^[2] + (1−y)/(1−a^[2])] · a^[2](1−a^[2])
= −y(1−a^[2]) + (1−y)a^[2]
= a^[2] − y    ✓ (beautifully simple!)

Step 2: Hidden Layer Gradients (Layer 1)

Now we propagate the gradient backwards through W^[2] and the activation function g^[1]:

Derivation of dZ^[1]:

∂J/∂z^[1] = ∂J/∂z^[2] · ∂z^[2]/∂a^[1] · ∂a^[1]/∂z^[1]
= dz^[2] · W^[2] · g^[1]'(z^[1])
Vectorized: dZ^[1] = W^[2]T · dZ^[2] ⊙ g^[1]'(Z^[1])

The ⊙ symbol denotes element-wise multiplication (Hadamard product). The term g^[1]'(Z^[1]) is the derivative of the hidden layer's activation function applied element-wise.

Derivatives for Common Activations

Step 3: Parameter Updates

Where α is the learning rate (a hyperparameter you choose, e.g., 0.01 or 1.2).

3f. Random Initialization — Breaking Symmetry

In logistic regression, we could initialize weights to zero. Can we do the same for neural networks? No! Here's why:

From-Scratch Code — Building a Neural Network in NumPy

Let's build a complete NeuralNetwork class with one hidden layer. We'll train it on a planar XOR dataset — the classic problem that a single neuron (perceptron) cannot solve.

Step 1: Generate Planar XOR Data

Python
import numpy as np
import matplotlib.pyplot as plt

def generate_xor_data(n_samples=400, noise=0.15):
    """Generate planar XOR-like dataset.
    Class 1 (y=1): points in quadrants I and III
    Class 0 (y=0): points in quadrants II and IV
    """
    np.random.seed(42)
    n = n_samples // 4

    # Quadrant I: (+, +) → class 1
    q1 = np.random.randn(n, 2) * 0.5 + np.array([1, 1])
    # Quadrant II: (-, +) → class 0
    q2 = np.random.randn(n, 2) * 0.5 + np.array([-1, 1])
    # Quadrant III: (-, -) → class 1
    q3 = np.random.randn(n, 2) * 0.5 + np.array([-1, -1])
    # Quadrant IV: (+, -) → class 0
    q4 = np.random.randn(n, 2) * 0.5 + np.array([1, -1])

    X = np.vstack([q1, q2, q3, q4]).T  # Shape: (2, 400)
    Y = np.array([[1]*n + [0]*n + [1]*n + [0]*n])  # Shape: (1, 400)

    # Shuffle
    perm = np.random.permutation(X.shape[1])
    X, Y = X[:, perm], Y[:, perm]

    return X, Y

X, Y = generate_xor_data()
print(f"X shape: {X.shape}, Y shape: {Y.shape}")

Step 2: The Complete NeuralNetwork Class

Python
class ShallowNeuralNetwork:
    """
    A 2-layer neural network (1 hidden layer) for binary classification.

    Architecture: Input(n_x) → Hidden(n_h, tanh) → Output(1, sigmoid)
    """

    def __init__(self, n_x, n_h, learning_rate=1.2):
        """
        Parameters:
            n_x : int — number of input features
            n_h : int — number of hidden units
            learning_rate : float — step size for gradient descent
        """
        self.lr = learning_rate

        # Random initialization (small weights, zero biases)
        self.W1 = np.random.randn(n_h, n_x) * 0.01
        self.b1 = np.zeros((n_h, 1))
        self.W2 = np.random.randn(1, n_h) * 0.01
        self.b2 = np.zeros((1, 1))

        print(f"Network initialized: {n_x} → {n_h} → 1")
        print(f"  W1: {self.W1.shape}, b1: {self.b1.shape}")
        print(f"  W2: {self.W2.shape}, b2: {self.b2.shape}")
        total = n_h * n_x + n_h + 1 * n_h + 1
        print(f"  Total parameters: {total}")

    def sigmoid(self, z):
        """Sigmoid activation function."""
        return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

    def forward(self, X):
        """
        Forward propagation.
        X: shape (n_x, m) — m examples
        Returns: A2 (predictions), cache (for backprop)
        """
        # Layer 1: Hidden
        Z1 = self.W1 @ X + self.b1       # (n_h, m)
        A1 = np.tanh(Z1)                   # (n_h, m)

        # Layer 2: Output
        Z2 = self.W2 @ A1 + self.b2      # (1, m)
        A2 = self.sigmoid(Z2)              # (1, m)

        cache = (Z1, A1, Z2, A2)
        return A2, cache

    def compute_cost(self, A2, Y):
        """Binary cross-entropy loss."""
        m = Y.shape[1]
        # Clip to avoid log(0)
        A2 = np.clip(A2, 1e-8, 1 - 1e-8)
        cost = -(1/m) * np.sum(
            Y * np.log(A2) + (1 - Y) * np.log(1 - A2)
        )
        return float(cost)

    def backward(self, X, Y, cache):
        """
        Backward propagation.
        Returns: gradients dict {dW1, db1, dW2, db2}
        """
        m = X.shape[1]
        Z1, A1, Z2, A2 = cache

        # Output layer gradients
        dZ2 = A2 - Y                          # (1, m)
        dW2 = (1/m) * dZ2 @ A1.T               # (1, n_h)
        db2 = (1/m) * np.sum(dZ2, axis=1, keepdims=True)  # (1, 1)

        # Hidden layer gradients
        dZ1 = (self.W2.T @ dZ2) * (1 - A1**2)  # tanh derivative: 1 - tanh²(z)
        dW1 = (1/m) * dZ1 @ X.T                # (n_h, n_x)
        db1 = (1/m) * np.sum(dZ1, axis=1, keepdims=True)  # (n_h, 1)

        return {'dW1': dW1, 'db1': db1,
                'dW2': dW2, 'db2': db2}

    def update_parameters(self, grads):
        """Gradient descent update."""
        self.W1 -= self.lr * grads['dW1']
        self.b1 -= self.lr * grads['db1']
        self.W2 -= self.lr * grads['dW2']
        self.b2 -= self.lr * grads['db2']

    def train(self, X, Y, epochs=10000, print_every=1000):
        """Full training loop."""
        costs = []
        for i in range(epochs):
            # Forward
            A2, cache = self.forward(X)
            # Cost
            cost = self.compute_cost(A2, Y)
            # Backward
            grads = self.backward(X, Y, cache)
            # Update
            self.update_parameters(grads)

            if i % print_every == 0:
                costs.append(cost)
                print(f"Epoch {i:5d} | Cost: {cost:.6f}")

        return costs

    def predict(self, X):
        """Binary predictions (threshold = 0.5)."""
        A2, _ = self.forward(X)
        return (A2 > 0.5).astype(int)

    def accuracy(self, X, Y):
        """Compute classification accuracy."""
        preds = self.predict(X)
        return np.mean(preds == Y) * 100

Step 3: Train the Network

Python
# Create and train the network
nn = ShallowNeuralNetwork(n_x=2, n_h=8, learning_rate=1.2)
costs = nn.train(X, Y, epochs=10000, print_every=1000)
print(f"\nFinal Accuracy: {nn.accuracy(X, Y):.1f}%")

Step 4: Visualize the Decision Boundary

Python
def plot_decision_boundary(model, X, Y):
    """Visualize the non-linear decision boundary."""
    x_min, x_max = X[0].min() - 0.5, X[0].max() + 0.5
    y_min, y_max = X[1].min() - 0.5, X[1].max() + 0.5

    xx, yy = np.meshgrid(
        np.linspace(x_min, x_max, 200),
        np.linspace(y_min, y_max, 200)
    )
    grid = np.c_[xx.ravel(), yy.ravel()].T  # (2, 40000)
    Z = model.predict(grid)
    Z = Z.reshape(xx.shape)

    plt.figure(figsize=(10, 8))
    plt.contourf(xx, yy, Z, levels=[0, 0.5, 1],
                 colors=['#fecaca', '#bbf7d0'], alpha=0.7)
    plt.contour(xx, yy, Z, levels=[0.5], colors=['#7c3aed'],
                linewidths=2)

    # Plot data points
    plt.scatter(X[0, Y[0]==0], X[1, Y[0]==0],
                c='#ef4444', edgecolors='k', s=30, label='Class 0')
    plt.scatter(X[0, Y[0]==1], X[1, Y[0]==1],
                c='#22c55e', edgecolors='k', s=30, label='Class 1')

    plt.title('Decision Boundary — Shallow Neural Network (XOR Data)',
              fontweight='bold', fontsize=14)
    plt.xlabel('Feature x₁')
    plt.ylabel('Feature x₂')
    plt.legend()
    plt.grid(True, alpha=0.3)
    plt.tight_layout()
    plt.show()

plot_decision_boundary(nn, X, Y)

Industry Code — scikit-learn & TensorFlow Equivalents

In production, you'd use optimized libraries. Here's how our from-scratch network maps to industry tools:

scikit-learn: MLPClassifier

Python
from sklearn.neural_network import MLPClassifier

# X_train shape: (m, n_features) — sklearn uses row vectors!
clf = MLPClassifier(
    hidden_layer_sizes=(8,),      # 1 hidden layer, 8 neurons
    activation='tanh',             # hidden layer activation
    solver='sgd',                  # stochastic gradient descent
    learning_rate_init=0.01,       # initial learning rate
    max_iter=10000,                # epochs
    random_state=42
)

clf.fit(X.T, Y.ravel())  # sklearn expects (m, n_features)
print(f"Accuracy: {clf.score(X.T, Y.ravel()) * 100:.1f}%")

TensorFlow/Keras: Sequential API

Python
import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Dense(8, activation='tanh',
                          input_shape=(2,)),       # hidden layer
    tf.keras.layers.Dense(1, activation='sigmoid')  # output layer
])

model.compile(
    optimizer=tf.keras.optimizers.SGD(learning_rate=1.2),
    loss='binary_crossentropy',
    metrics=['accuracy']
)

# Keras expects (m, n_features)
history = model.fit(X.T, Y.T, epochs=200, verbose=0)
loss, acc = model.evaluate(X.T, Y.T, verbose=0)
print(f"Keras Accuracy: {acc * 100:.1f}%")

Comparison: From Scratch vs Industry

Visual Diagrams — Computation Graph & Architecture

Computation Graph for Forward & Backward Pass

Shape Flow Through the Network

Activation Function Shapes

Worked Example — Hand-Computing One Forward & Backward Pass

Let's trace through one complete forward + backward pass with actual numbers. This is the best way to build intuition.

Forward Pass

Hand Calculation
# Layer 1: z[1] = W[1]·x + b[1]
z₁⁽¹⁾ = 0.1×1.0 + 0.3×0.5 + 0.0 = 0.25
z₂⁽¹⁾ = 0.2×1.0 + 0.4×0.5 + 0.0 = 0.40

# Layer 1: a[1] = tanh(z[1])
a₁⁽¹⁾ = tanh(0.25) = 0.2449
a₂⁽¹⁾ = tanh(0.40) = 0.3799

# Layer 2: z[2] = W[2]·a[1] + b[2]
z⁽²⁾ = 0.5×0.2449 + 0.6×0.3799 + 0.0 = 0.3504

# Layer 2: a[2] = σ(z[2])
a⁽²⁾ = σ(0.3504) = 1/(1+e⁻⁰·³⁵⁰⁴) = 0.5867

# ŷ = 0.5867 (predicted probability)
# True y = 1, so the network needs to push ŷ higher

Cost

Hand Calculation
J = -[y·log(ŷ) + (1-y)·log(1-ŷ)]
  = -[1·log(0.5867) + 0·log(0.4133)]
  = -log(0.5867)
  = 0.5333

Backward Pass

Hand Calculation
# Layer 2 gradients
dz⁽²⁾ = a⁽²⁾ - y = 0.5867 - 1 = -0.4133

dW₁⁽²⁾ = dz⁽²⁾ × a₁⁽¹⁾ = -0.4133 × 0.2449 = -0.1013
dW₂⁽²⁾ = dz⁽²⁾ × a₂⁽¹⁾ = -0.4133 × 0.3799 = -0.1570
db⁽²⁾  = dz⁽²⁾ = -0.4133

# Layer 1 gradients (tanh derivative: 1 - tanh²)
dz₁⁽¹⁾ = W₁⁽²⁾·dz⁽²⁾ × (1 - a₁⁽¹⁾²) = 0.5×(-0.4133) × (1 - 0.2449²)
       = -0.2067 × 0.9400 = -0.1943

dz₂⁽¹⁾ = W₂⁽²⁾·dz⁽²⁾ × (1 - a₂⁽¹⁾²) = 0.6×(-0.4133) × (1 - 0.3799²)
       = -0.2480 × 0.8557 = -0.1922

dW⁽¹⁾ = [[dz₁⁽¹⁾·x₁, dz₁⁽¹⁾·x₂],    = [[-0.1943, -0.0972],
          [dz₂⁽¹⁾·x₁, dz₂⁽¹⁾·x₂]]       [-0.1922, -0.0961]]

Parameter Update (α = 1.0)

Hand Calculation
# W[1] := W[1] - α·dW[1]
W⁽¹⁾_new = [[0.1 - 1.0×(-0.1943), 0.3 - 1.0×(-0.0972)],
             [0.2 - 1.0×(-0.1922), 0.4 - 1.0×(-0.0961)]]
          = [[0.2943, 0.3972],
             [0.3922, 0.4961]]

# All weights increased — pushing ŷ toward 1 ✓

# W[2] := W[2] - α·dW[2]
W⁽²⁾_new = [[0.5 - 1.0×(-0.1013), 0.6 - 1.0×(-0.1570)]]
          = [[0.6013, 0.7570]]

Case Study — Mu Sigma: Retail Analytics with Shallow Networks

Common Mistakes & Misconceptions

Comparison Table — Logistic Regression vs. Shallow Neural Network

Exercises

Section A — Multiple Choice Questions (10)

Section B — Short Answer Questions (5)

Section C — Long Answer Questions (3)

Section D — Programming Exercises (2)

Section E — Mini-Project

Chapter Summary

References & Further Reading

Primary Textbooks

• Goodfellow, Bengio & Courville (2016). Deep Learning. MIT Press. Chapters 6 (Deep Feedforward Networks) — the definitive reference for activation functions and network architectures. Free at deeplearningbook.org.
• Andrew Ng — Coursera Deep Learning Specialization. Course 1, Weeks 3-4 — Shallow Neural Networks and Deep Neural Networks. The notation used in this chapter follows Ng's conventions.
• Michael Nielsen (2015). Neural Networks and Deep Learning. Free online book (neuralnetworksanddeeplearning.com). Chapter 1 has excellent visualizations of how neural networks learn.

Landmark Papers

• Cybenko, G. (1989). "Approximation by Superpositions of a Sigmoidal Function." Mathematics of Control, Signals, and Systems. — The Universal Approximation Theorem.
• Hornik, K. (1991). "Approximation Capabilities of Multilayer Feedforward Networks." Neural Networks. — Extended the theorem to arbitrary activation functions.
• Nair, V. & Hinton, G. (2010). "Rectified Linear Units Improve Restricted Boltzmann Machines." ICML. — The paper that popularized ReLU.
• Glorot, X. & Bengio, Y. (2010). "Understanding the Difficulty of Training Deep Feedforward Neural Networks." AISTATS. — Xavier initialization.
• Clevert, D., Unterthiner, T. & Hochreiter, S. (2016). "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)." ICLR.

Indian Industry Context

• Mu Sigma: musigma.com — Case studies in retail analytics, CPG, and insurance. Blog articles on decision science methodology.
• NASSCOM AI Report (2024): India's analytics industry overview — growth trends, talent pipeline, and adoption across sectors.
• NPTEL Deep Learning Course (IIT Madras): Prof. Mitesh Khapra's course covers shallow networks in Weeks 4-5 with excellent Hindi/English explanations — nptel.ac.in.
• IndiaAI Portal: indiaai.gov.in — Government AI resources, datasets, and case studies from Indian industry.

Visualization Tools

• TensorFlow Playground: playground.tensorflow.org — Interactively build and train shallow networks. See how hidden neurons create decision boundaries in real time. Start with the XOR dataset!
• 3Blue1Brown — Neural Networks series (YouTube): Grant Sanderson's visual explanations of forward/backward propagation are the best visual resource available.