Chapter 12: Convolutional Neural Networks

Bloom's Taxonomy Map for This Chapter

Learning Objectives

By the end of this chapter, you will be able to:

• Explain why flattening an image into a 1-D vector destroys spatial information and causes a parameter explosion (e.g., 224×224×3 = 150,528 inputs per neuron)
• Define the discrete 2D convolution operation, draw a kernel sliding over an input, and compute the output feature map dimensions: (n − f + 2p) / s + 1
• Distinguish between "valid" padding (no padding) and "same" padding (p = (f−1)/2) and explain when each is appropriate
• Describe how multiple convolutional filters produce multiple feature maps, each detecting different features (edges, textures, patterns)
• Compare Max Pooling and Average Pooling — their formulas, behaviour, and the fact that pooling layers have zero learnable parameters
• Sketch the canonical CNN pipeline: [CONV → ReLU → POOL] × N → Flatten → FC → Softmax and trace tensor shapes through each layer
• Summarise the evolution of classic architectures: LeNet-5 (1998), AlexNet (2012), VGG-16 (2014), ResNet (2015) and explain residual connections
• Implement 2D convolution from scratch in NumPy, verifying output against SciPy's correlate2d
• Build a complete CNN in TensorFlow/Keras for CIFAR-10 achieving ≥ 75% test accuracy
• Apply transfer learning using pretrained VGG-16 / ResNet-50 / MobileNetV2 to a custom Indian dataset with fine-tuning

Opening Hook — The Eyes of E-Commerce

Core Concepts

From-Scratch Code — NumPy 2D Convolution

Let's implement the core 2D convolution operation from scratch, then verify it against SciPy.

4.1 — Single-Channel 2D Convolution

Python
import numpy as np

def conv2d(image, kernel, padding=0, stride=1):
    """
    Perform 2D convolution (cross-correlation) from scratch.
    
    Parameters:
        image  : np.array of shape (H, W)
        kernel : np.array of shape (f, f)
        padding: int, number of zero-padding rings
        stride : int, step size
    
    Returns:
        output : np.array of shape (H_out, W_out)
    """
    # Step 1: Pad the input image
    if padding > 0:
        image = np.pad(image, pad_width=padding, mode='constant', constant_values=0)
    
    H, W = image.shape
    f = kernel.shape[0]
    
    # Step 2: Calculate output dimensions
    H_out = (H - f) // stride + 1
    W_out = (W - f) // stride + 1
    
    # Step 3: Initialize output
    output = np.zeros((H_out, W_out))
    
    # Step 4: Slide kernel and compute element-wise multiply + sum
    for i in range(H_out):
        for j in range(W_out):
            h_start = i * stride
            w_start = j * stride
            receptive_field = image[h_start:h_start+f, w_start:w_start+f]
            output[i, j] = np.sum(receptive_field * kernel)
    
    return output

# ── Demo: Vertical edge detection ──
image = np.array([
    [10, 10, 10, 0, 0],
    [10, 10, 10, 0, 0],
    [10, 10, 10, 0, 0],
    [10, 10, 10, 0, 0],
    [10, 10, 10, 0, 0]
], dtype=np.float64)

vertical_edge_kernel = np.array([
    [ 1,  0, -1],
    [ 1,  0, -1],
    [ 1,  0, -1]
], dtype=np.float64)

result = conv2d(image, vertical_edge_kernel, padding=0, stride=1)
print("Output shape:", result.shape)
print("Feature map:\n", result)

4.2 — Multi-Channel Convolution (RGB Support)

Python
def conv2d_multichannel(image, kernels, bias, padding=0, stride=1):
    """
    Multi-channel, multi-filter 2D convolution.
    
    Parameters:
        image   : np.array of shape (H, W, C_in)
        kernels : np.array of shape (n_filters, f, f, C_in)
        bias    : np.array of shape (n_filters,)
        padding : int
        stride  : int
    
    Returns:
        output  : np.array of shape (H_out, W_out, n_filters)
    """
    n_filters = kernels.shape[0]
    f = kernels.shape[1]
    
    # Pad spatial dimensions only (not channels)
    if padding > 0:
        image = np.pad(image, 
            [(padding, padding), (padding, padding), (0, 0)], 
            mode='constant')
    
    H, W, C = image.shape
    H_out = (H - f) // stride + 1
    W_out = (W - f) // stride + 1
    output = np.zeros((H_out, W_out, n_filters))
    
    for k in range(n_filters):
        for i in range(H_out):
            for j in range(W_out):
                h_s, w_s = i * stride, j * stride
                patch = image[h_s:h_s+f, w_s:w_s+f, :]  # (f, f, C_in)
                output[i, j, k] = np.sum(patch * kernels[k]) + bias[k]
    
    return output

# ── Demo: 2 filters on a 5×5 RGB image ──
np.random.seed(42)
rgb_image = np.random.randint(0, 255, (5, 5, 3)).astype(np.float64)
kernels = np.random.randn(2, 3, 3, 3)   # 2 filters, each 3×3×3
biases = np.zeros(2)

output = conv2d_multichannel(rgb_image, kernels, biases, padding=1, stride=1)
print("Input shape :", rgb_image.shape)   # (5, 5, 3)
print("Output shape:", output.shape)      # (5, 5, 2) — same padding, 2 filters

4.3 — Max Pooling From Scratch

Python
def max_pool2d(feature_map, pool_size=2, stride=2):
    """
    Max pooling on a multi-channel feature map.
    
    Parameters:
        feature_map : np.array of shape (H, W, C)
        pool_size   : int, size of pooling window
        stride      : int, step size
    
    Returns:
        output : np.array of shape (H_out, W_out, C)
    """
    H, W, C = feature_map.shape
    H_out = (H - pool_size) // stride + 1
    W_out = (W - pool_size) // stride + 1
    output = np.zeros((H_out, W_out, C))
    
    for i in range(H_out):
        for j in range(W_out):
            h_s, w_s = i * stride, j * stride
            window = feature_map[h_s:h_s+pool_size, w_s:w_s+pool_size, :]
            output[i, j, :] = np.max(window, axis=(0, 1))
    
    return output

# ── Demo ──
fmap = np.random.randn(4, 4, 2)
pooled = max_pool2d(fmap, pool_size=2, stride=2)
print("Before pooling:", fmap.shape)    # (4, 4, 2)
print("After pooling :", pooled.shape)  # (2, 2, 2)

4.4 — Verification Against SciPy

Python
from scipy.signal import correlate2d

# Our implementation
our_result = conv2d(image, vertical_edge_kernel, padding=0, stride=1)

# SciPy's implementation (cross-correlation, valid mode)
scipy_result = correlate2d(image, vertical_edge_kernel, mode='valid')

print("Match:", np.allclose(our_result, scipy_result))
print("Max difference:", np.max(np.abs(our_result - scipy_result)))

Industry Code — Full CNN with TensorFlow/Keras (CIFAR-10)

Now let's build a production-quality CNN using TensorFlow/Keras on the CIFAR-10 dataset (60,000 32×32 colour images across 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck).

5.1 — Data Loading and Preprocessing

Python
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, models
import numpy as np

# ── Load CIFAR-10 ──
(X_train, y_train), (X_test, y_test) = keras.datasets.cifar10.load_data()

# ── Normalize pixels to [0, 1] ──
X_train = X_train.astype('float32') / 255.0
X_test  = X_test.astype('float32') / 255.0

# ── Class names ──
class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck']

print(f"Training set  : {X_train.shape}")    # (50000, 32, 32, 3)
print(f"Test set      : {X_test.shape}")     # (10000, 32, 32, 3)
print(f"Pixel range   : [{X_train.min()}, {X_train.max()}]")

5.2 — Data Augmentation

Python
# ── Data augmentation to reduce overfitting ──
data_augmentation = keras.Sequential([
    layers.RandomFlip("horizontal"),
    layers.RandomRotation(0.1),          # ±10% rotation
    layers.RandomTranslation(0.1, 0.1),  # ±10% shift
    layers.RandomZoom(0.1),               # ±10% zoom
], name="data_augmentation")

5.3 — CNN Model Architecture

Python
def build_cnn(input_shape=(32, 32, 3), num_classes=10):
    """Build a CNN following the [CONV→BN→ReLU→POOL] pattern."""
    
    inputs = layers.Input(shape=input_shape)
    x = data_augmentation(inputs)  # Augmentation (only active during training)
    
    # ── Block 1: 32 filters ──
    x = layers.Conv2D(32, (3,3), padding='same')(x)
    x = layers.BatchNormalization()(x)
    x = layers.Activation('relu')(x)
    x = layers.Conv2D(32, (3,3), padding='same')(x)
    x = layers.BatchNormalization()(x)
    x = layers.Activation('relu')(x)
    x = layers.MaxPooling2D((2,2))(x)          # 32×32 → 16×16
    x = layers.Dropout(0.25)(x)
    
    # ── Block 2: 64 filters ──
    x = layers.Conv2D(64, (3,3), padding='same')(x)
    x = layers.BatchNormalization()(x)
    x = layers.Activation('relu')(x)
    x = layers.Conv2D(64, (3,3), padding='same')(x)
    x = layers.BatchNormalization()(x)
    x = layers.Activation('relu')(x)
    x = layers.MaxPooling2D((2,2))(x)          # 16×16 → 8×8
    x = layers.Dropout(0.25)(x)
    
    # ── Block 3: 128 filters ──
    x = layers.Conv2D(128, (3,3), padding='same')(x)
    x = layers.BatchNormalization()(x)
    x = layers.Activation('relu')(x)
    x = layers.Conv2D(128, (3,3), padding='same')(x)
    x = layers.BatchNormalization()(x)
    x = layers.Activation('relu')(x)
    x = layers.MaxPooling2D((2,2))(x)          # 8×8 → 4×4
    x = layers.Dropout(0.25)(x)
    
    # ── Classification Head ──
    x = layers.GlobalAveragePooling2D()(x)     # 4×4×128 → 128
    x = layers.Dense(256)(x)
    x = layers.BatchNormalization()(x)
    x = layers.Activation('relu')(x)
    x = layers.Dropout(0.5)(x)
    outputs = layers.Dense(num_classes, activation='softmax')(x)
    
    return models.Model(inputs, outputs, name="CIFAR10_CNN")

model = build_cnn()
model.summary()

5.4 — Compile and Train

Python
# ── Compile with Adam optimizer and learning rate schedule ──
model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

# ── Callbacks ──
callbacks = [
    keras.callbacks.EarlyStopping(
        monitor='val_accuracy', patience=10, restore_best_weights=True
    ),
    keras.callbacks.ReduceLROnPlateau(
        monitor='val_loss', factor=0.5, patience=5, min_lr=1e-6
    )
]

# ── Train ──
history = model.fit(
    X_train, y_train,
    epochs=50,
    batch_size=64,
    validation_split=0.1,
    callbacks=callbacks,
    verbose=1
)

5.5 — Evaluate and Visualise

Python
import matplotlib.pyplot as plt

# ── Test set evaluation ──
test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
print(f"Test Accuracy: {test_acc:.4f}")
print(f"Test Loss    : {test_loss:.4f}")

# ── Plot training curves ──
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))

ax1.plot(history.history['accuracy'], label='Train')
ax1.plot(history.history['val_accuracy'], label='Validation')
ax1.set_title('Accuracy')
ax1.set_xlabel('Epoch')
ax1.legend()
ax1.grid(True, alpha=0.3)

ax2.plot(history.history['loss'], label='Train')
ax2.plot(history.history['val_loss'], label='Validation')
ax2.set_title('Loss')
ax2.set_xlabel('Epoch')
ax2.legend()
ax2.grid(True, alpha=0.3)

plt.tight_layout()
plt.savefig('cifar10_training_curves.png', dpi=150)
plt.show()

5.6 — Transfer Learning with MobileNetV2

Python
# ── Transfer Learning: MobileNetV2 pretrained on ImageNet ──
base_model = keras.applications.MobileNetV2(
    input_shape=(32, 32, 3),
    include_top=False,          # Remove ImageNet classification head
    weights='imagenet'
)
base_model.trainable = False   # Freeze all pretrained layers

# ── New classification head for CIFAR-10 ──
inputs = layers.Input(shape=(32, 32, 3))
x = data_augmentation(inputs)
x = keras.applications.mobilenet_v2.preprocess_input(x)
x = base_model(x, training=False)
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dropout(0.3)(x)
outputs = layers.Dense(10, activation='softmax')(x)

transfer_model = models.Model(inputs, outputs)

print(f"Total params     : {transfer_model.count_params():,}")
print(f"Trainable params : {sum(p.numpy().size for p in transfer_model.trainable_weights):,}")

# ── Train (feature extraction phase) ──
transfer_model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

transfer_model.fit(X_train, y_train, epochs=10, batch_size=64, 
                   validation_split=0.1)

# ── Fine-tuning phase: unfreeze last 30 layers ──
base_model.trainable = True
for layer in base_model.layers[:-30]:
    layer.trainable = False

transfer_model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-5),  # Very small LR!
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

transfer_model.fit(X_train, y_train, epochs=10, batch_size=64,
                   validation_split=0.1)

Visual Diagrams

6.1 — CNN Feature Hierarchy

6.2 — Convolution Operation Step-by-Step

6.3 — VGG-16 Architecture

6.4 — ResNet Skip Connection

Worked Example — Shape Tracing Through a CNN

Problem: Trace the tensor shapes and parameter counts through a CNN designed for Myntra's product category classification (224×224 RGB images, 50 categories).

Step-by-Step Shape Trace

Block 1: Initial Feature Extraction

Conv(64, 7×7, stride=2, same):

BN + ReLU: Shape unchanged → 112 × 112 × 64. BN params: 64×4 = 256

MaxPool(3×3, stride=2, same):

Block 2: Deepening Features

Conv(128, 3×3, stride=1, same):

MaxPool(2×2, stride=2): → 28 × 28 × 128, Params: 0

Block 3: High-Level Features

Conv(256, 3×3, same) × 2:

MaxPool(2×2, stride=2): → 14 × 14 × 256

Classification Head

Complete Summary

Compare this to a fully connected network: first layer alone would need 224×224×3 × 512 = 76.8 million parameters. Our CNN achieves similar expressive power with just 1.1M parameters — a 68× reduction.

Case Study — DigiYatra: CNN-Powered Face Recognition at Indian Airports

Common Mistakes & Misconceptions

Comparison Tables

10.1 — CNN vs. Fully Connected Network

10.2 — Padding Comparison

10.3 — Pooling Comparison

10.4 — When to Use Which Architecture

Exercises

Chapter Summary

References & Further Reading

Foundational Papers

1. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). "Gradient-Based Learning Applied to Document Recognition." Proceedings of the IEEE, 86(11), 2278–2324. [The LeNet paper — foundation of all modern CNNs]
2. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). "ImageNet Classification with Deep Convolutional Neural Networks." NeurIPS 2012. [AlexNet — started the deep learning revolution]
3. Simonyan, K. & Zisserman, A. (2015). "Very Deep Convolutional Networks for Large-Scale Image Recognition." ICLR 2015. [VGG — proved that depth with small filters wins]
4. He, K., Zhang, X., Ren, S., & Sun, J. (2016). "Deep Residual Learning for Image Recognition." CVPR 2016. [ResNet — skip connections enable 150+ layer networks]
5. Sandler, M., Howard, A., et al. (2018). "MobileNetV2: Inverted Residuals and Linear Bottlenecks." CVPR 2018. [MobileNetV2 — efficient CNN for mobile devices]

Textbooks

6. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning, Chapter 9: Convolutional Networks. MIT Press. [Comprehensive theoretical treatment]
7. Chollet, F. (2021). Deep Learning with Python, 2nd Ed., Chapter 8: Computer Vision. Manning. [Practical Keras implementation guide]
8. Géron, A. (2022). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 3rd Ed., Chapter 14. O'Reilly. [Excellent practical coverage with code]

Indian Context

9. DigiYatra Foundation. (2024). "DigiYatra Technical Specifications." Ministry of Civil Aviation, Government of India. [Official documentation on India's facial recognition boarding system]
10. Digital Personal Data Protection Act (DPDPA), 2023. Government of India Gazette. [India's primary data protection legislation]
11. Sharma, A., et al. (2022). "Crop Disease Detection using Transfer Learning on Indian Agricultural Images." TCS Research. [CNN applications in Indian agriculture]
12. Niramai Health Analytix. (2023). "Thermalytix: AI-Powered Breast Cancer Screening." Technical Whitepaper. [CNN for medical imaging in rural India]

Online Resources

13. CS231n: Convolutional Neural Networks for Visual Recognition — Stanford University (2023). [The gold-standard course on CNNs]
14. TensorFlow CNN Tutorial: https://www.tensorflow.org/tutorials/images/cnn
15. Keras Applications API: https://keras.io/api/applications/ [Pretrained models documentation]

Datasets

16. CIFAR-10 & CIFAR-100 — Alex Krizhevsky (2009). [60K 32×32 images, 10/100 classes]
17. ImageNet (ILSVRC) — Deng et al. (2009). [14M images, 1000 classes — the benchmark that drove CNN evolution]
18. German Traffic Sign Recognition Benchmark (GTSRB). [~50K images, 43 classes — proxy for Indian traffic signs]