Chapter 0: The Math You Need

Bloom's Taxonomy Map for This Chapter

Learning Objectives

By the end of this chapter, you will be able to:

• Define scalars, vectors, matrices, and tensors, and translate between mathematical notation and NumPy shapes
• Compute dot products, matrix multiplications, transposes, inverses, and element-wise operations both by hand and in Python
• Derive derivatives from first principles, apply the chain rule to composite functions, and compute gradient vectors
• Explain why the chain rule is the mathematical engine behind backpropagation in neural networks
• Apply Bayes' theorem to real-world problems and derive Maximum Likelihood Estimates step-by-step
• Calculate entropy, KL divergence, and cross-entropy, and justify why cross-entropy is the natural loss for classification
• Connect eigenvalues to PCA, gradients to optimization, and probability to model training
• Implement all key operations in NumPy from scratch and verify with PyTorch

Opening Hook — The Three Languages Every Neural Network Speaks

The Intuition First — Why These Four Topics?

Before we dive in, let's understand the big picture. Every neural network does exactly three things in a loop:

The "Aha!" Question: Imagine you're blindfolded on a hilly landscape, trying to find the lowest valley. You can't see, but you can feel the slope under your feet. The slope (calculus) tells you which direction to step. The terrain itself (linear algebra) defines the landscape. And the valley you're looking for (probability) is where your model's predictions best match reality. That's it. That's deep learning.

Linear Algebra — The Language of Data

0.1.1 Scalars, Vectors, Matrices, and Tensors

Think of a scalar as a single number — the temperature today (38°C). A vector is an ordered list of numbers — the temperatures for a week [38, 36, 40, 37, 35, 39, 41]. A matrix is a grid of numbers — temperatures across 7 days and 3 cities. And a tensor? It's the general term: any n-dimensional array of numbers.

Mathematical Notation Convention

Throughout this book, we follow these conventions (same as Goodfellow's Deep Learning book):

• Scalars: lowercase italic — a, x, α, λ
• Vectors: lowercase bold — x, w, b (column vectors by default)
• Matrices: uppercase bold — W, X, A
• Tensors: uppercase bold calligraphic — 𝒳, 𝒲

Pythonimport numpy as np

# Scalar — 0D
temperature = 38.5
print(f"Scalar: {temperature}, type: {type(temperature)}")

# Vector — 1D
marks = np.array([85, 92, 78, 90, 88])
print(f"Vector shape: {marks.shape}")  # (5,)

# Matrix — 2D: 4 students × 3 subjects
gradebook = np.array([
    [85, 92, 78],   # Student 1
    [90, 88, 95],   # Student 2
    [76, 80, 82],   # Student 3
    [95, 97, 91]    # Student 4
])
print(f"Matrix shape: {gradebook.shape}")  # (4, 3)

# 3D Tensor — batch of images
batch = np.random.randn(32, 28, 28)
print(f"3D Tensor shape: {batch.shape}")  # (32, 28, 28)

0.1.2 Dot Product — The Fundamental Operation of Neural Networks

Physical Analogy: Imagine you're a shopkeeper. A customer buys 3 samosas (₹15 each), 2 teas (₹10 each), and 1 jalebi (₹20). The total bill is:

Formally, given two vectors a = [a₁, a₂, ..., aₙ] and b = [b₁, b₂, ..., bₙ], their dot product is:

Matrix Multiplication — Derived from Dot Products

If you can do a dot product, you can do matrix multiplication. Matrix multiplication C = AB simply computes the dot product of every row of A with every column of B:

Pythonimport numpy as np

# Dot product
a = np.array([3, 2, 1])       # quantities
b = np.array([15, 10, 20])    # prices

# Method 1: manual
dot_manual = sum(ai * bi for ai, bi in zip(a, b))
print(f"Manual dot: {dot_manual}")    # 85

# Method 2: NumPy
dot_numpy = np.dot(a, b)               # 85
dot_at    = a @ b                       # 85 (preferred syntax)

# Matrix multiplication
A = np.array([[1,2,3], [4,5,6]])   # (2, 3)
B = np.array([[7,8], [9,10], [11,12]])  # (3, 2)

C = A @ B                               # (2, 2)
print(f"A @ B =\n{C}")
# [[ 58  64]
#  [139 154]]

# Verify C[0,0] by hand:
print(f"C[0,0] = {1*7 + 2*9 + 3*11}")  # 58 ✓

# Simulating IRCTC passenger matrix
np.random.seed(42)
n_passengers = 100

# Feature matrix X: (100, 5)
X = np.column_stack([
    np.random.randint(18, 75, n_passengers),   # age
    np.random.randint(50, 3000, n_passengers),  # distance_km
    np.random.randint(150, 5000, n_passengers), # ticket_price (₹)
    np.random.randint(0, 20, n_passengers),    # bookings_last_year
    np.random.randint(1, 200, n_passengers)    # waitlist_position
])
print(f"Data matrix shape: {X.shape}")  # (100, 5)

# Weight vector (learned by model)
w = np.array([[0.01], [-0.001], [0.0005], [0.1], [-0.05]])  # (5, 1)
b = 0.5

# Prediction: ONE matrix multiply handles ALL 100 passengers!
z = X @ w + b  # (100,5) @ (5,1) = (100,1)
print(f"Predictions shape: {z.shape}")  # (100, 1)
print(f"First 5 scores: {z[:5].flatten()}")

# Netflix-style rating matrix (? = unknown)
# Movies: Avengers, Notebook, Inception, DDLJ
R = np.array([
    [5, 1, 4, 2],   # User 1: likes action
    [4, 2, 5, 1],   # User 2: likes action+sci-fi
    [1, 5, 2, 5],   # User 3: likes romance
    [2, 4, 1, 4],   # User 4: likes romance
    [5, 2, 4, 1]    # User 5: likes action
])

# Approximate R ≈ U @ M (low-rank factorization)
# U captures user preferences for 2 latent factors
# M captures movie associations with those factors
U = np.array([[2.1,0.3],[2.3,0.1],[0.2,2.4],[0.4,2.1],[2.2,0.2]])
M = np.array([[2.3,0.5,1.9,0.3],[0.2,2.0,0.4,2.1]])

R_approx = U @ M
print("Approximate ratings:")
print(np.round(R_approx, 1))
print(f"\nReconstruction error: {np.mean((R - R_approx)**2):.3f}")

0.1.3 Transpose, Inverse, and Element-wise Operations

Transpose — Flipping Rows and Columns

The transpose of matrix A (shape m×n) is Aᵀ (shape n×m), obtained by swapping rows and columns: (Aᵀ)ᵢⱼ = Aⱼᵢ

Inverse — "Undoing" a Matrix

For a square matrix A, its inverse A⁻¹ satisfies: A·A⁻¹ = A⁻¹·A = I (identity matrix).

Analogy: If matrix A represents "encrypt," then A⁻¹ represents "decrypt." Applying both gives you back the original.

Hadamard (Element-wise) Product ⊙

Unlike matrix multiplication, the Hadamard product multiplies corresponding elements: (A ⊙ B)ᵢⱼ = Aᵢⱼ × Bᵢⱼ. Both matrices must have the same shape.

Python# Transpose
A = np.array([[1,2,3],[4,5,6]])  # (2,3)
print(f"A.T shape: {A.T.shape}")    # (3,2)

# Inverse
B = np.array([[2,1],[5,3]])      # det = 6-5 = 1 ≠ 0, invertible
B_inv = np.linalg.inv(B)
print(f"B @ B_inv =\n{np.round(B @ B_inv)}")  # Identity!

# Hadamard (element-wise) product
X = np.array([[1,2],[3,4]])
Y = np.array([[5,6],[7,8]])
hadamard = X * Y                     # Element-wise: [[5,12],[21,32]]
matmul   = X @ Y                     # Matrix mult:  [[19,22],[43,50]]

print(f"Hadamard:\n{hadamard}")
print(f"MatMul:\n{matmul}")

0.1.4 Eigenvalues and Eigenvectors — The DNA of a Matrix

Analogy: Imagine stretching a rubber sheet. Most points on the sheet move to new positions AND change direction. But some special points only get stretched along their original direction — they don't rotate, they only scale. These special directions are eigenvectors, and the scaling factor is the eigenvalue.

Python# Eigenvalue decomposition
A = np.array([[4, 2], [1, 3]])
eigenvalues, eigenvectors = np.linalg.eig(A)

print(f"Eigenvalues: {eigenvalues}")     # [5. 2.]
print(f"Eigenvectors:\n{eigenvectors}")

# Verify: A @ v = λ * v
v1 = eigenvectors[:, 0]
print(f"A @ v1    = {A @ v1}")
print(f"λ1 * v1   = {eigenvalues[0] * v1}")
# Both should be equal!

🔮 Preview: PCA — Why Eigenvalues Matter for Deep Learning

Principal Component Analysis (PCA) finds the directions of maximum variance in your data. How? By computing the eigenvalues and eigenvectors of the covariance matrix. The eigenvector with the largest eigenvalue points in the direction of greatest spread — that's your first principal component.

In Chapter 10 (Batch Normalization) and Chapter 17 (Applied CV), you'll use PCA to visualize high-dimensional feature spaces. The math starts here.

Calculus — The Engine of Learning

0.2.1 Derivatives from First Principles

Physical Analogy: You're driving on the Mumbai-Pune Expressway. Your speedometer shows 90 km/h — that's the derivative of your position with respect to time. The derivative tells you how fast something is changing, right now.

Essential Derivative Rules

Python# Numerical derivative — verify the definition
def numerical_derivative(f, x, h=1e-7):
    """f'(x) ≈ [f(x+h) - f(x)] / h"""
    return (f(x + h) - f(x)) / h

# Test with f(x) = x², expected f'(3) = 6
f = lambda x: x**2
print(f"d/dx(x²) at x=3: {numerical_derivative(f, 3):.6f}")  # ≈ 6.0

# Test with f(x) = eˣ, expected f'(2) = e² ≈ 7.389
import math
g = lambda x: math.exp(x)
print(f"d/dx(eˣ) at x=2: {numerical_derivative(g, 2):.6f}")
print(f"Exact e²:        {math.exp(2):.6f}")

0.2.2 The Chain Rule — THE Most Important Rule for Deep Learning

If you remember only one thing from this chapter, make it the chain rule. Backpropagation — the algorithm that trains every neural network — IS the chain rule applied systematically.

Analogy: Imagine a Rube Goldberg machine: a ball hits a lever, which pushes a domino, which rings a bell. If you move the ball 1 cm, how much does the bell volume change? You multiply the effect at each stage: (ball→lever effect) × (lever→domino effect) × (domino→bell effect).

Python# Chain rule in action — computing gradients
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def sigmoid_derivative(z):
    s = sigmoid(z)
    return s * (1 - s)

# Forward pass: y = sigmoid(w*x + b)
w, x, b = 2.0, 3.0, -1.0
z = w * x + b                  # z = 5.0
y = sigmoid(z)                 # y ≈ 0.9933

# Backward pass: chain rule
dy_dz = sigmoid_derivative(z)  # σ'(z) ≈ 0.0066
dz_dw = x                      # ∂z/∂w = x = 3.0
dz_db = 1.0                    # ∂z/∂b = 1

dy_dw = dy_dz * dz_dw          # Chain rule! ≈ 0.0198
dy_db = dy_dz * dz_db          # ≈ 0.0066

print(f"y = {y:.4f}")
print(f"∂y/∂w = {dy_dw:.4f}")
print(f"∂y/∂b = {dy_db:.4f}")

# Verify with numerical gradient
h = 1e-7
numerical_dw = (sigmoid((w+h)*x + b) - sigmoid(w*x + b)) / h
print(f"Numerical ∂y/∂w = {numerical_dw:.4f}")  # Should match!

x = 1.0
# y = (x³ + 2x)²
u = x**3 + 2*x         # u = 3
y = u**2                # y = 9

# Student's gradient computation:
dy_du = 2 * u            # = 6
du_dx = 3 * x**2        # = 3  ← BUG HERE!
dy_dx = dy_du * du_dx    # = 18  ← WRONG!

0.2.3 Partial Derivatives, Gradients, Jacobian, and Hessian

In deep learning, functions have millions of inputs (parameters). We need derivatives with respect to each input separately — these are partial derivatives.

Analogy: You're adjusting the volume AND bass on a speaker. The partial derivative ∂sound/∂volume tells you how sound changes when you turn only the volume knob. The partial derivative ∂sound/∂bass tells you the effect of only the bass knob.

Jacobian Matrix (Brief)

When you have a vector-valued function f: ℝⁿ → ℝᵐ (like a neural network layer), the Jacobian is the matrix of all partial derivatives:

Hessian Matrix (Brief)

The Hessian H is the matrix of second derivatives: H[i,j] = ∂²f/∂xᵢ∂xⱼ. It tells you about the curvature — is the surface a bowl (easy to optimize) or a saddle (tricky)? You'll meet the Hessian in Chapter 8 (Optimization) when studying Newton's method and saddle points.

# Google ad revenue gradient
def revenue(b, r, p):
    return b * r**2 * np.log(1 + p)

def revenue_gradient(b, r, p):
    dR_db = r**2 * np.log(1 + p)
    dR_dr = 2 * b * r * np.log(1 + p)
    dR_dp = b * r**2 / (1 + p)
    return np.array([dR_db, dR_dr, dR_dp])

# At current values:
b, r, p = 2.5, 0.8, 0.3
grad = revenue_gradient(b, r, p)
print(f"∂R/∂b = {grad[0]:.4f} (increase bid)")
print(f"∂R/∂r = {grad[1]:.4f} (improve relevance)")
print(f"∂R/∂p = {grad[2]:.4f} (improve click rate)")
print(f"\nBiggest lever: {'bid' if grad[0]>grad[1] else 'relevance'}")

Probability — The Language of Uncertainty

0.3.1 Key Distributions: Bernoulli and Gaussian

Why Probability? Real-world data is noisy. A neural network doesn't output certainties — it outputs probabilities. "This email is spam with probability 0.97." "This X-ray shows pneumonia with probability 0.83." Understanding probability is understanding what your model's outputs mean.

Bernoulli Distribution — The Coin Flip

A random variable X that takes value 1 with probability p and 0 with probability (1-p):

Gaussian (Normal) Distribution — The Bell Curve

Most natural phenomena — heights, measurement errors, neural network weight initializations — follow a Gaussian:

Pythonimport numpy as np

# Bernoulli: simulate coin flips
p = 0.7  # biased coin
flips = np.random.binomial(1, p, size=10000)
print(f"Empirical mean: {flips.mean():.3f} (expected: {p})")
print(f"Empirical var:  {flips.var():.3f} (expected: {p*(1-p):.3f})")

# Gaussian: sample from N(0, 1)
samples = np.random.randn(10000)  # standard normal
print(f"\nGaussian mean: {samples.mean():.3f} (expected: 0)")
print(f"Gaussian std:  {samples.std():.3f} (expected: 1)")

# Weight initialization: N(0, 0.01)
weights = np.random.randn(784, 128) * 0.01
print(f"\nWeight matrix shape: {weights.shape}")
print(f"Weight mean: {weights.mean():.5f}")
print(f"Weight std:  {weights.std():.5f}")

0.3.2 Conditional Probability and Bayes' Theorem

Conditional probability P(A|B) means "probability of A given that B has occurred."

Analogy: You're a doctor. A patient tests positive for a rare disease. The test is 99% accurate. Should you panic? Bayes says: not necessarily. If the disease affects only 1 in 10,000 people, even with a 99% accurate test, the chance the patient actually has the disease is only about 1%! The prior (disease rarity) overwhelms the likelihood (test accuracy).

# PhonePe fraud detection with Bayes' theorem
P_fraud = 0.00001      # Prior: 1 in 100,000
P_flag_given_fraud = 0.95    # Sensitivity
P_flag_given_legit = 0.02    # False positive rate

# P(Flagged) = P(Flag|Fraud)·P(Fraud) + P(Flag|Legit)·P(Legit)
P_flag = (P_flag_given_fraud * P_fraud + 
          P_flag_given_legit * (1 - P_fraud))

# Bayes' theorem
P_fraud_given_flag = (P_flag_given_fraud * P_fraud) / P_flag

print(f"P(Fraud | Flagged) = {P_fraud_given_flag:.6f}")
print(f"That's about 1 in {int(1/P_fraud_given_flag):,}")
print(f"\n→ Only {P_fraud_given_flag*100:.4f}% of flagged transactions")
print(f"  are actually fraudulent!")
print(f"→ This is why PhonePe uses multiple detection layers.")

# A/B testing with Bayesian reasoning
n_A, clicks_A = 50000, 2600   # 5.2%
n_B, clicks_B = 50000, 2900   # 5.8%

p_A = clicks_A / n_A
p_B = clicks_B / n_B

# Standard error of difference
se = np.sqrt(p_A * (1 - p_A) / n_A + p_B * (1 - p_B) / n_B)

# Z-score (how many standard errors away from 0?)
z = (p_B - p_A) / se

print(f"Control rate:    {p_A:.3f}")
print(f"Treatment rate:  {p_B:.3f}")
print(f"Difference:      {p_B - p_A:.3f}")
print(f"Z-score:         {z:.2f}")
print(f"Significant?     {'YES (z > 1.96)' if z > 1.96 else 'NO'}")

# Relative lift
print(f"\nRelative lift: {(p_B - p_A) / p_A * 100:.1f}%")
print(f"→ With 230M subscribers, this could mean")
print(f"  {int(230e6 * (p_B - p_A)):,} more clicks per impression!")

0.3.3 Maximum Likelihood Estimation (MLE) — Full Derivation

The Big Idea: You have data. You have a model with parameters θ. MLE asks: "What value of θ makes this observed data most probable?"

Analogy: You find 7 heads out of 10 coin flips. What's the most likely bias of the coin? Your gut says p = 0.7 — and MLE gives you exactly that, mathematically.

Python# MLE from scratch
import numpy as np

# Bernoulli MLE
coin_flips = np.array([1,1,0,1,1,0,1,1,1,0])  # 7 heads, 3 tails
p_mle = coin_flips.mean()
print(f"MLE for p: {p_mle}")  # 0.7

# Gaussian MLE
data = np.array([4.2, 3.8, 4.5, 4.1, 3.9, 4.3, 4.0, 4.4])
mu_mle = data.mean()
sigma2_mle = np.mean((data - mu_mle)**2)  # biased variance (MLE)

print(f"MLE for μ:  {mu_mle:.3f}")
print(f"MLE for σ²: {sigma2_mle:.4f}")

# Log-likelihood as a function of p (for visualization)
p_values = np.linspace(0.01, 0.99, 100)
k = coin_flips.sum()  # 7
n = len(coin_flips)   # 10
log_lik = k * np.log(p_values) + (n - k) * np.log(1 - p_values)

# The peak is at p = 0.7 — the MLE!
best_p = p_values[np.argmax(log_lik)]
print(f"\nPeak of log-likelihood at p = {best_p:.2f}")

MAP Estimation — MLE with a Prior

Sometimes you have prior knowledge. MAP (Maximum A Posteriori) combines the likelihood with a prior distribution:

Information Theory — Why Cross-Entropy?

0.4.1 Entropy — Measuring Surprise

Analogy: Imagine you're watching a cricket match. If India is batting against Bermuda, a wicket falling has high information (it's surprising — India is expected to dominate). If India is batting against Australia at 45/5, another wicket has low information (you expected it). Entropy measures the average surprise in a probability distribution.

Pythonimport numpy as np

def entropy(probs):
    """Shannon entropy in bits"""
    probs = np.array(probs)
    probs = probs[probs > 0]  # avoid log(0)
    return -np.sum(probs * np.log2(probs))

# Fair coin — maximum uncertainty for 2 outcomes
print(f"Fair coin:     {entropy([0.5, 0.5]):.3f} bits")  # 1.0

# Biased coin — less uncertain
print(f"90/10 coin:    {entropy([0.9, 0.1]):.3f} bits")  # 0.469

# Certain outcome — zero uncertainty
print(f"Sure thing:    {entropy([1.0, 0.0]):.3f} bits")  # 0.0

# Uniform over 8 outcomes — max entropy for 8 classes
print(f"Uniform(8):    {entropy([1/8]*8):.3f} bits")     # 3.0

0.4.2 KL Divergence — Distance Between Distributions

KL Divergence D_KL(P || Q) measures how different distribution Q is from the "true" distribution P:

0.4.3 Cross-Entropy — THE Classification Loss

Here's the beautiful connection:

Pythondef cross_entropy(y_true, y_pred):
    """Binary cross-entropy loss"""
    epsilon = 1e-15  # avoid log(0)
    y_pred = np.clip(y_pred, epsilon, 1 - epsilon)
    return -np.mean(y_true * np.log(y_pred) + 
                     (1 - y_true) * np.log(1 - y_pred))

# Good predictions → low loss
y_true = np.array([1, 0, 1, 1, 0])
y_good = np.array([0.9, 0.1, 0.8, 0.95, 0.05])
y_bad  = np.array([0.5, 0.5, 0.5, 0.5, 0.5])
y_terrible = np.array([0.1, 0.9, 0.2, 0.1, 0.8])

print(f"Good predictions:     CE = {cross_entropy(y_true, y_good):.4f}")
print(f"Random predictions:   CE = {cross_entropy(y_true, y_bad):.4f}")
print(f"Terrible predictions: CE = {cross_entropy(y_true, y_terrible):.4f}")

Python Implementation — NumPy from Scratch + PyTorch Verification

All Key Operations: NumPy vs PyTorch

Python — NumPy from scratchimport numpy as np

# ══════════════ LINEAR ALGEBRA ══════════════

# Dot product from scratch
def dot_product(a, b):
    """Manual dot product — no NumPy shortcuts"""
    assert len(a) == len(b), "Vectors must have same length"
    return sum(ai * bi for ai, bi in zip(a, b))

# Matrix multiply from scratch
def matmul(A, B):
    """Manual matrix multiplication"""
    m, k1 = len(A), len(A[0])
    k2, n = len(B), len(B[0])
    assert k1 == k2, f"Cannot multiply ({m},{k1}) × ({k2},{n})"
    C = [[0] * n for _ in range(m)]
    for i in range(m):
        for j in range(n):
            for p in range(k1):
                C[i][j] += A[i][p] * B[p][j]
    return C

# ══════════════ CALCULUS ══════════════

# Numerical gradient
def numerical_gradient(f, x, h=1e-7):
    """Compute gradient of f at vector x"""
    grad = np.zeros_like(x, dtype=float)
    for i in range(len(x)):
        x_plus = x.copy(); x_plus[i] += h
        x_minus = x.copy(); x_minus[i] -= h
        grad[i] = (f(x_plus) - f(x_minus)) / (2 * h)
    return grad

# Test: f(x,y) = x² + 3xy + y²
f = lambda v: v[0]**2 + 3*v[0]*v[1] + v[1]**2
point = np.array([1.0, 2.0])
grad = numerical_gradient(f, point)
print(f"Numerical gradient at (1,2): {grad}")  # [8, 7]
print(f"Analytical gradient: [2*1+3*2, 3*1+2*2] = [8, 7]")

# ══════════════ PROBABILITY ══════════════

# MLE for Gaussian from scratch
def gaussian_mle(data):
    """Returns MLE estimates of μ and σ²"""
    mu = sum(data) / len(data)
    sigma2 = sum((x - mu)**2 for x in data) / len(data)
    return mu, sigma2

Python — PyTorch verificationimport torch

# ══════════════ LINEAR ALGEBRA ══════════════
A = torch.tensor([[1.,2.,3.],[4.,5.,6.]])
B = torch.tensor([[7.,8.],[9.,10.],[11.,12.]])
print("MatMul:", A @ B)
print("Eigenvalues:", torch.linalg.eig(
    torch.tensor([[4.,2.],[1.,3.]])).eigenvalues)

# ══════════════ AUTOGRAD (chain rule!) ══════════════
x = torch.tensor(3.0, requires_grad=True)
y = (x**3 + 2*x)**2
y.backward()
print(f"dy/dx at x=3: {x.grad}")  # PyTorch computes it for you!

# ══════════════ CROSS-ENTROPY ══════════════
import torch.nn as nn
loss_fn = nn.BCELoss()
y_true = torch.tensor([1., 0., 1., 1., 0.])
y_pred = torch.tensor([0.9, 0.1, 0.8, 0.95, 0.05])
print(f"PyTorch BCE: {loss_fn(y_pred, y_true):.4f}")

Visual Aids — ASCII Diagrams

Common Misconceptions

GATE/Exam Corner

Formula Sheet — Keep This Page Bookmarked!

5 GATE-Style Practice Questions

GATE Prediction Table

Interview Prep — India + US Focus

5 Conceptual Questions

3 Coding Questions

def softmax(z):
    """Numerically stable softmax"""
    z_shifted = z - np.max(z)  # stability trick
    exp_z = np.exp(z_shifted)
    return exp_z / np.sum(exp_z)

# Test
logits = np.array([2.0, 1.0, 0.1])
probs = softmax(logits)
print(f"Softmax: {probs}")        # [0.659, 0.242, 0.098]
print(f"Sum: {probs.sum():.4f}")  # 1.0

def gradient_check(f, df, x, h=1e-7):
    """Compare analytical vs numerical gradient"""
    numerical = (f(x + h) - f(x - h)) / (2 * h)
    analytical = df(x)
    rel_error = abs(numerical - analytical) / (abs(numerical) + 1e-8)
    print(f"Numerical: {numerical:.8f}")
    print(f"Analytical: {analytical:.8f}")
    print(f"Relative error: {rel_error:.2e}")
    return rel_error < 1e-5

# Test: f(x) = sin(x²), f'(x) = 2x·cos(x²)
f = lambda x: np.sin(x**2)
df = lambda x: 2 * x * np.cos(x**2)
print("Passed!" if gradient_check(f, df, 1.5) else "FAILED")

def bce_loss_and_grad(y_true, y_pred):
    """Binary cross-entropy with gradient"""
    eps = 1e-15
    y_pred = np.clip(y_pred, eps, 1-eps)
    loss = -np.mean(y_true * np.log(y_pred) + 
                     (1-y_true) * np.log(1-y_pred))
    # Gradient: ∂L/∂ŷ = (-y/ŷ + (1-y)/(1-ŷ)) / n
    grad = (-y_true/y_pred + (1-y_true)/(1-y_pred)) / len(y_true)
    return loss, grad

y = np.array([1,0,1])
p = np.array([0.9,0.2,0.8])
loss, grad = bce_loss_and_grad(y, p)
print(f"Loss: {loss:.4f}, Gradient: {grad}")

Hands-On Lab / Mini-Project: Linear Regression from Matrix Calculus

🎯 Project: Build Linear Regression from Scratch — No sklearn Allowed!

You will derive the normal equation using matrix calculus, implement it in NumPy, and then implement gradient descent — all from scratch.

Background: The Normal Equation

Python — Complete Mini-Projectimport numpy as np

# ═══════════ GENERATE DATA ═══════════
np.random.seed(42)
n_samples = 100

# True relationship: house_price = 50*area + 30*rooms + 1000 + noise
area = np.random.uniform(500, 3000, n_samples)
rooms = np.random.randint(1, 6, n_samples).astype(float)
noise = np.random.randn(n_samples) * 500
price = 50 * area + 30 * rooms + 1000 + noise

# ═══════════ METHOD 1: NORMAL EQUATION ═══════════
# Add bias column (column of 1s)
X = np.column_stack([np.ones(n_samples), area, rooms])  # (100, 3)
y = price  # (100,)

# w* = (XᵀX)⁻¹ Xᵀy — from our derivation!
w_normal = np.linalg.inv(X.T @ X) @ X.T @ y

print("═══ METHOD 1: Normal Equation ═══")
print(f"Bias (intercept):  {w_normal[0]:.2f} (true: 1000)")
print(f"Area coefficient:  {w_normal[1]:.2f} (true: 50)")
print(f"Rooms coefficient: {w_normal[2]:.2f} (true: 30)")

# ═══════════ METHOD 2: GRADIENT DESCENT ═══════════
# Normalize features for gradient descent
X_norm = X.copy()
X_norm[:, 1] = (X_norm[:, 1] - X_norm[:, 1].mean()) / X_norm[:, 1].std()
X_norm[:, 2] = (X_norm[:, 2] - X_norm[:, 2].mean()) / X_norm[:, 2].std()
y_norm = (y - y.mean()) / y.std()

w_gd = np.zeros(3)
learning_rate = 0.01
n_epochs = 1000

print("\n═══ METHOD 2: Gradient Descent ═══")
for epoch in range(n_epochs):
    # Forward: predictions
    y_pred = X_norm @ w_gd
    
    # Loss: MSE
    loss = np.mean((y_pred - y_norm) ** 2) / 2
    
    # Gradient: ∂L/∂w = (1/n) Xᵀ(ŷ - y)
    gradient = (1 / n_samples) * X_norm.T @ (y_pred - y_norm)
    
    # Update: w = w - α·∇L
    w_gd = w_gd - learning_rate * gradient
    
    if epoch % 200 == 0:
        print(f"  Epoch {epoch:4d}: Loss = {loss:.6f}")

print(f"  Final weights (normalized): {w_gd}")
print(f"\n✅ Both methods converge to the same solution!")

Rubric

Exercises — 24 Questions Across All Bloom's Levels

Section A: Conceptual (5 Questions)

Section B: Mathematical (8 Questions)

Section C: Coding (4 Questions)

Section D: Critical Thinking (3 Questions)

★ Starred Research Questions (2 Questions)

Connections — Where This Math Goes Next

Chapter Summary — 7 Key Takeaways

🎯 Key Intuition: Linear algebra stores the data, calculus teaches the network, probability measures the truth, and information theory bridges them all.

Further Reading

🇮🇳 Indian Resources

• NPTEL — Mathematics for Machine Learning by Prof. Arun Rajkumar, IIT Madras — Excellent free video lectures covering exactly this chapter's topics
• NPTEL — Probability and Statistics by Prof. Krishna Jagannathan, IIT Madras — Deep dive into MLE, MAP, and Bayesian estimation
• GATE Mathematics Previous Year Questions (2015-2024) — Practice eigenvalue, matrix, probability questions
• "Higher Engineering Mathematics" by B.S. Grewal — Classic Indian textbook, great for derivatives and matrix problems
• "Engineering Mathematics" by Erwin Kreyszig — Used in most Indian engineering curricula

🌍 Global Resources

• "Mathematics for Machine Learning" by Deisenroth, Faisal & Ong (free PDF at mml-book.com) — The gold standard textbook
• 3Blue1Brown — "Essence of Linear Algebra" (YouTube) — The most beautiful visual explanations of vectors, matrices, and eigenvalues
• 3Blue1Brown — "Essence of Calculus" (YouTube) — Derivatives and chain rule visualized brilliantly
• Distill.pub — "Why Momentum Really Works" — Visual explanation connecting gradients to optimization
• "Deep Learning" Chapter 2-4 by Goodfellow, Bengio & Courville — The definitive reference for math prerequisites
• Gilbert Strang — "Linear Algebra and Its Applications" and MIT OpenCourseWare 18.06 — The best linear algebra course
• StatQuest with Josh Starmer (YouTube) — MLE, Bayes, and distributions explained with humor

📑 Key Papers

• Vaswani et al. (2017) — "Attention Is All You Need" — Matrix multiplication IS attention
• Glorot & Bengio (2010) — "Understanding the difficulty of training deep feedforward neural networks" — Eigenvalue analysis of Jacobians
• Lin et al. (2017) — "Focal Loss for Dense Object Detection" — Cross-entropy modification
• Jacot et al. (2018) — "Neural Tangent Kernel" — Connection between linear algebra and infinite-width networks