PyTorch • EduArtha

PyTorch: Fundamentals

Master the most popular deep learning framework. From tensors to production deployment — build, train, and ship real AI models with PyTorch.

⏱ 3–5 months | 14 Chapters | 50+ Exercises | 14 Projects | Industry Problems

Part I

Tensor Foundations

The building blocks of every PyTorch program

Chapter 1

Tensors & Operations

Learning Objectives

Create tensors from Python lists, NumPy arrays, and built-in factories
Understand dtypes, shapes, strides, and memory layout
Master reshaping, indexing, slicing, and advanced indexing
Apply broadcasting rules for element-wise operations
Convert between PyTorch tensors and NumPy arrays

What Is a Tensor?

A tensor is a multi-dimensional array — the fundamental data structure in PyTorch. Scalars are 0-D tensors, vectors are 1-D, matrices are 2-D, and anything higher is an n-D tensor. Every neural network input, output, weight, and gradient is a tensor.

Dimension	Name	Shape Example	Use Case
0-D	Scalar	`torch.tensor(3.14)`	Loss value
1-D	Vector	`(512,)`	Bias, embedding
2-D	Matrix	`(64, 784)`	Batch of flat images
3-D	3-Tensor	`(32, 100, 512)`	Batch of sequences
4-D	4-Tensor	`(16, 3, 224, 224)`	Batch of RGB images

Creating Tensors

Python
import torch

# From Python data
a = torch.tensor([1, 2, 3])                      # int64 by default
b = torch.tensor([[1.0, 2.0], [3.0, 4.0]])      # float32

# Factory functions
zeros = torch.zeros(3, 4)                        # 3×4 of zeros
ones  = torch.ones(2, 3, dtype=torch.float16)    # specify dtype
rand  = torch.randn(5, 5)                        # standard normal
eye   = torch.eye(4)                              # 4×4 identity
arange = torch.arange(0, 10, 2)                  # [0, 2, 4, 6, 8]
linspace = torch.linspace(0, 1, 5)              # 5 evenly spaced

# From NumPy (shares memory!)
import numpy as np
np_arr = np.array([1, 2, 3])
t = torch.from_numpy(np_arr)                      # zero-copy
back = t.numpy()                                   # back to numpy

Shared Memory Warning

torch.from_numpy() shares memory with the NumPy array. Modifying one changes the other. Use .clone() if you need an independent copy.

Reshaping & Views

Python
x = torch.arange(12)

# Reshape (may copy)
a = x.reshape(3, 4)
b = x.reshape(2, -1)    # -1 = infer → (2, 6)

# View (never copies, requires contiguous)
c = x.view(4, 3)

# Squeeze / Unsqueeze
t = torch.randn(1, 3, 1)
print(t.squeeze().shape)      # (3,)   — remove all dim=1
print(t.unsqueeze(0).shape)   # (1,1,3,1) — add dim at 0

# Permute / Transpose
img = torch.randn(3, 224, 224)   # C,H,W
hwc = img.permute(1, 2, 0)         # H,W,C for matplotlib

Indexing & Broadcasting

Python
m = torch.randn(4, 5)

# Basic indexing
row0 = m[0]           # first row
elem  = m[2, 3]       # element at row 2, col 3
cols  = m[:, 1:3]     # all rows, cols 1-2

# Boolean indexing
mask = m > 0
positives = m[mask]    # flat tensor of positive values

# Broadcasting: (4,5) + (5,) → element-wise add
bias = torch.randn(5)
result = m + bias      # bias broadcast across rows

# Broadcasting rules:
# 1. Align shapes from the right
# 2. Dimensions must be equal OR one of them is 1
# (4,1,3) + (1,5,3) → (4,5,3) ✓
# (4,3)   + (5,3)   → ERROR     ✗

Exercises

Ex 1.1: Create a 5×5 tensor with values 1-25, then extract the 3×3 center sub-matrix.

Solution

t = torch.arange(1, 26).reshape(5, 5)
center = t[1:4, 1:4]
print(center)

Ex 1.2: Given a batch of images (B, C, H, W), compute the mean pixel value per channel.

Solution

imgs = torch.randn(16, 3, 32, 32)
channel_means = imgs.mean(dim=(0, 2, 3))  # shape: (3,)
print(channel_means)

Ex 1.3: Use broadcasting to add a row-vector (1, 5) and a column-vector (3, 1) to get a (3, 5) matrix.

Solution

row = torch.tensor([[1, 2, 3, 4, 5]])
col = torch.tensor([[10], [20], [30]])
result = row + col  # (3, 5)
print(result)

Project: Tensor Statistics Explorer

Build a function that takes any tensor and prints comprehensive statistics:

Python
def tensor_report(t, name="tensor"):
    print(f"═══ {name} ═══")
    print(f"Shape:    {t.shape}")
    print(f"Dtype:    {t.dtype}")
    print(f"Device:   {t.device}")
    print(f"Numel:    {t.numel():,}")
    print(f"Memory:   {t.element_size() * t.numel() / 1024:.1f} KB")
    if t.is_floating_point():
        print(f"Mean:     {t.mean():.4f}")
        print(f"Std:      {t.std():.4f}")
        print(f"Min/Max:  {t.min():.4f} / {t.max():.4f}")
        print(f"Has NaN:  {t.isnan().any()}")
        print(f"Has Inf:  {t.isinf().any()}")

# Test it
x = torch.randn(16, 3, 224, 224)
tensor_report(x, "ImageNet Batch")

Industry: Tensor Operations at Scale

Netflix Recommendation System: Netflix's recommendation engine processes user-item interaction matrices as sparse tensors. A typical matrix has ~200M users × ~15K titles. PyTorch's sparse tensor operations (torch.sparse_coo_tensor) enable efficient matrix factorization on GPUs, computing millions of recommendations in seconds rather than hours.

Why This Matters for AI

Every AI model processes tensors. GPT-4 transforms text into tensors of shape (batch, seq_len, 12288). Stable Diffusion processes image tensors of (batch, 4, 64, 64) in latent space. Understanding shapes, broadcasting, and memory layout is essential — most bugs in deep learning are shape mismatches.

Key Takeaways

Tensors are multi-dimensional arrays — the core data structure in PyTorch
Use factory functions (zeros, randn, arange) for common patterns
view never copies memory; reshape may copy if not contiguous
Broadcasting follows right-alignment rules — dimensions must match or be 1
NumPy interop shares memory — use .clone() for independence