Chapter 16: Generative Models — VAEs and GANs

Bloom's Taxonomy Map for This Chapter

Learning Objectives

By the end of this chapter, you will be able to:

• Distinguish generative models (learn P(x)) from discriminative models (learn P(y|x)) and explain when each paradigm is preferred
• Derive the ELBO (Evidence Lower Bound) loss function as the sum of reconstruction loss and KL divergence, and explain why it is a lower bound on log P(x)
• Implement the reparameterization trick z = μ + ε · σ and explain why it enables gradient flow through stochastic nodes
• Explain the GAN minimax game: min_G max_D 𝔼[log D(x)] + 𝔼[log(1 − D(G(z)))]
• Diagnose GAN training pathologies: mode collapse, vanishing gradients, training instability
• Compare WGAN (Wasserstein distance) with vanilla GAN (Jensen-Shannon divergence) and explain why WGAN provides more stable gradients
• Build a complete VAE and GAN from scratch using TensorFlow/Keras for MNIST generation
• Evaluate generative model quality using FID (Fréchet Inception Distance) and IS (Inception Score)
• Analyze the ethical implications of deepfake technology, especially in the Indian context — elections, misinformation, and legal frameworks
• Apply generative models to practical Indian use cases: virtual try-on, AR filters, content creation

Opening Hook — When Machines Learn to Dream

Core Concepts

16.1 Generative vs. Discriminative Models

Throughout this book, we've been building discriminative models — classifiers that learn the conditional probability P(y|x): "Given an image x, what is the label y?" But what if we flip the question?

Why Go Generative?

16.2 Variational Autoencoders (VAEs)

16.2.1 From Autoencoders to VAEs

Recall from Chapter 15 that a standard autoencoder learns a compressed representation (encoding) of the data. But standard autoencoders have a critical problem for generation: the latent space is not structured. Points between two encodings may decode to garbage.

A Variational Autoencoder (VAE) solves this by forcing the latent space to be smooth and continuous — specifically, by making the encoder output a probability distribution rather than a single point.

16.2.2 The Reparameterization Trick

The key insight that makes VAE training possible:

By moving the randomness into ε (which doesn't depend on the parameters), gradients can now flow through μ and σ back to the encoder weights. This is the most elegant trick in all of deep generative modeling.

16.2.3 The ELBO Loss Function

The VAE's loss function is the Evidence Lower Bound (ELBO), which is a lower bound on the log-likelihood log P(x):

The KL term is the "regularizer" that keeps the latent space smooth. Without it, the VAE degenerates into a regular autoencoder with an unstructured latent space.

16.3 Generative Adversarial Networks (GANs)

16.3.1 The Adversarial Game

If VAEs are the "careful statistician" approach to generation, GANs are the "street artist vs. art critic" approach. The idea is beautifully simple and profoundly powerful:

16.3.2 The Minimax Objective

Let's unpack this formula term by term:

• 𝔼_x~pdata[log D(x)] — Discriminator tries to maximize this: it wants D(x) → 1 for real data (log 1 = 0, the maximum)
• 𝔼_z~pz[log(1 − D(G(z)))] — Discriminator tries to maximize: it wants D(G(z)) → 0 for fake data (log(1−0) = 0). Generator tries to minimize: it wants D(G(z)) → 1 (log(1−1) = −∞)

16.3.3 GAN Training Algorithm

The training alternates between updating D and G:

Algorithm
# GAN Training — Alternating Gradient Updates

for each training iteration:
    # ── Step 1: Train Discriminator ──
    # Sample minibatch of m real examples {x₁, ..., xₘ} from data
    # Sample minibatch of m noise vectors {z₁, ..., zₘ} from p(z)
    # Update D by ASCENDING its stochastic gradient:
    ∇θ_D (1/m) Σ [log D(xᵢ) + log(1 − D(G(zᵢ)))]

    # ── Step 2: Train Generator ──
    # Sample minibatch of m noise vectors {z₁, ..., zₘ} from p(z)
    # Update G by DESCENDING its stochastic gradient:
    ∇θ_G (1/m) Σ log(1 − D(G(zᵢ)))

16.3.4 Mode Collapse — The GAN's Achilles Heel

16.4 GAN Variants & Wasserstein GAN

16.4.1 The Problem with Jensen-Shannon Divergence

The original GAN's optimal discriminator leads to minimizing the Jensen-Shannon Divergence between p_data and p_G:

The problem: when p_data and p_G have non-overlapping supports (very common in high dimensions), JSD is a constant (log 2), providing zero useful gradient. This is why vanilla GANs suffer from training instability.

16.4.2 Wasserstein GAN (WGAN)

The WGAN (Arjovsky et al., 2017) replaces JSD with the Earth Mover's Distance (Wasserstein-1 distance):

Key changes in WGAN vs. vanilla GAN:

16.4.3 A Taxonomy of Important GANs

16.5 Ethics of Generative AI — The Indian Context

From-Scratch Code

4A. Simple GAN for MNIST Digit Generation

We build a complete GAN from scratch using TensorFlow — no high-level libraries, no shortcuts. Every gradient step is visible.

Python / TensorFlow
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

# ── Load MNIST ──
(x_train, _), (_, _) = tf.keras.datasets.mnist.load_data()
x_train = (x_train.astype('float32') - 127.5) / 127.5  # Normalize to [-1, 1]
x_train = x_train.reshape(-1, 784)

NOISE_DIM = 100
BATCH_SIZE = 256
EPOCHS = 200

# ── Generator Network ──
def build_generator():
    model = tf.keras.Sequential([
        tf.keras.layers.Dense(256, input_dim=NOISE_DIM),
        tf.keras.layers.LeakyReLU(0.2),
        tf.keras.layers.BatchNormalization(momentum=0.8),
        tf.keras.layers.Dense(512),
        tf.keras.layers.LeakyReLU(0.2),
        tf.keras.layers.BatchNormalization(momentum=0.8),
        tf.keras.layers.Dense(1024),
        tf.keras.layers.LeakyReLU(0.2),
        tf.keras.layers.BatchNormalization(momentum=0.8),
        tf.keras.layers.Dense(784, activation='tanh'),  # Output in [-1, 1]
    ])
    return model

# ── Discriminator Network ──
def build_discriminator():
    model = tf.keras.Sequential([
        tf.keras.layers.Dense(512, input_dim=784),
        tf.keras.layers.LeakyReLU(0.2),
        tf.keras.layers.Dropout(0.3),
        tf.keras.layers.Dense(256),
        tf.keras.layers.LeakyReLU(0.2),
        tf.keras.layers.Dropout(0.3),
        tf.keras.layers.Dense(1, activation='sigmoid'),  # Real/Fake probability
    ])
    return model

# ── Instantiate ──
generator = build_generator()
discriminator = build_discriminator()
cross_entropy = tf.keras.losses.BinaryCrossentropy()
gen_optimizer = tf.keras.optimizers.Adam(lr=0.0002, beta_1=0.5)
disc_optimizer = tf.keras.optimizers.Adam(lr=0.0002, beta_1=0.5)

# ── Training Step (manual GradientTape) ──
@tf.function
def train_step(real_images):
    noise = tf.random.normal([BATCH_SIZE, NOISE_DIM])

    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:
        # Generator creates fake images
        fake_images = generator(noise, training=True)

        # Discriminator evaluates both
        real_output = discriminator(real_images, training=True)
        fake_output = discriminator(fake_images, training=True)

        # ── Discriminator Loss ──
        # Real images → label 1; Fake images → label 0
        d_loss_real = cross_entropy(tf.ones_like(real_output), real_output)
        d_loss_fake = cross_entropy(tf.zeros_like(fake_output), fake_output)
        d_loss = d_loss_real + d_loss_fake

        # ── Generator Loss ──
        # Generator wants D to output 1 for fake images (non-saturating)
        g_loss = cross_entropy(tf.ones_like(fake_output), fake_output)

    # Compute and apply gradients
    gen_grads = gen_tape.gradient(g_loss, generator.trainable_variables)
    disc_grads = disc_tape.gradient(d_loss, discriminator.trainable_variables)
    gen_optimizer.apply_gradients(zip(gen_grads, generator.trainable_variables))
    disc_optimizer.apply_gradients(zip(disc_grads, discriminator.trainable_variables))

    return d_loss, g_loss

# ── Training Loop ──
dataset = tf.data.Dataset.from_tensor_slices(x_train).shuffle(60000).batch(BATCH_SIZE)

for epoch in range(EPOCHS):
    for batch in dataset:
        d_loss, g_loss = train_step(batch)

    if (epoch + 1) % 20 == 0:
        print(f"Epoch {epoch+1}: D_loss={d_loss:.4f}, G_loss={g_loss:.4f}")
        # Generate and display sample images
        noise = tf.random.normal([16, NOISE_DIM])
        generated = generator(noise, training=False)
        fig, axes = plt.subplots(4, 4, figsize=(4, 4))
        for i, ax in enumerate(axes.flat):
            ax.imshow(generated[i].numpy().reshape(28, 28), cmap='gray')
            ax.axis('off')
        plt.savefig(f'gan_epoch_{epoch+1}.png')
        plt.close()

4B. Variational Autoencoder for MNIST with Latent Space Interpolation

Python / TensorFlow
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

LATENT_DIM = 2  # 2D for visualization
EPOCHS = 50
BATCH_SIZE = 128

# ── Sampling Layer (Reparameterization Trick) ──
class Sampling(tf.keras.layers.Layer):
    """z = mu + eps * sigma (reparameterization trick)"""
    def call(self, inputs):
        mu, log_var = inputs
        # Sample epsilon from N(0, I)
        epsilon = tf.random.normal(shape=tf.shape(mu))
        # z = μ + ε · exp(½ log σ²)
        return mu + tf.exp(0.5 * log_var) * epsilon

# ── Encoder ──
encoder_inputs = tf.keras.Input(shape=(784,))
h = tf.keras.layers.Dense(512, activation='relu')(encoder_inputs)
h = tf.keras.layers.Dense(256, activation='relu')(h)
z_mean = tf.keras.layers.Dense(LATENT_DIM, name='z_mean')(h)
z_log_var = tf.keras.layers.Dense(LATENT_DIM, name='z_log_var')(h)
z = Sampling()([z_mean, z_log_var])
encoder = tf.keras.Model(encoder_inputs, [z_mean, z_log_var, z], name='encoder')

# ── Decoder ──
decoder_inputs = tf.keras.Input(shape=(LATENT_DIM,))
h = tf.keras.layers.Dense(256, activation='relu')(decoder_inputs)
h = tf.keras.layers.Dense(512, activation='relu')(h)
decoder_outputs = tf.keras.layers.Dense(784, activation='sigmoid')(h)
decoder = tf.keras.Model(decoder_inputs, decoder_outputs, name='decoder')

# ── VAE Model with Custom Training ──
class VAE(tf.keras.Model):
    def __init__(self, encoder, decoder):
        super().__init__()
        self.encoder = encoder
        self.decoder = decoder

    def train_step(self, data):
        with tf.GradientTape() as tape:
            z_mean, z_log_var, z = self.encoder(data)
            reconstruction = self.decoder(z)

            # ── Reconstruction Loss (Binary Cross-Entropy) ──
            recon_loss = tf.reduce_mean(
                tf.reduce_sum(
                    tf.keras.losses.binary_crossentropy(data, reconstruction),
                    axis=-1
                )
            )

            # ── KL Divergence Loss ──
            # D_KL = -0.5 * Σ(1 + log(σ²) - μ² - σ²)
            kl_loss = -0.5 * tf.reduce_mean(
                tf.reduce_sum(
                    1 + z_log_var - tf.square(z_mean) - tf.exp(z_log_var),
                    axis=-1
                )
            )

            # ── Total ELBO Loss ──
            total_loss = recon_loss + kl_loss

        grads = tape.gradient(total_loss, self.trainable_weights)
        self.optimizer.apply_gradients(zip(grads, self.trainable_weights))
        return {"loss": total_loss, "recon": recon_loss, "kl": kl_loss}

# ── Train ──
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = x_train.reshape(-1, 784).astype('float32') / 255.0

vae = VAE(encoder, decoder)
vae.compile(optimizer=tf.keras.optimizers.Adam(lr=0.001))
vae.fit(x_train, epochs=EPOCHS, batch_size=BATCH_SIZE)

# ── Visualize Latent Space ──
z_mean, _, _ = encoder.predict(x_train)
plt.figure(figsize=(10, 8))
plt.scatter(z_mean[:, 0], z_mean[:, 1], c=y_train, cmap='tab10', s=1, alpha=0.5)
plt.colorbar()
plt.title('VAE Latent Space — Digit Clusters')
plt.savefig('vae_latent_space.png')

# ── Latent Space Interpolation (Face Morphing Concept) ──
def interpolate(z1, z2, steps=10):
    """Linear interpolation between two latent vectors."""
    ratios = np.linspace(0, 1, steps)
    vectors = np.array([(1 - r) * z1 + r * z2 for r in ratios])
    images = decoder.predict(vectors)
    fig, axes = plt.subplots(1, steps, figsize=(20, 2))
    for i, ax in enumerate(axes):
        ax.imshow(images[i].reshape(28, 28), cmap='gray')
        ax.axis('off')
    plt.suptitle('Latent Space Interpolation: Smooth Morphing')
    plt.savefig('vae_interpolation.png')

# Interpolate between digit "3" and digit "8"
z1 = np.array([[-1.5, 0.5]])   # Approximate location of "3" in latent space
z2 = np.array([[1.0, -1.0]])   # Approximate location of "8" in latent space
interpolate(z1, z2)

Industry Code — DCGAN with TensorFlow/Keras

Production GANs use convolutional architectures (DCGAN). Here's a production-ready implementation with best practices:

Python / TensorFlow
import tensorflow as tf
from tensorflow.keras import layers

# ── DCGAN Generator (Transposed Convolutions) ──
def build_dcgan_generator(latent_dim=100):
    model = tf.keras.Sequential(name='generator')

    # Foundation: 7×7×256 from noise vector
    model.add(layers.Dense(7 * 7 * 256, use_bias=False, input_shape=(latent_dim,)))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU(0.2))
    model.add(layers.Reshape((7, 7, 256)))

    # Upsample: 7×7 → 14×14
    model.add(layers.Conv2DTranspose(128, (5, 5), strides=(2, 2),
                                     padding='same', use_bias=False))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU(0.2))

    # Upsample: 14×14 → 28×28
    model.add(layers.Conv2DTranspose(64, (5, 5), strides=(2, 2),
                                     padding='same', use_bias=False))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU(0.2))

    # Output: 28×28×1 (grayscale image)
    model.add(layers.Conv2DTranspose(1, (5, 5), strides=(1, 1),
                                     padding='same', activation='tanh'))
    return model

# ── DCGAN Discriminator (Strided Convolutions) ──
def build_dcgan_discriminator():
    model = tf.keras.Sequential(name='discriminator')

    model.add(layers.Conv2D(64, (5, 5), strides=(2, 2), padding='same',
                          input_shape=(28, 28, 1)))
    model.add(layers.LeakyReLU(0.2))
    model.add(layers.Dropout(0.3))

    model.add(layers.Conv2D(128, (5, 5), strides=(2, 2), padding='same'))
    model.add(layers.LeakyReLU(0.2))
    model.add(layers.Dropout(0.3))

    model.add(layers.Flatten())
    model.add(layers.Dense(1, activation='sigmoid'))
    return model

# ── DCGAN Training with Best Practices ──
generator = build_dcgan_generator()
discriminator = build_dcgan_discriminator()

# Key DCGAN guidelines from Radford et al. 2015:
# 1. Use strided convolutions (not pooling) in discriminator
# 2. Use transposed convolutions in generator
# 3. BatchNorm in both G and D (except D's input and G's output)
# 4. LeakyReLU in D, ReLU in G (here we use LeakyReLU in both)
# 5. Adam with lr=0.0002, beta1=0.5

gen_optimizer = tf.keras.optimizers.Adam(0.0002, beta_1=0.5)
disc_optimizer = tf.keras.optimizers.Adam(0.0002, beta_1=0.5)

print(generator.summary())
print(f"Generator params:     {generator.count_params():,}")
print(f"Discriminator params: {discriminator.count_params():,}")

Visual Diagrams

6.1 VAE Architecture — End to End

6.2 GAN Architecture — The Adversarial Game

6.3 VAE vs. GAN — Side-by-Side Comparison

6.4 Mode Collapse Visualization

Worked Example — KL Divergence and ELBO Computation

Problem

A VAE encoder outputs the following for a single input image x:

• μ = [0.8, −0.3] (mean of latent distribution)
• log σ² = [−0.5, 0.2] (log-variance of latent distribution)

The prior is p(z) = N(0, I). Compute the KL divergence D_KL(q(z|x) ‖ p(z)).

Step-by-Step Solution

Step 1: Recall the KL Formula for Gaussians

Step 2: Extract Values for Each Dimension

Step 3: Compute Each Dimension's Contribution

Dimension 1:

term₁ = 1 + (−0.5) − 0.64 − 0.6065 = 1 − 0.5 − 0.64 − 0.6065 = −0.7465

Dimension 2:

term₂ = 1 + 0.2 − 0.09 − 1.2214 = 1 + 0.2 − 0.09 − 1.2214 = −0.1114

Step 4: Sum and Negate

D_KL = −½ × (−0.7465 + (−0.1114))

D_KL = −½ × (−0.8579)

Step 5: Interpretation

• The KL divergence is 0.4290 nats (natural log units). This means the encoder's learned distribution is moderately far from the standard normal prior.
• Dimension 1 contributes more to the KL (0.3733) than dimension 2 (0.0557), mainly because μ₁ = 0.8 is further from 0.
• If the encoder output μ = [0, 0] and log σ² = [0, 0], the KL would be exactly 0 — meaning the encoder perfectly matches the prior (but then it learns nothing useful!).

Case Study — Snapchat India AR Filters & Deepfake Detection

Common Mistakes & Misconceptions

Comparison Table

10.1 VAE vs. GAN vs. Diffusion Models — Comprehensive Comparison

10.2 GAN Variants — When to Use What

Exercises

Section A: Multiple Choice Questions (10)

Section B: Short Answer Questions (5)

Section C: Long Answer Questions (3)

Section D: Programming Questions (2)

Section E: Mini-Project

Chapter Summary

Formulas to Remember

What's Next?

In Chapter 17: Attention Mechanisms & Transformers, we'll explore the architecture that revolutionized both NLP and computer vision — the Transformer. The self-attention mechanism at its core has replaced RNNs and is now the foundation of GPT, BERT, and Vision Transformers. Interestingly, modern diffusion models (DALL-E 2, Stable Diffusion) combine the generative principles from this chapter with the Transformer architecture from Chapter 17.

References

Foundational Papers

1. Goodfellow, I., et al. (2014). Generative Adversarial Nets. NeurIPS. — The original GAN paper.
2. Kingma, D. P., & Welling, M. (2013). Auto-Encoding Variational Bayes. ICLR 2014. — The original VAE paper.
3. Radford, A., Metz, L., & Chintala, S. (2015). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. ICLR 2016. — DCGAN paper with architectural guidelines.
4. Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein Generative Adversarial Networks. ICML. — WGAN paper.
5. Gulrajani, I., et al. (2017). Improved Training of Wasserstein GANs. NeurIPS. — WGAN-GP (gradient penalty).
6. Karras, T., Laine, S., & Aila, T. (2019). A Style-Based Generator Architecture for Generative Adversarial Networks. CVPR. — StyleGAN.
7. Isola, P., et al. (2017). Image-to-Image Translation with Conditional Adversarial Networks. CVPR. — Pix2Pix.
8. Zhu, J.-Y., et al. (2017). Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. ICCV. — CycleGAN.

Textbooks

9. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. — Chapter 20: Deep Generative Models.
10. Foster, D. (2023). Generative Deep Learning: Teaching Machines to Paint, Write, Compose, and Play. 2nd Edition. O'Reilly. — Practical guide to VAEs, GANs, and diffusion models.
11. Prince, S. J. D. (2023). Understanding Deep Learning. MIT Press. — Chapters 14–15 on GANs and VAEs.

Indian Context & Ethics

12. BOOM Live. (2024). Deepfake Detection in Indian Elections: A Comprehensive Report. boomlive.in
13. Ministry of Electronics & IT (MeitY). (2023). Digital Personal Data Protection Act, 2023. Government of India.
14. Ministry of Electronics & IT (MeitY). (2024). Advisory on AI Regulation and Labeling Requirements.
15. Information Technology Act, 2000. Section 66D: Punishment for cheating by personation by using computer resource. — Government of India.

Industry & Applications

16. Meesho Engineering Blog. (2023). Building Virtual Try-On for Indian Fashion at Scale.
17. Snap Inc. (2024). Snapchat India: AR and Machine Learning Innovations. Engineering Blog.
18. TCS Research. (2023). Generative Models for Drug Discovery: A Latent Space Approach. Technical Report.

Evaluation Metrics

19. Heusel, M., et al. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. NeurIPS. — FID score paper.
20. Salimans, T., et al. (2016). Improved Techniques for Training GANs. NeurIPS. — Inception Score and training techniques.