Recurrent Neural Networks
& LSTMs

Learning Objectives

Introduction

Consider reading this sentence. You do not understand each word in isolation — the meaning of "bank" changes depending on whether the preceding words were "river" or "investment." Your brain processes text sequentially, carrying a running summary of what came before. Feedforward neural networks (Chapter 12) cannot do this: each input is processed independently, with no memory of the past.

Recurrent Neural Networks (RNNs) solve this by introducing loops in the network, allowing information to persist from one time step to the next. They were the first neural architecture capable of handling variable-length sequences — sentences, audio waveforms, stock prices, DNA sequences — by maintaining a hidden state that encodes a compressed history of everything seen so far.

However, vanilla RNNs have a critical flaw: they struggle to learn long-range dependencies. The meaning of a word at position 1 might be crucial for understanding word 50, but the gradient signal carrying this information vanishes exponentially during training. Long Short-Term Memory (LSTM) networks, invented by Hochreiter & Schmidhuber in 1997, solve this with a gating mechanism that controls what information to remember, what to forget, and what to output.

In this chapter, we will build RNNs and LSTMs from first principles — starting with the math, proving why gradients vanish, deriving every LSTM gate equation, and then implementing everything in both raw NumPy and TensorFlow. Along the way, we will forecast Nifty50 stock prices, generate Hindi text, and understand the systems behind Google Translate and voice assistants.

Historical Background

Timeline of Sequence Modeling

Conceptual Explanation

4.1 Why Sequences Need Special Networks

Consider three challenges that feedforward networks cannot handle:

• Variable length: Sentences have different lengths. A feedforward net needs a fixed-size input.
• Order matters: "Dog bites man" ≠ "Man bites dog" — but a bag-of-words feedforward net treats them identically.
• Long-range dependencies: "The cat, which sat on the mat and played with the yarn, was happy" — the verb "was" must agree with "cat" many words earlier.

4.2 The Core Idea: Hidden State as Memory

An RNN processes one element of the sequence at a time. At each time step t, it receives input x_t and combines it with the hidden state from the previous time step h_{t-1} to produce a new hidden state h_t. Think of h_t as a "compressed summary" of everything the network has seen from time step 1 through time step t.

4.3 Vanilla RNN

The simplest RNN uses a single recurrence equation:

Where: W_hh = hidden-to-hidden weights, W_xh = input-to-hidden weights, W_hy = hidden-to-output weights. The same weights are shared across all time steps — this is called weight sharing and is what makes RNNs parameter-efficient.

4.4 The Vanishing Gradient Problem

During backpropagation through time (BPTT), gradients must flow backward through many time steps. Each step multiplies by the weight matrix W_hh and the derivative of tanh. Since |tanh'(x)| ≤ 1, after multiplying many values less than 1, the gradient shrinks exponentially → vanishing gradient. Conversely, if eigenvalues of W_hh > 1, gradients grow exponentially → exploding gradient.

4.5 LSTM: The Solution

LSTM introduces a separate cell state C_t that runs like a conveyor belt through time, with gates that control information flow:

• Forget gate (f_t): Decides what to erase from cell state
• Input gate (i_t): Decides what new info to write
• Candidate gate (C̃_t): Creates candidate values to add
• Output gate (o_t): Decides what to output from cell state

4.6 GRU: Simplified LSTM

The Gated Recurrent Unit merges the forget and input gates into a single update gate, and uses a reset gate to control how much past information to forget. It has fewer parameters and trains faster, with comparable performance on many tasks.

4.7 Bidirectional RNNs

In many tasks (like named entity recognition), the meaning of a word depends on both preceding and following context. A Bidirectional RNN runs two separate RNNs — one forward, one backward — and concatenates their hidden states.

4.8 Sequence-to-Sequence (Seq2Seq)

For tasks where input and output sequences have different lengths (e.g., translation from English to Hindi), we use an encoder-decoder architecture: the encoder RNN compresses the input into a fixed-size context vector, and the decoder RNN generates the output sequence from this vector.

Mathematical Foundation

5.1 Vanilla RNN Equations

Given an input sequence (x_1, x_2, ..., x_T) where x_t ∈ ℝ^d, hidden state h_t ∈ ℝ^n:

5.2 LSTM Full Equations

Let [h_{t-1}, x_t] denote the concatenation of previous hidden state and current input. The LSTM computes:

Where σ is the sigmoid function and ⊙ is element-wise (Hadamard) product.

5.3 GRU Equations

5.4 Parameter Count Comparison

5.5 Bidirectional RNN

Formula Derivations

6.1 Deriving Backpropagation Through Time (BPTT)

We derive BPTT from first principles. The total loss across all T time steps is:

The hidden state at time t is:

To compute ∂L/∂W_hh, we apply the chain rule. Since W_hh affects all future losses through the hidden state chain:

The key term is the Jacobian product:

Here, diag(1 - h_i²) is the diagonal matrix of tanh derivatives. This product of matrices is the source of the vanishing/exploding gradient.

6.2 Proving the Vanishing Gradient

Theorem: For a vanilla RNN with hidden state dimension n, the gradient ∂h_t/∂h_k decays exponentially as (t-k) increases.

Proof:

6.3 Why LSTM Solves Vanishing Gradients

In LSTM, the cell state gradient flows through:

Crucially, the forget gate f_t ∈ (0,1) is learned. When the network learns to set f_t ≈ 1, gradients flow perfectly with no decay. This is the "constant error carousel" — the cell state acts as a highway for gradient flow. Unlike vanilla RNN where gradients must pass through tanh and a fixed weight matrix, LSTM gradients pass through learned gate values that can be close to 1.

6.4 Deriving GRU from LSTM

GRU simplifies LSTM by:

1. Merging forget gate and input gate into one update gate: z_t (where input = z_t, forget = 1-z_t)
2. Removing the separate cell state — the hidden state serves both roles
3. Adding a reset gate to control past hidden state influence on candidates

Worked Numerical Examples

Example 1: RNN Forward Pass (3 Time Steps)

Example 2: LSTM Gate Computation (1 Time Step)

Visual Diagrams

Diagram 1: Vanilla RNN Unrolled

Diagram 2: LSTM Cell Architecture

Diagram 3: GRU Cell

Diagram 4: Seq2Seq Encoder-Decoder

Flowcharts

Flowchart 1: Choosing RNN Architecture

Flowchart 2: BPTT Training Algorithm

Python Implementation (From Scratch)

10.1 Vanilla RNN in NumPy

import numpy as np

class VanillaRNN:
    """
    Vanilla RNN implementation from scratch using NumPy.
    h_t = tanh(W_hh @ h_{t-1} + W_xh @ x_t + b_h)
    y_t = W_hy @ h_t + b_y
    """
    def __init__(self, input_dim, hidden_dim, output_dim):
        self.hidden_dim = hidden_dim
        # Xavier initialization
        scale_xh = np.sqrt(2.0 / (input_dim + hidden_dim))
        scale_hh = np.sqrt(2.0 / (hidden_dim + hidden_dim))
        scale_hy = np.sqrt(2.0 / (hidden_dim + output_dim))

        self.W_xh = np.random.randn(hidden_dim, input_dim) * scale_xh
        self.W_hh = np.random.randn(hidden_dim, hidden_dim) * scale_hh
        self.b_h = np.zeros((hidden_dim, 1))

        self.W_hy = np.random.randn(output_dim, hidden_dim) * scale_hy
        self.b_y = np.zeros((output_dim, 1))

    def forward(self, inputs, h_prev=None):
        """
        Forward pass through the entire sequence.
        inputs: list of column vectors [x_1, x_2, ..., x_T]
        Returns: outputs, hidden_states
        """
        if h_prev is None:
            h_prev = np.zeros((self.hidden_dim, 1))

        self.inputs = inputs
        self.hidden_states = {0: h_prev}
        outputs = []

        for t in range(1, len(inputs) + 1):
            x_t = inputs[t - 1]
            # Core RNN equation
            z_t = self.W_hh @ self.hidden_states[t-1] + self.W_xh @ x_t + self.b_h
            h_t = np.tanh(z_t)
            y_t = self.W_hy @ h_t + self.b_y

            self.hidden_states[t] = h_t
            outputs.append(y_t)

        return outputs, self.hidden_states

    def backward(self, d_outputs, learning_rate=0.001):
        """
        Backpropagation Through Time (BPTT).
        d_outputs: list of gradients dL/dy_t for each time step
        """
        T = len(d_outputs)
        dW_xh = np.zeros_like(self.W_xh)
        dW_hh = np.zeros_like(self.W_hh)
        db_h = np.zeros_like(self.b_h)
        dW_hy = np.zeros_like(self.W_hy)
        db_y = np.zeros_like(self.b_y)

        dh_next = np.zeros((self.hidden_dim, 1))

        for t in reversed(range(1, T + 1)):
            dy = d_outputs[t - 1]
            # Gradient from output layer
            dW_hy += dy @ self.hidden_states[t].T
            db_y += dy

            # Gradient into hidden state
            dh = self.W_hy.T @ dy + dh_next

            # Backprop through tanh: dtanh/dz = 1 - tanh^2
            dz = dh * (1 - self.hidden_states[t] ** 2)

            # Accumulate gradients
            dW_xh += dz @ self.inputs[t - 1].T
            dW_hh += dz @ self.hidden_states[t - 1].T
            db_h += dz

            # Gradient flowing to previous time step
            dh_next = self.W_hh.T @ dz

        # Gradient clipping to prevent explosion
        for grad in [dW_xh, dW_hh, db_h, dW_hy, db_y]:
            np.clip(grad, -5, 5, out=grad)

        # Update weights
        self.W_xh -= learning_rate * dW_xh
        self.W_hh -= learning_rate * dW_hh
        self.b_h -= learning_rate * db_h
        self.W_hy -= learning_rate * dW_hy
        self.b_y -= learning_rate * db_y


# ========== Demo: Character-level language model ==========
# Simple example with tiny vocab
text = "hello world hello"
chars = sorted(list(set(text)))
char_to_idx = {c: i for i, c in enumerate(chars)}
idx_to_char = {i: c for i, c in enumerate(chars)}
vocab_size = len(chars)

# Create training pairs
inputs_idx = [char_to_idx[c] for c in text[:-1]]
targets_idx = [char_to_idx[c] for c in text[1:]]

rnn = VanillaRNN(input_dim=vocab_size, hidden_dim=16, output_dim=vocab_size)

# Training loop
for epoch in range(200):
    # One-hot encode
    xs = [np.eye(vocab_size)[:, [i]] for i in inputs_idx]
    ys_true = targets_idx

    # Forward pass
    outputs, _ = rnn.forward(xs)

    # Compute softmax + cross-entropy loss
    loss = 0
    d_outputs = []
    for t in range(len(outputs)):
        # Softmax
        exp_y = np.exp(outputs[t] - np.max(outputs[t]))
        probs = exp_y / np.sum(exp_y)
        loss -= np.log(probs[ys_true[t], 0] + 1e-8)
        # Gradient of cross-entropy + softmax
        dy = probs.copy()
        dy[ys_true[t]] -= 1
        d_outputs.append(dy)

    # Backward pass
    rnn.backward(d_outputs, learning_rate=0.01)

    if epoch % 50 == 0:
        print(f"Epoch {epoch}, Loss: {loss:.4f}")

print("Training complete!")

10.2 LSTM in NumPy

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-np.clip(x, -500, 500)))

def sigmoid_derivative(s):
    return s * (1 - s)

def tanh_derivative(t):
    return 1 - t ** 2


class LSTMCell:
    """
    Single LSTM Cell implementation from scratch.
    Implements all 4 gates: forget, input, candidate, output.
    """
    def __init__(self, input_dim, hidden_dim):
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        concat_dim = input_dim + hidden_dim
        scale = np.sqrt(2.0 / concat_dim)

        # Forget gate parameters
        self.W_f = np.random.randn(hidden_dim, concat_dim) * scale
        self.b_f = np.ones((hidden_dim, 1))  # bias=1 for forget gate (important!)

        # Input gate parameters
        self.W_i = np.random.randn(hidden_dim, concat_dim) * scale
        self.b_i = np.zeros((hidden_dim, 1))

        # Candidate parameters
        self.W_c = np.random.randn(hidden_dim, concat_dim) * scale
        self.b_c = np.zeros((hidden_dim, 1))

        # Output gate parameters
        self.W_o = np.random.randn(hidden_dim, concat_dim) * scale
        self.b_o = np.zeros((hidden_dim, 1))

    def forward(self, x_t, h_prev, c_prev):
        """Single time step forward pass."""
        # Concatenate [h_{t-1}, x_t]
        concat = np.vstack([h_prev, x_t])

        # Forget gate: what to erase from cell state
        f_t = sigmoid(self.W_f @ concat + self.b_f)

        # Input gate: what new info to write
        i_t = sigmoid(self.W_i @ concat + self.b_i)

        # Candidate cell state
        c_tilde = np.tanh(self.W_c @ concat + self.b_c)

        # New cell state
        c_t = f_t * c_prev + i_t * c_tilde

        # Output gate: what to output
        o_t = sigmoid(self.W_o @ concat + self.b_o)

        # New hidden state
        h_t = o_t * np.tanh(c_t)

        # Cache for backward pass
        cache = (concat, f_t, i_t, c_tilde, c_t, o_t, h_prev, c_prev, x_t)
        return h_t, c_t, cache

    def backward(self, dh_t, dc_t, cache):
        """Single time step backward pass."""
        concat, f_t, i_t, c_tilde, c_t, o_t, h_prev, c_prev, x_t = cache

        # Gradient through output gate
        tanh_c_t = np.tanh(c_t)
        do_t = dh_t * tanh_c_t
        dc_t += dh_t * o_t * tanh_derivative(tanh_c_t)

        # Gradient through cell state update
        df_t = dc_t * c_prev
        di_t = dc_t * c_tilde
        dc_tilde = dc_t * i_t
        dc_prev = dc_t * f_t

        # Gradient through activations
        df_raw = df_t * sigmoid_derivative(f_t)
        di_raw = di_t * sigmoid_derivative(i_t)
        dc_raw = dc_tilde * tanh_derivative(c_tilde)
        do_raw = do_t * sigmoid_derivative(o_t)

        # Weight gradients
        dW_f = df_raw @ concat.T
        dW_i = di_raw @ concat.T
        dW_c = dc_raw @ concat.T
        dW_o = do_raw @ concat.T
        db_f = df_raw
        db_i = di_raw
        db_c = dc_raw
        db_o = do_raw

        # Gradient to concat = [h_prev, x_t]
        d_concat = (self.W_f.T @ df_raw + self.W_i.T @ di_raw +
                    self.W_c.T @ dc_raw + self.W_o.T @ do_raw)

        dh_prev = d_concat[:self.hidden_dim]
        dx_t = d_concat[self.hidden_dim:]

        grads = {
            'dW_f': dW_f, 'dW_i': dW_i, 'dW_c': dW_c, 'dW_o': dW_o,
            'db_f': db_f, 'db_i': db_i, 'db_c': db_c, 'db_o': db_o
        }
        return dh_prev, dc_prev, dx_t, grads


class LSTM:
    """Full LSTM for sequence processing."""
    def __init__(self, input_dim, hidden_dim, output_dim):
        self.cell = LSTMCell(input_dim, hidden_dim)
        self.hidden_dim = hidden_dim
        scale = np.sqrt(2.0 / (hidden_dim + output_dim))
        self.W_y = np.random.randn(output_dim, hidden_dim) * scale
        self.b_y = np.zeros((output_dim, 1))

    def forward(self, inputs):
        """Process entire sequence."""
        T = len(inputs)
        h = np.zeros((self.hidden_dim, 1))
        c = np.zeros((self.hidden_dim, 1))

        self.caches = []
        self.h_states = [h]
        outputs = []

        for t in range(T):
            h, c, cache = self.cell.forward(inputs[t], h, c)
            self.caches.append(cache)
            self.h_states.append(h)
            y = self.W_y @ h + self.b_y
            outputs.append(y)

        return outputs

    def predict_sequence(self, seed_input, length, temperature=1.0):
        """Generate a sequence given a seed."""
        h = np.zeros((self.hidden_dim, 1))
        c = np.zeros((self.hidden_dim, 1))
        x = seed_input
        generated = []

        for _ in range(length):
            h, c, _ = self.cell.forward(x, h, c)
            y = self.W_y @ h + self.b_y
            # Temperature-scaled softmax
            y = y / temperature
            exp_y = np.exp(y - np.max(y))
            probs = exp_y / np.sum(exp_y)
            idx = np.random.choice(len(probs.flatten()), p=probs.flatten())
            generated.append(idx)
            # Next input is one-hot of predicted char
            x = np.zeros_like(seed_input)
            x[idx] = 1
        return generated


# ========== Demo: LSTM on simple sequence ==========
print("=== LSTM Forward Pass Demo ===")
lstm_cell = LSTMCell(input_dim=3, hidden_dim=4)
h = np.zeros((4, 1))
c = np.zeros((4, 1))

# Process 3 time steps
for t in range(3):
    x = np.random.randn(3, 1)
    h, c, cache = lstm_cell.forward(x, h, c)
    print(f"t={t+1}: h={h.flatten()[:3].round(4)}... c={c.flatten()[:3].round(4)}...")

print("\nLSTM cell maintains separate h and c states!")

10.3 GRU in NumPy

class GRUCell:
    """GRU Cell: simplified LSTM with update + reset gates."""
    def __init__(self, input_dim, hidden_dim):
        self.hidden_dim = hidden_dim
        concat_dim = input_dim + hidden_dim
        scale = np.sqrt(2.0 / concat_dim)

        # Update gate (merges forget + input)
        self.W_z = np.random.randn(hidden_dim, concat_dim) * scale
        self.b_z = np.zeros((hidden_dim, 1))

        # Reset gate
        self.W_r = np.random.randn(hidden_dim, concat_dim) * scale
        self.b_r = np.zeros((hidden_dim, 1))

        # Candidate hidden state
        self.W_h = np.random.randn(hidden_dim, concat_dim) * scale
        self.b_h = np.zeros((hidden_dim, 1))

    def forward(self, x_t, h_prev):
        concat = np.vstack([h_prev, x_t])

        # Update gate: how much to keep from old state
        z_t = sigmoid(self.W_z @ concat + self.b_z)

        # Reset gate: how much past to use for candidate
        r_t = sigmoid(self.W_r @ concat + self.b_r)

        # Candidate with reset applied
        concat_reset = np.vstack([r_t * h_prev, x_t])
        h_tilde = np.tanh(self.W_h @ concat_reset + self.b_h)

        # Final hidden state: interpolation
        h_t = (1 - z_t) * h_prev + z_t * h_tilde

        return h_t

# Demo
print("\n=== GRU Forward Pass Demo ===")
gru = GRUCell(input_dim=3, hidden_dim=4)
h = np.zeros((4, 1))
for t in range(3):
    x = np.random.randn(3, 1)
    h = gru.forward(x, h)
    print(f"t={t+1}: h={h.flatten().round(4)}")

TensorFlow Implementation

11.1 Text Generation with LSTM

import tensorflow as tf
import numpy as np

# ========== Text Generation with LSTM ==========
# Sample text (use a larger corpus in practice)
text = """India is a land of diversity. From the Himalayas in the north
to the beaches of Kerala in the south, every region has its own culture.
The country is home to over a billion people speaking hundreds of languages."""

# Character-level tokenization
chars = sorted(list(set(text)))
char_to_idx = {c: i for i, c in enumerate(chars)}
idx_to_char = {i: c for i, c in enumerate(chars)}
vocab_size = len(chars)
print(f"Vocabulary size: {vocab_size} unique characters")

# Create training sequences
seq_length = 40
X_data, y_data = [], []
for i in range(len(text) - seq_length):
    X_data.append([char_to_idx[c] for c in text[i:i+seq_length]])
    y_data.append(char_to_idx[text[i+seq_length]])

X = tf.keras.utils.to_categorical(X_data, num_classes=vocab_size)
y = tf.keras.utils.to_categorical(y_data, num_classes=vocab_size)
print(f"Training samples: {len(X_data)}")

# Build LSTM model
model = tf.keras.Sequential([
    tf.keras.layers.LSTM(128, input_shape=(seq_length, vocab_size),
                         return_sequences=True),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.LSTM(128),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(vocab_size, activation='softmax')
])

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    loss='categorical_crossentropy',
    metrics=['accuracy']
)

model.summary()

# Train
history = model.fit(X, y, epochs=50, batch_size=32, verbose=1)

# Text generation function
def generate_text(model, seed_text, length=200, temperature=0.8):
    """Generate text character by character."""
    generated = seed_text
    for _ in range(length):
        # Encode the last seq_length characters
        x_pred = [char_to_idx.get(c, 0) for c in generated[-seq_length:]]
        x_pred = tf.keras.utils.to_categorical([x_pred], num_classes=vocab_size)

        # Predict next character
        probs = model.predict(x_pred, verbose=0)[0]

        # Temperature sampling
        probs = np.log(probs + 1e-8) / temperature
        exp_probs = np.exp(probs)
        probs = exp_probs / np.sum(exp_probs)

        next_idx = np.random.choice(len(probs), p=probs)
        generated += idx_to_char[next_idx]

    return generated

# Generate sample text
seed = text[:seq_length]
print("\n=== Generated Text ===")
print(generate_text(model, seed, length=200))

11.2 Stock Price Prediction with LSTM

import tensorflow as tf
import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_absolute_error, mean_squared_error

# ========== Nifty50 Stock Price Prediction ==========
# In production, load from NSE API or CSV
# Here we simulate realistic Nifty50 data
np.random.seed(42)
dates = pd.date_range('2020-01-01', periods=1000, freq='B')  # Business days
# Simulate with trend + seasonality + noise
trend = np.linspace(11000, 22000, 1000)
seasonal = 500 * np.sin(np.linspace(0, 8*np.pi, 1000))
noise = np.random.randn(1000) * 200
nifty_data = trend + seasonal + noise

df = pd.DataFrame({'Date': dates, 'Close': nifty_data})
print(f"Dataset: {len(df)} trading days")
print(f"Price range: {df['Close'].min():.0f} - {df['Close'].max():.0f}")

# Normalize
scaler = MinMaxScaler(feature_range=(0, 1))
scaled_data = scaler.fit_transform(df['Close'].values.reshape(-1, 1))

# Create sequences
def create_sequences(data, lookback=60):
    X, y = [], []
    for i in range(lookback, len(data)):
        X.append(data[i-lookback:i, 0])
        y.append(data[i, 0])
    return np.array(X), np.array(y)

lookback = 60
X, y = create_sequences(scaled_data, lookback)
X = X.reshape(X.shape[0], X.shape[1], 1)  # Add feature dimension

# Train/test split (80/20)
split = int(0.8 * len(X))
X_train, X_test = X[:split], X[split:]
y_train, y_test = y[:split], y[split:]
print(f"Train: {len(X_train)}, Test: {len(X_test)}")

# Build Stacked LSTM model
model = tf.keras.Sequential([
    tf.keras.layers.LSTM(64, return_sequences=True,
                         input_shape=(lookback, 1)),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.LSTM(64, return_sequences=True),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.LSTM(32),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(16, activation='relu'),
    tf.keras.layers.Dense(1)
])

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    loss='mse',
    metrics=['mae']
)

model.summary()

# Train with early stopping
early_stop = tf.keras.callbacks.EarlyStopping(
    monitor='val_loss', patience=10, restore_best_weights=True
)

history = model.fit(
    X_train, y_train,
    epochs=100,
    batch_size=32,
    validation_split=0.1,
    callbacks=[early_stop],
    verbose=1
)

# Predict
y_pred = model.predict(X_test)

# Inverse transform
y_test_actual = scaler.inverse_transform(y_test.reshape(-1, 1))
y_pred_actual = scaler.inverse_transform(y_pred)

# Metrics
mae = mean_absolute_error(y_test_actual, y_pred_actual)
rmse = np.sqrt(mean_squared_error(y_test_actual, y_pred_actual))
mape = np.mean(np.abs((y_test_actual - y_pred_actual) / y_test_actual)) * 100

print(f"\n=== Results ===")
print(f"MAE:  ₹{mae:.2f}")
print(f"RMSE: ₹{rmse:.2f}")
print(f"MAPE: {mape:.2f}%")

11.3 Bidirectional LSTM for Sentiment Analysis

# Bidirectional LSTM for text classification
model_bilstm = tf.keras.Sequential([
    tf.keras.layers.Embedding(input_dim=10000, output_dim=128,
                               input_length=200),
    tf.keras.layers.Bidirectional(
        tf.keras.layers.LSTM(64, return_sequences=True)
    ),
    tf.keras.layers.Bidirectional(
        tf.keras.layers.LSTM(32)
    ),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

model_bilstm.compile(
    optimizer='adam',
    loss='binary_crossentropy',
    metrics=['accuracy']
)

model_bilstm.summary()
# Train on IMDB or your own Hindi/English sentiment dataset

11.4 GRU Comparison

# GRU — fewer parameters, often comparable performance
model_gru = tf.keras.Sequential([
    tf.keras.layers.GRU(64, return_sequences=True,
                        input_shape=(lookback, 1)),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.GRU(32),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(1)
])

model_gru.compile(optimizer='adam', loss='mse', metrics=['mae'])
model_gru.summary()

# Compare parameter counts
print(f"\nLSTM params: {model.count_params():,}")
print(f"GRU  params: {model_gru.count_params():,}")
print(f"GRU saves {(1 - model_gru.count_params()/model.count_params())*100:.1f}% parameters")

Scikit-Learn Integration

While scikit-learn doesn't natively support RNNs, we can wrap TensorFlow/Keras models in a scikit-learn compatible interface for use in pipelines, cross-validation, and hyperparameter tuning.

from sklearn.base import BaseEstimator, RegressorMixin
from sklearn.model_selection import TimeSeriesSplit
from sklearn.metrics import mean_squared_error
import numpy as np

class LSTMRegressor(BaseEstimator, RegressorMixin):
    """Scikit-learn compatible LSTM wrapper for time series."""

    def __init__(self, lookback=60, units=64, epochs=50,
                 batch_size=32, learning_rate=0.001):
        self.lookback = lookback
        self.units = units
        self.epochs = epochs
        self.batch_size = batch_size
        self.learning_rate = learning_rate

    def _build_model(self, input_shape):
        import tensorflow as tf
        model = tf.keras.Sequential([
            tf.keras.layers.LSTM(self.units, input_shape=input_shape),
            tf.keras.layers.Dense(1)
        ])
        model.compile(
            optimizer=tf.keras.optimizers.Adam(lr=self.learning_rate),
            loss='mse'
        )
        return model

    def fit(self, X, y):
        self.model_ = self._build_model((X.shape[1], X.shape[2]))
        self.model_.fit(X, y, epochs=self.epochs,
                       batch_size=self.batch_size, verbose=0)
        return self

    def predict(self, X):
        return self.model_.predict(X, verbose=0).flatten()

    def score(self, X, y):
        y_pred = self.predict(X)
        return -mean_squared_error(y, y_pred)  # Negative MSE for sklearn


# Time Series Cross-Validation
tscv = TimeSeriesSplit(n_splits=5)
lstm_reg = LSTMRegressor(lookback=60, units=32, epochs=20)

scores = []
for fold, (train_idx, val_idx) in enumerate(tscv.split(X)):
    X_tr, X_val = X[train_idx], X[val_idx]
    y_tr, y_val = y[train_idx], y[val_idx]

    lstm_reg.fit(X_tr, y_tr)
    score = lstm_reg.score(X_val, y_val)
    scores.append(-score)  # Convert back to positive MSE
    print(f"Fold {fold+1}: MSE = {-score:.6f}")

print(f"\nMean MSE: {np.mean(scores):.6f} ± {np.std(scores):.6f}")

Indian Case Studies

Global Case Studies

Startup Applications

Government Applications

Industry Applications

Mini Projects

import tensorflow as tf
import numpy as np

# ========== Hindi Text Generator ==========
# Sample Hindi text (use larger corpus in production)
hindi_text = """भारत एक विशाल देश है। यहाँ अनेक भाषाएँ बोली जाती हैं।
हिंदी भारत की राजभाषा है। भारत की संस्कृति बहुत प्राचीन है।
यहाँ के लोग मेहनती और दयालु हैं। भारत में अनेक त्योहार मनाए जाते हैं।
दीपावली, होली, ईद, क्रिसमस सभी धर्मों के त्योहार मनाए जाते हैं।
भारत की अर्थव्यवस्था तेजी से बढ़ रही है। प्रौद्योगिकी क्षेत्र में भारत अग्रणी है।"""

# Character-level tokenization for Hindi
chars = sorted(list(set(hindi_text)))
char_to_idx = {c: i for i, c in enumerate(chars)}
idx_to_char = {i: c for i, c in enumerate(chars)}
vocab_size = len(chars)
print(f"Hindi vocab size: {vocab_size} characters")
print(f"Sample chars: {chars[:20]}")

# Prepare training data
seq_length = 30  # Shorter for Hindi due to character density
X_data, y_data = [], []
for i in range(len(hindi_text) - seq_length):
    seq_in = hindi_text[i:i + seq_length]
    seq_out = hindi_text[i + seq_length]
    X_data.append([char_to_idx[c] for c in seq_in])
    y_data.append(char_to_idx[seq_out])

X = np.array(X_data)
y = tf.keras.utils.to_categorical(y_data, num_classes=vocab_size)

# Reshape for LSTM: (samples, timesteps, features)
X = X.reshape(X.shape[0], X.shape[1], 1) / float(vocab_size)

# Build model
model = tf.keras.Sequential([
    tf.keras.layers.LSTM(256, input_shape=(seq_length, 1),
                         return_sequences=True),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.LSTM(256),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(vocab_size, activation='softmax')
])

model.compile(loss='categorical_crossentropy',
              optimizer='adam', metrics=['accuracy'])

# Train
model.fit(X, y, epochs=100, batch_size=64, verbose=1)

# Generate Hindi text
def generate_hindi(model, seed_text, length=200, temperature=0.7):
    generated = seed_text
    pattern = [char_to_idx[c] for c in seed_text[-seq_length:]]

    for _ in range(length):
        x = np.array(pattern).reshape(1, seq_length, 1) / float(vocab_size)
        probs = model.predict(x, verbose=0)[0]

        # Temperature sampling
        probs = np.log(probs + 1e-8) / temperature
        exp_probs = np.exp(probs)
        probs = exp_probs / np.sum(exp_probs)

        next_idx = np.random.choice(vocab_size, p=probs)
        generated += idx_to_char[next_idx]
        pattern = pattern[1:] + [next_idx]

    return generated

# Generate
seed = hindi_text[:seq_length]
print("\n=== Generated Hindi Text ===")
print(generate_hindi(model, seed, length=300))

import numpy as np
import tensorflow as tf
from sklearn.preprocessing import MinMaxScaler
import json

class StockPredictor:
    """End-to-end LSTM stock prediction system."""

    def __init__(self, lookback=60, units=64, epochs=50):
        self.lookback = lookback
        self.units = units
        self.epochs = epochs
        self.scaler = MinMaxScaler()
        self.model = None

    def prepare_data(self, prices):
        """Scale and create sequences."""
        scaled = self.scaler.fit_transform(prices.reshape(-1, 1))
        X, y = [], []
        for i in range(self.lookback, len(scaled)):
            X.append(scaled[i-self.lookback:i, 0])
            y.append(scaled[i, 0])
        X = np.array(X).reshape(-1, self.lookback, 1)
        y = np.array(y)
        return X, y

    def build_model(self):
        """Build stacked LSTM."""
        self.model = tf.keras.Sequential([
            tf.keras.layers.LSTM(self.units, return_sequences=True,
                                input_shape=(self.lookback, 1)),
            tf.keras.layers.Dropout(0.2),
            tf.keras.layers.LSTM(self.units // 2),
            tf.keras.layers.Dropout(0.2),
            tf.keras.layers.Dense(1)
        ])
        self.model.compile(optimizer='adam', loss='mse')

    def walk_forward_validate(self, prices, n_splits=5):
        """Walk-forward validation — proper time series CV."""
        X, y = self.prepare_data(prices)
        fold_size = len(X) // (n_splits + 1)
        results = []

        for fold in range(n_splits):
            train_end = fold_size * (fold + 2)
            test_end = min(train_end + fold_size, len(X))

            X_train = X[:train_end]
            y_train = y[:train_end]
            X_test = X[train_end:test_end]
            y_test = y[train_end:test_end]

            self.build_model()
            self.model.fit(X_train, y_train, epochs=self.epochs,
                          batch_size=32, verbose=0)

            y_pred = self.model.predict(X_test, verbose=0).flatten()

            # Directional accuracy
            actual_dir = np.sign(np.diff(y_test))
            pred_dir = np.sign(np.diff(y_pred))
            dir_acc = np.mean(actual_dir == pred_dir)

            mse = np.mean((y_test - y_pred) ** 2)
            results.append({'fold': fold+1, 'mse': mse,
                           'dir_accuracy': dir_acc})
            print(f"Fold {fold+1}: MSE={mse:.6f}, "
                  f"Direction Accuracy={dir_acc:.2%}")

        return results

    def predict_next(self, prices, n_days=5):
        """Predict next n days."""
        X, y = self.prepare_data(prices)
        self.build_model()
        self.model.fit(X, y, epochs=self.epochs, batch_size=32, verbose=0)

        # Recursive prediction
        last_seq = X[-1:].copy()
        predictions = []

        for _ in range(n_days):
            pred = self.model.predict(last_seq, verbose=0)[0, 0]
            predictions.append(pred)
            # Shift window
            last_seq = np.roll(last_seq, -1, axis=1)
            last_seq[0, -1, 0] = pred

        # Inverse scale
        pred_prices = self.scaler.inverse_transform(
            np.array(predictions).reshape(-1, 1)
        ).flatten()

        return pred_prices


# ========== Usage ==========
# Simulate Nifty50 data
np.random.seed(42)
prices = np.cumsum(np.random.randn(500)) + 18000
prices = np.abs(prices)  # Ensure positive

predictor = StockPredictor(lookback=30, units=32, epochs=30)

# Walk-forward validation
print("=== Walk-Forward Validation ===")
results = predictor.walk_forward_validate(prices, n_splits=3)

# Predict next 5 days
print("\n=== 5-Day Forecast ===")
next_prices = predictor.predict_next(prices, n_days=5)
for i, p in enumerate(next_prices):
    print(f"Day {i+1}: ₹{p:,.2f}")

import tensorflow as tf
import numpy as np

# Sample transliteration pairs
pairs = [
    ("rahul", "राहुल"), ("priya", "प्रिया"), ("amit", "अमित"),
    ("neha", "नेहा"), ("vijay", "विजय"), ("sunita", "सुनीता"),
    ("deepak", "दीपक"), ("anita", "अनिता"), ("suresh", "सुरेश"),
    ("kavita", "कविता"), ("rajesh", "राजेश"), ("pooja", "पूजा"),
]

# Build character vocabularies
eng_chars = sorted(set(''.join([p[0] for p in pairs]))) + ['', '', '']
hin_chars = sorted(set(''.join([p[1] for p in pairs]))) + ['', '', '']

eng_to_idx = {c: i for i, c in enumerate(eng_chars)}
hin_to_idx = {c: i for i, c in enumerate(hin_chars)}
idx_to_hin = {i: c for c, i in hin_to_idx.items()}

# Encode sequences
max_eng = max(len(p[0]) for p in pairs) + 2
max_hin = max(len(p[1]) for p in pairs) + 2

encoder_input = np.zeros((len(pairs), max_eng, len(eng_chars)))
decoder_input = np.zeros((len(pairs), max_hin, len(hin_chars)))
decoder_target = np.zeros((len(pairs), max_hin, len(hin_chars)))

for i, (eng, hin) in enumerate(pairs):
    for t, c in enumerate(eng):
        encoder_input[i, t, eng_to_idx[c]] = 1
    hin_seq = '' + hin + ''
    for t in range(len(hin_seq)):
        if t < len(hin_seq):
            ch = hin_seq[t] if hin_seq[t] in hin_to_idx else ''
            decoder_input[i, t, hin_to_idx.get(ch, 0)] = 1
        if t > 0:
            ch = hin_seq[t] if hin_seq[t] in hin_to_idx else ''
            decoder_target[i, t-1, hin_to_idx.get(ch, 0)] = 1

# Encoder
encoder_inputs = tf.keras.Input(shape=(max_eng, len(eng_chars)))
encoder_lstm = tf.keras.layers.LSTM(64, return_state=True)
_, state_h, state_c = encoder_lstm(encoder_inputs)

# Decoder
decoder_inputs = tf.keras.Input(shape=(max_hin, len(hin_chars)))
decoder_lstm = tf.keras.layers.LSTM(64, return_sequences=True, return_state=True)
decoder_out, _, _ = decoder_lstm(decoder_inputs, initial_state=[state_h, state_c])
decoder_dense = tf.keras.layers.Dense(len(hin_chars), activation='softmax')
decoder_outputs = decoder_dense(decoder_out)

# Model
model = tf.keras.Model([encoder_inputs, decoder_inputs], decoder_outputs)
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit([encoder_input, decoder_input], decoder_target,
          epochs=200, batch_size=4, verbose=1)

print("Seq2Seq transliterator trained!")
print("This demonstrates encoder-decoder architecture for")
print("mapping English character sequences to Hindi characters.")

End-of-Chapter Exercises (25 Questions)

Multiple Choice Questions (12 MCQs)

Interview Questions (12 Questions)

Research Problems

Key Takeaways

References & Further Reading