Chapter 18: Applied Deep Learning — Natural Language Processing

Bloom's Taxonomy Map for This Chapter

Learning Objectives

By the end of this chapter, you will be able to:

• Build a Hindi/Hinglish sentiment analysis system using MuRIL (Multilingual Representations for Indian Languages) from HuggingFace, handling code-mixed reviews from Flipkart
• Implement an extractive legal document summarizer using IndicBERT on the ILDC (Indian Legal Documents Corpus) from IIIT Hyderabad, evaluating with ROUGE scores
• Design an IRCTC-style chatbot that handles 2 crore+ daily queries using BERT-based intent classification and response retrieval
• Train Named Entity Recognition models for Indian news articles — comparing Bi-LSTM-CRF vs Transformer architectures for Person, Organization, Location, Date, and Currency entities
• Explain Automatic Speech Recognition (ASR) for Indian languages using Wav2Vec 2.0 and AI4Bharat's IndicWav2Vec for 9+ languages
• Handle Indian-language-specific challenges: Devanagari/multi-script tokenization, code-switching, morphological richness, and low-resource data augmentation
• Evaluate NLP systems using appropriate metrics: Accuracy/F1 (classification), ROUGE (summarization), Entity F1 (NER), WER (ASR)

Opening Hook — India's Language AI Revolution

Core Concepts — Five NLP Projects for India

This chapter is organized as five complete projects, each addressing a real Indian NLP problem. Every project follows a consistent structure: Problem → Dataset → Model Architecture → Full Code → Evaluation → Indian Language Challenges.

18.1 The Indian NLP Pipeline — Unique Challenges

Before diving into projects, let's understand why Indian NLP is fundamentally harder than English NLP:

The Standard NLP Pipeline for Indian Languages

18.2 Project 1 — Hindi/Hinglish Sentiment Analysis

Dataset: Flipkart Hindi/Hinglish Reviews

We use a curated dataset of Flipkart product reviews in Hindi and code-mixed Hindi-English (Hinglish), labeled as Positive, Negative, or Neutral.

Why MuRIL? — Multilingual Representations for Indian Languages

Code-Mixing: The Key Challenge

Code-mixing occurs at multiple levels:

Full Implementation: MuRIL Sentiment Classifier

Python# ─── Project 1: Hindi Sentiment Analysis with MuRIL ───
# Fine-tune Google's MuRIL on Flipkart Hindi/Hinglish reviews

import torch
import pandas as pd
import numpy as np
from torch.utils.data import Dataset, DataLoader
from transformers import (
    AutoTokenizer, AutoModelForSequenceClassification,
    TrainingArguments, Trainer
)
from sklearn.metrics import classification_report, confusion_matrix
import re

# ─── Step 1: Preprocessing for Hindi/Hinglish ───

class HindiTextPreprocessor:
    """Handles code-mixed Hindi-English text preprocessing."""

    def __init__(self):
        # Common Hindi stopwords (in Devanagari)
        self.hindi_stopwords = {
            'है', 'हैं', 'का', 'की', 'के', 'में',
            'को', 'से', 'पर', 'और', 'यह', 'वह'
        }
        # Hinglish normalization map
        self.normalize_map = {
            'accha': 'अच्छा', 'acha': 'अच्छा',
            'bahut': 'बहुत', 'bohot': 'बहुत',
            'sahi': 'सही', 'shi': 'सही',
            'kharab': 'खराब', 'khrb': 'खराब',
            'mast': 'मस्त', 'bakwas': 'बकवास',
        }

    def clean_text(self, text):
        """Clean and normalize Hindi/Hinglish text."""
        text = str(text).lower()
        # Remove URLs, mentions, hashtags
        text = re.sub(r'http\S+', '', text)
        text = re.sub(r'@\w+', '', text)
        # Keep Devanagari chars (U+0900-U+097F), English, numbers
        text = re.sub(r'[^\u0900-\u097F\w\s]', ' ', text)
        # Normalize repeated characters: "bahuttttt" → "bahut"
        text = re.sub(r'(.)\1{2,}', r'\1\1', text)
        # Normalize common Hinglish spellings
        words = text.split()
        words = [self.normalize_map.get(w, w) for w in words]
        return ' '.join(words).strip()

    def detect_language_ratio(self, text):
        """Detect Hindi vs English ratio in text."""
        hindi_chars = len(re.findall(r'[\u0900-\u097F]', text))
        eng_chars = len(re.findall(r'[a-zA-Z]', text))
        total = hindi_chars + eng_chars
        if total == 0:
            return 0.0
        return hindi_chars / total  # 1.0 = pure Hindi, 0.0 = pure English


# ─── Step 2: Dataset Class ───

class FlipkartHindiDataset(Dataset):
    """PyTorch Dataset for Flipkart Hindi/Hinglish reviews."""

    LABEL_MAP = {'positive': 0, 'negative': 1, 'neutral': 2}

    def __init__(self, texts, labels, tokenizer, max_length=128):
        self.texts = texts
        self.labels = [self.LABEL_MAP[l] for l in labels]
        self.tokenizer = tokenizer
        self.max_length = max_length
        self.preprocessor = HindiTextPreprocessor()

    def __len__(self):
        return len(self.texts)

    def __getitem__(self, idx):
        text = self.preprocessor.clean_text(self.texts[idx])
        encoding = self.tokenizer(
            text,
            max_length=self.max_length,
            padding='max_length',
            truncation=True,
            return_tensors='pt'
        )
        return {
            'input_ids': encoding['input_ids'].squeeze(),
            'attention_mask': encoding['attention_mask'].squeeze(),
            'labels': torch.tensor(self.labels[idx], dtype=torch.long)
        }


# ─── Step 3: Load MuRIL and Fine-Tune ───

MODEL_NAME = "google/muril-base-cased"
NUM_LABELS = 3

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForSequenceClassification.from_pretrained(
    MODEL_NAME,
    num_labels=NUM_LABELS,
    problem_type="single_label_classification"
)

# Load data (assumes CSV with 'review_text' and 'sentiment' columns)
df_train = pd.read_csv("flipkart_hindi_train.csv")
df_val   = pd.read_csv("flipkart_hindi_val.csv")
df_test  = pd.read_csv("flipkart_hindi_test.csv")

train_dataset = FlipkartHindiDataset(
    df_train['review_text'].tolist(),
    df_train['sentiment'].tolist(),
    tokenizer
)
val_dataset = FlipkartHindiDataset(
    df_val['review_text'].tolist(),
    df_val['sentiment'].tolist(),
    tokenizer
)

# ─── Step 4: Training Configuration ───

def compute_metrics(eval_pred):
    """Compute accuracy and macro F1 for evaluation."""
    from sklearn.metrics import accuracy_score, f1_score
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    acc = accuracy_score(labels, preds)
    f1 = f1_score(labels, preds, average='macro')
    return {'accuracy': acc, 'f1_macro': f1}


training_args = TrainingArguments(
    output_dir="./muril-flipkart-sentiment",
    num_train_epochs=4,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,
    warmup_ratio=0.1,
    weight_decay=0.01,
    learning_rate=2e-5,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="f1_macro",
    logging_steps=50,
    fp16=True,  # Mixed precision for faster training
    report_to="none",
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    compute_metrics=compute_metrics,
)

# Train!
trainer.train()

# ─── Step 5: Evaluation ───

test_dataset = FlipkartHindiDataset(
    df_test['review_text'].tolist(),
    df_test['sentiment'].tolist(),
    tokenizer
)
results = trainer.evaluate(test_dataset)
print(f"Test Accuracy: {results['eval_accuracy']:.4f}")
print(f"Test F1 Macro: {results['eval_f1_macro']:.4f}")

# ─── Step 6: Inference on New Reviews ───

def predict_sentiment(text, model, tokenizer):
    """Predict sentiment of a Hindi/Hinglish review."""
    preprocessor = HindiTextPreprocessor()
    clean = preprocessor.clean_text(text)
    inputs = tokenizer(clean, return_tensors="pt", truncation=True,
                       max_length=128, padding=True)
    inputs = {k: v.to(model.device) for k, v in inputs.items()}

    with torch.no_grad():
        outputs = model(**inputs)
    probs = torch.softmax(outputs.logits, dim=-1)
    pred = torch.argmax(probs, dim=-1).item()
    labels = ['Positive', 'Negative', 'Neutral']
    return labels[pred], probs[0].tolist()


# Test with code-mixed reviews
reviews = [
    "bahut accha product hai, quality mast hai 👍",
    "paise barbaad! bilkul kharab quality, return karna padega",
    "theek hai, not great not bad, average product",
    "बहुत अच्छा फोन है, कैमरा quality शानदार",
    "delivery late thi but product sahi hai",
]

for review in reviews:
    sentiment, probs = predict_sentiment(review, model, tokenizer)
    print(f"Review: {review[:50]}...")
    print(f"  → {sentiment} (conf: {max(probs):.2%})\n")

Model Comparison: Why MuRIL Wins for Hindi

18.3 Project 2 — Legal Document Summarization

Dataset: ILDC — Indian Legal Documents Corpus

The ILDC (Indian Legal Documents Corpus) was created by researchers at IIIT Hyderabad and contains Supreme Court judgments with expert-written summaries.

Approach: Extractive Summarization with IndicBERT

Python# ─── Project 2: Legal Document Summarization ───
# Extractive summarization for Indian court judgments using BERT

import torch
import torch.nn as nn
from transformers import AutoTokenizer, AutoModel
from rouge_score import rouge_scorer
import numpy as np
import re

# ─── Step 1: Sentence Segmentation for Legal Text ───

class LegalSentenceSegmenter:
    """Segment Indian legal documents into sentences.

    Legal text has unique patterns:
    - Section references: 'Sec. 302 I.P.C.'
    - Case citations: 'AIR 1950 SC 27'
    - Abbreviations: 'Hon'ble', 'vs.', 'Ltd.'
    """

    def __init__(self):
        # Patterns that look like sentence-ends but aren't
        self.abbreviations = {
            'vs', 'sec', 'art', 'no', 'sr',
            'dr', 'mr', 'mrs', 'smt', 'shri',
            'hon', 'ltd', 'pvt', 'govt', 'i.e',
            'e.g', 'etc', 'i.p.c', 'cr.p.c', 'c.p.c'
        }

    def segment(self, text):
        """Split legal document into sentences."""
        # Protect abbreviations
        for abbr in self.abbreviations:
            text = re.sub(
                rf'\b{re.escape(abbr)}\.',
                abbr.replace('.', '_DOT_') + '_ABBR_',
                text,
                flags=re.IGNORECASE
            )
        # Split on sentence-ending punctuation
        sentences = re.split(r'(?<=[.!?])\s+(?=[A-Z\d])', text)
        # Restore abbreviations
        sentences = [s.replace('_DOT_', '.').replace('_ABBR_', '.')
                     for s in sentences]
        # Filter very short sentences (noise)
        sentences = [s.strip() for s in sentences if len(s.split()) > 5]
        return sentences


# ─── Step 2: BertSumExt Model ───

class LegalBertSumExt(nn.Module):
    """Extractive summarizer using BERT sentence representations.

    Architecture:
    1. Encode each sentence with BERT (CLS token)
    2. Inter-sentence Transformer (2 layers) for context
    3. Binary classifier: important (1) or not (0)
    """

    def __init__(self, bert_model_name="ai4bharat/indic-bert",
                 n_heads=8, n_inter_layers=2):
        super().__init__()
        self.bert = AutoModel.from_pretrained(bert_model_name)
        hidden_size = self.bert.config.hidden_size  # 768

        # Inter-sentence Transformer layers
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=hidden_size,
            nhead=n_heads,
            dim_feedforward=hidden_size * 4,
            dropout=0.1,
            activation='gelu',
            batch_first=True
        )
        self.inter_sentence_transformer = nn.TransformerEncoder(
            encoder_layer, num_layers=n_inter_layers
        )

        # Binary classifier for each sentence
        self.classifier = nn.Sequential(
            nn.Linear(hidden_size, 256),
            nn.GELU(),
            nn.Dropout(0.1),
            nn.Linear(256, 1),
            nn.Sigmoid()
        )

    def forward(self, input_ids, attention_mask, sentence_mask):
        """
        Args:
            input_ids: (batch, num_sents, seq_len)
            attention_mask: (batch, num_sents, seq_len)
            sentence_mask: (batch, num_sents) — 1 for real, 0 for pad
        Returns:
            scores: (batch, num_sents) — importance score per sentence
        """
        batch_size, num_sents, seq_len = input_ids.shape

        # Encode each sentence independently with BERT
        input_ids = input_ids.view(-1, seq_len)
        attention_mask = attention_mask.view(-1, seq_len)

        bert_out = self.bert(input_ids, attention_mask=attention_mask)
        cls_embeddings = bert_out.last_hidden_state[:, 0, :]  # CLS tokens
        cls_embeddings = cls_embeddings.view(batch_size, num_sents, -1)

        # Inter-sentence Transformer for document-level context
        src_key_padding_mask = (sentence_mask == 0)
        contextualized = self.inter_sentence_transformer(
            cls_embeddings,
            src_key_padding_mask=src_key_padding_mask
        )

        # Score each sentence
        scores = self.classifier(contextualized).squeeze(-1)
        scores = scores * sentence_mask  # Zero out padded sentences

        return scores


# ─── Step 3: Oracle Label Creation ───

def create_oracle_labels(document_sentences, reference_summary, top_k=5):
    """Create oracle extractive labels using greedy ROUGE optimization.

    Greedily select sentences that maximize ROUGE-2 F1 with reference.
    """
    scorer = rouge_scorer.RougeScorer(['rouge2'], use_stemmer=True)
    selected = []
    remaining = list(range(len(document_sentences)))
    labels = [0] * len(document_sentences)

    for _ in range(min(top_k, len(document_sentences))):
        best_idx = -1
        best_score = -1

        for idx in remaining:
            candidate = ' '.join(
                [document_sentences[i] for i in selected + [idx]]
            )
            score = scorer.score(reference_summary, candidate)
            rouge2_f1 = score['rouge2'].fmeasure

            if rouge2_f1 > best_score:
                best_score = rouge2_f1
                best_idx = idx

        if best_idx >= 0 and best_score > 0:
            selected.append(best_idx)
            remaining.remove(best_idx)
            labels[best_idx] = 1

    return labels


# ─── Step 4: Training Loop ───

def train_summarizer(model, train_loader, val_loader,
                     epochs=5, lr=2e-5):
    optimizer = torch.optim.AdamW(model.parameters(), lr=lr,
                                  weight_decay=0.01)
    criterion = nn.BCELoss(reduction='none')
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model.to(device)

    for epoch in range(epochs):
        model.train()
        total_loss = 0

        for batch in train_loader:
            input_ids = batch['input_ids'].to(device)
            attn_mask = batch['attention_mask'].to(device)
            sent_mask = batch['sentence_mask'].to(device)
            labels = batch['labels'].to(device).float()

            scores = model(input_ids, attn_mask, sent_mask)
            loss = criterion(scores, labels)
            loss = (loss * sent_mask).sum() / sent_mask.sum()

            optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
            optimizer.step()

            total_loss += loss.item()

        # Evaluate on validation set
        val_rouge = evaluate_summarizer(model, val_loader, device)
        print(f"Epoch {epoch+1}/{epochs} | "
              f"Loss: {total_loss/len(train_loader):.4f} | "
              f"ROUGE-2: {val_rouge['rouge2']:.4f} | "
              f"ROUGE-L: {val_rouge['rougeL']:.4f}")


# ─── Step 5: Evaluation ───

def evaluate_summarizer(model, data_loader, device, top_k=5):
    """Evaluate using ROUGE scores."""
    model.eval()
    scorer = rouge_scorer.RougeScorer(
        ['rouge1', 'rouge2', 'rougeL'], use_stemmer=True
    )
    all_scores = {'rouge1': [], 'rouge2': [], 'rougeL': []}

    with torch.no_grad():
        for batch in data_loader:
            scores = model(
                batch['input_ids'].to(device),
                batch['attention_mask'].to(device),
                batch['sentence_mask'].to(device)
            )
            # Select top-k sentences
            for i in range(scores.shape[0]):
                sent_scores = scores[i].cpu().numpy()
                top_indices = np.argsort(sent_scores)[-top_k:]
                top_indices = sorted(top_indices)  # Maintain order

                pred_summary = ' '.join(
                    [batch['sentences'][i][j] for j in top_indices]
                )
                ref_summary = batch['reference'][i]

                rouge = scorer.score(ref_summary, pred_summary)
                for key in all_scores:
                    all_scores[key].append(rouge[key].fmeasure)

    return {k: np.mean(v) for k, v in all_scores.items()}

18.4 Project 3 — IRCTC Railway Chatbot

Architecture: Intent Classification + Response Retrieval

Training Data: Intent Categories

Python# ─── Project 3: IRCTC Chatbot ───
# BERT-based Intent Classification + Entity Extraction

import torch
import torch.nn as nn
from transformers import AutoTokenizer, AutoModel
from torch.utils.data import Dataset, DataLoader
import json, re
import numpy as np

# ─── Step 1: IRCTC-specific Entity Extractor ───

class IRCTCEntityExtractor:
    """Extract railway-specific entities from queries."""

    def __init__(self):
        self.patterns = {
            'pnr': r'\b(\d{10})\b',
            'train_number': r'\b(1[2-9]\d{3}|[2-9]\d{4})\b',
            'date': r'\b(\d{1,2}[/-]\d{1,2}[/-]\d{2,4})\b',
            'coach': r'\b([SB]\d{1,2}|[ABCD]\d|H1|HA1)\b',
            'class': r'\b(1AC|2AC|3AC|SL|CC|2S|GEN|1A|2A|3A)\b',
            'platform': r'platform\s*(\d{1,2})',
        }
        # Major Indian stations
        self.stations = {
            'delhi': 'NDLS', 'new delhi': 'NDLS',
            'mumbai': 'CSTM', 'mumbai central': 'BCT',
            'chennai': 'MAS', 'kolkata': 'HWH',
            'bangalore': 'SBC', 'bengaluru': 'SBC',
            'hyderabad': 'SC', 'pune': 'PUNE',
            'jaipur': 'JP', 'lucknow': 'LKO',
            'ahmedabad': 'ADI', 'patna': 'PNBE',
            'varanasi': 'BSB', 'kanpur': 'CNB',
        }

    def extract(self, text):
        """Extract all entities from a query."""
        entities = {}
        text_lower = text.lower()

        # Regex-based entity extraction
        for entity_type, pattern in self.patterns.items():
            match = re.search(pattern, text, re.IGNORECASE)
            if match:
                entities[entity_type] = match.group(1)

        # Station name extraction
        stations_found = []
        for name, code in self.stations.items():
            if name in text_lower:
                stations_found.append({'name': name, 'code': code})
        if stations_found:
            entities['stations'] = stations_found

        return entities


# ─── Step 2: Intent Classification Model ───

class IRCTCIntentClassifier(nn.Module):
    """BERT-based intent classifier for IRCTC queries."""

    INTENTS = [
        'pnr_status', 'train_schedule', 'ticket_booking',
        'ticket_cancel', 'seat_availability', 'platform_info',
        'food_order', 'complaint', 'fare_enquiry',
        'live_status', 'tatkal_booking', 'general_query'
    ]

    def __init__(self, bert_model="google/muril-base-cased"):
        super().__init__()
        self.bert = AutoModel.from_pretrained(bert_model)
        hidden = self.bert.config.hidden_size

        self.classifier = nn.Sequential(
            nn.Dropout(0.3),
            nn.Linear(hidden, 256),
            nn.GELU(),
            nn.Dropout(0.1),
            nn.Linear(256, len(self.INTENTS))
        )

        # Confidence threshold — below this, escalate to human
        self.confidence_threshold = 0.75

    def forward(self, input_ids, attention_mask):
        outputs = self.bert(input_ids, attention_mask=attention_mask)
        cls_output = outputs.last_hidden_state[:, 0, :]
        logits = self.classifier(cls_output)
        return logits

    def predict(self, text, tokenizer, device):
        """Predict intent with confidence score."""
        self.eval()
        inputs = tokenizer(text, return_tensors="pt",
                          truncation=True, max_length=64,
                          padding=True)
        inputs = {k: v.to(device) for k, v in inputs.items()}

        with torch.no_grad():
            logits = self.forward(inputs['input_ids'],
                                  inputs['attention_mask'])
        probs = torch.softmax(logits, dim=-1)[0]
        top_prob, top_idx = probs.max(dim=0)

        intent = self.INTENTS[top_idx.item()]
        confidence = top_prob.item()

        return {
            'intent': intent,
            'confidence': confidence,
            'escalate': confidence < self.confidence_threshold
        }


# ─── Step 3: Response Templates ───

RESPONSE_TEMPLATES = {
    'pnr_status': "🚂 Aapka PNR {pnr} ka status: {status}. "
                  "Coach {coach}, Berth {berth}. Yatra mangalmay ho!",

    'train_schedule': "🕐 Train {train_number} ka schedule:\n"
                      "Departure: {dep_time} ({source})\n"
                      "Arrival: {arr_time} ({dest})",

    'ticket_booking': "🎫 {source} se {dest} ke liye {class} mein "
                      "ticket available hai. Fare: ₹{fare}. "
                      "Book karna chahte hain?",

    'seat_availability': "💺 {date} ko {train_number} mein {class}: "
                          "{available} seats available.",

    'live_status': "📍 Train {train_number} abhi {location} pe hai. "
                   "Expected delay: {delay} min.",

    'complaint': "📝 Aapki complaint register ho gayi hai. "
                 "Reference: {ref_no}. 24 ghante mein response milega.",

    'fare_enquiry': "💰 {source} → {dest} fare:\n"
                    "1AC: ₹{fare_1ac} | 2AC: ₹{fare_2ac} | "
                    "3AC: ₹{fare_3ac} | SL: ₹{fare_sl}",
}


# ─── Step 4: Full Chatbot Pipeline ───

class IRCTCChatbot:
    """End-to-end IRCTC chatbot combining intent + entities + response."""

    def __init__(self, model_path, device='cpu'):
        self.device = torch.device(device)
        self.tokenizer = AutoTokenizer.from_pretrained(
            "google/muril-base-cased"
        )
        self.intent_model = IRCTCIntentClassifier()
        self.intent_model.load_state_dict(
            torch.load(model_path, map_location=self.device)
        )
        self.intent_model.to(self.device)
        self.entity_extractor = IRCTCEntityExtractor()

    def respond(self, user_query):
        """Process user query and generate response."""
        # Step 1: Classify intent
        result = self.intent_model.predict(
            user_query, self.tokenizer, self.device
        )

        # Step 2: Extract entities
        entities = self.entity_extractor.extract(user_query)

        # Step 3: Check if escalation needed
        if result['escalate']:
            return {
                'response': "Main aapko humare agent se connect "
                           "karta hoon. Please hold karein.",
                'intent': result['intent'],
                'confidence': result['confidence'],
                'escalated': True
            }

        # Step 4: Generate response from template
        template = RESPONSE_TEMPLATES.get(
            result['intent'],
            "Main samajh nahi paaya. Kya aap dobara bata sakte hain?"
        )

        return {
            'response': template,
            'intent': result['intent'],
            'confidence': result['confidence'],
            'entities': entities,
            'escalated': False
        }


# ─── Step 5: Evaluation ───

def evaluate_chatbot(model, test_loader, device):
    """Evaluate intent classification accuracy."""
    model.eval()
    correct, total = 0, 0
    per_intent_correct = {}
    per_intent_total = {}

    with torch.no_grad():
        for batch in test_loader:
            logits = model(
                batch['input_ids'].to(device),
                batch['attention_mask'].to(device)
            )
            preds = logits.argmax(dim=-1)
            labels = batch['labels'].to(device)

            correct += (preds == labels).sum().item()
            total += labels.size(0)

            for p, l in zip(preds, labels):
                intent = IRCTCIntentClassifier.INTENTS[l.item()]
                per_intent_total[intent] = per_intent_total.get(intent, 0) + 1
                if p == l:
                    per_intent_correct[intent] = \
                        per_intent_correct.get(intent, 0) + 1

    overall_acc = correct / total
    print(f"Overall Accuracy: {overall_acc:.4f}\n")
    print(f"{'Intent':<20} {'Accuracy':>10}")
    print("-" * 32)
    for intent in IRCTCIntentClassifier.INTENTS:
        acc = per_intent_correct.get(intent, 0) / \
              max(per_intent_total.get(intent, 1), 1)
        print(f"{intent:<20} {acc:>10.2%}")

    return overall_acc

18.5 Project 4 — Named Entity Recognition for Indian News

Entity Types for Indian News

BIO Tagging Scheme

Token:  नरेंद्र    मोदी     ने      रिलायंस   इंडस्ट्रीज  को     ₹1,500   करोड़    दिये
Tag:    B-PER    I-PER    O      B-ORG     I-ORG       O     B-CUR    I-CUR    O

Architecture Comparison: Bi-LSTM-CRF vs Transformer

Python# ─── Project 4A: Bi-LSTM-CRF for Indian NER ───
# Classic sequence labeling approach

import torch
import torch.nn as nn
from torchcrf import CRF
import numpy as np

# Tag set for Indian news NER
TAG_TO_IDX = {
    'O': 0,
    'B-PER': 1, 'I-PER': 2,
    'B-ORG': 3, 'I-ORG': 4,
    'B-LOC': 5, 'I-LOC': 6,
    'B-DATE': 7, 'I-DATE': 8,
    'B-CUR': 9, 'I-CUR': 10,
}
IDX_TO_TAG = {v: k for k, v in TAG_TO_IDX.items()}
NUM_TAGS = len(TAG_TO_IDX)


class BiLSTMCRF(nn.Module):
    """Bi-LSTM-CRF for Named Entity Recognition.

    Architecture:
    ┌────────────┐
    │    CRF     │ ← Ensures valid tag transitions
    ├────────────┤    (e.g., I-PER can't follow B-ORG)
    │  Linear    │
    ├────────────┤
    │  Bi-LSTM   │ ← Captures bidirectional context
    │  (2 layers)│
    ├────────────┤
    │  Char-CNN  │ ← Character-level features (morphology)
    │  + Word Emb│
    └────────────┘
    """

    def __init__(self, vocab_size, char_vocab_size,
                 word_emb_dim=300, char_emb_dim=30,
                 char_hidden=50, lstm_hidden=256,
                 num_layers=2, dropout=0.5):
        super().__init__()

        # Word embeddings (can load Hindi fastText vectors)
        self.word_embedding = nn.Embedding(vocab_size, word_emb_dim,
                                           padding_idx=0)

        # Character-level CNN (captures morphological patterns)
        self.char_embedding = nn.Embedding(char_vocab_size, char_emb_dim,
                                           padding_idx=0)
        self.char_cnn = nn.Conv1d(
            char_emb_dim, char_hidden,
            kernel_size=3, padding=1
        )

        # Bi-LSTM
        input_dim = word_emb_dim + char_hidden
        self.lstm = nn.LSTM(
            input_dim, lstm_hidden,
            num_layers=num_layers,
            bidirectional=True,
            batch_first=True,
            dropout=dropout if num_layers > 1 else 0
        )

        # Emission scores
        self.hidden2tag = nn.Linear(lstm_hidden * 2, NUM_TAGS)
        self.dropout = nn.Dropout(dropout)

        # CRF layer — learns valid tag transitions
        self.crf = CRF(NUM_TAGS, batch_first=True)

    def _get_char_features(self, char_ids):
        """Compute character-level features using CNN."""
        batch, seq_len, char_len = char_ids.shape
        char_ids = char_ids.view(-1, char_len)
        char_emb = self.char_embedding(char_ids)  # (B*S, C, D)
        char_emb = char_emb.permute(0, 2, 1)      # (B*S, D, C)
        char_cnn = self.char_cnn(char_emb)         # (B*S, H, C)
        char_features = char_cnn.max(dim=2)[0]    # Max pool: (B*S, H)
        char_features = char_features.view(batch, seq_len, -1)
        return char_features

    def _get_emissions(self, word_ids, char_ids):
        """Compute emission scores from Bi-LSTM."""
        word_emb = self.word_embedding(word_ids)
        char_feat = self._get_char_features(char_ids)
        combined = torch.cat([word_emb, char_feat], dim=-1)
        combined = self.dropout(combined)

        lstm_out, _ = self.lstm(combined)
        lstm_out = self.dropout(lstm_out)
        emissions = self.hidden2tag(lstm_out)
        return emissions

    def forward(self, word_ids, char_ids, tags, mask):
        """Compute negative log-likelihood loss."""
        emissions = self._get_emissions(word_ids, char_ids)
        loss = -self.crf(emissions, tags, mask=mask,
                         reduction='mean')
        return loss

    def predict(self, word_ids, char_ids, mask):
        """Viterbi decoding for best tag sequence."""
        emissions = self._get_emissions(word_ids, char_ids)
        best_tags = self.crf.decode(emissions, mask=mask)
        return best_tags


# ─── Project 4B: Transformer NER (BERT-based) ───

from transformers import AutoModelForTokenClassification

class TransformerNER:
    """Transformer-based NER using MuRIL/IndicBERT."""

    def __init__(self, model_name="google/muril-base-cased"):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForTokenClassification.from_pretrained(
            model_name,
            num_labels=NUM_TAGS,
            id2label=IDX_TO_TAG,
            label2id=TAG_TO_IDX,
        )

    def predict_entities(self, text):
        """Predict entities in text, handling subword tokens."""
        inputs = self.tokenizer(
            text, return_tensors="pt",
            return_offsets_mapping=True,
            truncation=True, max_length=512
        )
        offset_mapping = inputs.pop('offset_mapping')[0]

        with torch.no_grad():
            outputs = self.model(**inputs)
        preds = outputs.logits.argmax(dim=-1)[0]

        # Map subword predictions back to words
        entities = []
        current_entity = None

        for idx, (pred, offset) in enumerate(
            zip(preds, offset_mapping)
        ):
            if offset[0] == 0 and offset[1] == 0:
                continue  # Skip [CLS], [SEP]

            tag = IDX_TO_TAG[pred.item()]
            start, end = offset[0].item(), offset[1].item()
            token_text = text[start:end]

            if tag.startswith('B-'):
                if current_entity:
                    entities.append(current_entity)
                current_entity = {
                    'type': tag[2:],
                    'text': token_text,
                    'start': start, 'end': end
                }
            elif tag.startswith('I-') and current_entity:
                current_entity['text'] += token_text
                current_entity['end'] = end
            else:
                if current_entity:
                    entities.append(current_entity)
                    current_entity = None

        if current_entity:
            entities.append(current_entity)

        return entities


# ─── Evaluation: Entity-Level F1 ───

from seqeval.metrics import classification_report as ner_report

def evaluate_ner(true_tags_list, pred_tags_list):
    """Evaluate NER using entity-level F1 (seqeval library)."""
    print(ner_report(true_tags_list, pred_tags_list, digits=4))

Bi-LSTM-CRF vs Transformer NER: Head-to-Head

18.6 Project 5 — Automatic Speech Recognition for Indian Languages

Wav2Vec 2.0 for Indian English

AI4Bharat's IndicWav2Vec

Python# ─── Project 5: ASR with IndicWav2Vec ───
# Automatic Speech Recognition for Indian languages

import torch
import torchaudio
from transformers import (
    Wav2Vec2ForCTC, Wav2Vec2Processor,
    Wav2Vec2CTCTokenizer, Wav2Vec2FeatureExtractor
)
import numpy as np

# ─── Step 1: Load IndicWav2Vec ───

MODEL_NAME = "ai4bharat/indicwav2vec_v1_hindi"

processor = Wav2Vec2Processor.from_pretrained(MODEL_NAME)
model = Wav2Vec2ForCTC.from_pretrained(MODEL_NAME)
model.eval()

# ─── Step 2: Inference Function ───

def transcribe_hindi(audio_path):
    """Transcribe Hindi speech to text."""
    # Load audio (must be 16kHz mono)
    waveform, sample_rate = torchaudio.load(audio_path)

    # Resample if needed
    if sample_rate != 16000:
        resampler = torchaudio.transforms.Resample(sample_rate, 16000)
        waveform = resampler(waveform)

    # Convert to mono if stereo
    if waveform.shape[0] > 1:
        waveform = waveform.mean(dim=0, keepdim=True)

    # Process through model
    inputs = processor(
        waveform.squeeze().numpy(),
        sampling_rate=16000,
        return_tensors="pt",
        padding=True
    )

    with torch.no_grad():
        logits = model(inputs.input_values).logits

    # CTC decode — greedy (argmax at each timestep)
    predicted_ids = torch.argmax(logits, dim=-1)
    transcription = processor.batch_decode(predicted_ids)[0]

    return transcription


# ─── Step 3: Evaluation (WER) ───

def word_error_rate(reference, hypothesis):
    """Compute Word Error Rate using dynamic programming."""
    ref_words = reference.split()
    hyp_words = hypothesis.split()
    n = len(ref_words)
    m = len(hyp_words)

    # DP table for edit distance
    dp = np.zeros((n + 1, m + 1))
    for i in range(n + 1):
        dp[i][0] = i
    for j in range(m + 1):
        dp[0][j] = j

    for i in range(1, n + 1):
        for j in range(1, m + 1):
            if ref_words[i-1] == hyp_words[j-1]:
                dp[i][j] = dp[i-1][j-1]
            else:
                dp[i][j] = 1 + min(
                    dp[i-1][j],    # Deletion
                    dp[i][j-1],    # Insertion
                    dp[i-1][j-1]  # Substitution
                )

    wer = dp[n][m] / n
    return wer

# Example evaluation
ref = "नमस्ते मैं हिंदी में बात कर रहा हूँ"
hyp = "नमस्ते मैं हिंदी में बात कर रहा हूं"
print(f"WER: {word_error_rate(ref, hyp):.2%}")
# WER: 11.11% (1 word different out of 9)

From-Scratch Code — Building a Minimal Attention-Based Classifier

To understand the fundamentals, let's build a simple attention-based text classifier from scratch in NumPy — no PyTorch, no HuggingFace. This demonstrates the core mechanism behind all five projects above.

Python (NumPy Only)# ─── From-Scratch: Attention-Based Text Classifier ───
# Demonstrates the attention mechanism powering all 5 projects

import numpy as np

class ScratchAttentionClassifier:
    """
    A simple attention-based text classifier built entirely in NumPy.

    Architecture:
    Input → Embedding → Self-Attention → Weighted Sum → Softmax → Class

    This is the core mechanism behind BERT/MuRIL fine-tuning.
    """

    def __init__(self, vocab_size, embed_dim=64,
                 num_classes=3, max_len=50):
        self.embed_dim = embed_dim
        self.num_classes = num_classes
        self.max_len = max_len

        # Xavier initialization
        scale = np.sqrt(2.0 / (vocab_size + embed_dim))
        self.W_emb = np.random.randn(vocab_size, embed_dim) * scale

        # Attention weights: Q, K, V projections
        scale_attn = np.sqrt(2.0 / (embed_dim + embed_dim))
        self.W_Q = np.random.randn(embed_dim, embed_dim) * scale_attn
        self.W_K = np.random.randn(embed_dim, embed_dim) * scale_attn
        self.W_V = np.random.randn(embed_dim, embed_dim) * scale_attn

        # Classification head
        scale_cls = np.sqrt(2.0 / (embed_dim + num_classes))
        self.W_cls = np.random.randn(embed_dim, num_classes) * scale_cls
        self.b_cls = np.zeros(num_classes)

    def softmax(self, x, axis=-1):
        """Numerically stable softmax."""
        e_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
        return e_x / e_x.sum(axis=axis, keepdims=True)

    def self_attention(self, X, mask=None):
        """
        Scaled dot-product self-attention.

        Q = X @ W_Q, K = X @ W_K, V = X @ W_V
        Attention(Q,K,V) = softmax(Q @ K^T / sqrt(d_k)) @ V
        """
        Q = X @ self.W_Q  # (seq_len, d)
        K = X @ self.W_K
        V = X @ self.W_V

        d_k = Q.shape[-1]
        scores = (Q @ K.T) / np.sqrt(d_k)  # (seq_len, seq_len)

        if mask is not None:
            scores = np.where(mask, scores, -1e9)

        attn_weights = self.softmax(scores, axis=-1)
        context = attn_weights @ V  # (seq_len, d)

        return context, attn_weights

    def forward(self, token_ids):
        """
        Forward pass: tokens → embedding → attention → classify.

        Args:
            token_ids: list of integer token IDs

        Returns:
            class_probs: (num_classes,) probability distribution
            attn_weights: (seq_len, seq_len) attention matrix
        """
        # Embedding lookup
        X = self.W_emb[token_ids]  # (seq_len, embed_dim)

        # Self-attention
        context, attn_weights = self.self_attention(X)

        # Pool: mean of context vectors (like [CLS] in BERT)
        pooled = context.mean(axis=0)  # (embed_dim,)

        # Classify
        logits = pooled @ self.W_cls + self.b_cls
        probs = self.softmax(logits)

        return probs, attn_weights

    def predict(self, token_ids):
        """Get predicted class and confidence."""
        probs, attn = self.forward(token_ids)
        pred_class = np.argmax(probs)
        labels = ['Positive', 'Negative', 'Neutral']
        return labels[pred_class], probs[pred_class], attn


# ─── Demo ───
np.random.seed(42)
classifier = ScratchAttentionClassifier(vocab_size=5000)

# Simulate tokenized input: "bahut accha product hai"
token_ids = [142, 87, 1203, 56]
label, confidence, attn = classifier.predict(token_ids)
print(f"Prediction: {label} (conf: {confidence:.2%})")
print(f"Attention matrix shape: {attn.shape}")
print(f"Token 'accha' attends most to: token {np.argmax(attn[1])}")

Industry Code — Production-Ready NLP Pipeline

Here's a production-grade pipeline combining all five projects into a unified Indian NLP system, as you might deploy at a company like Flipkart or Jio.

Python# ─── Production Indian NLP Pipeline ───
# Unified system for multi-task Indian language processing

from transformers import pipeline
import torch

class IndianNLPPipeline:
    """Production NLP pipeline for Indian languages.

    Supports: Sentiment, NER, Summarization, Intent Classification
    Languages: Hindi, Hinglish, English, + regional via IndicBERT
    """

    def __init__(self, device="cuda" if torch.cuda.is_available()
                 else "cpu"):
        self.device = device
        print(f"Initializing on {device}...")

        # Sentiment Analysis (MuRIL fine-tuned)
        self.sentiment = pipeline(
            "text-classification",
            model="./muril-flipkart-sentiment",
            device=0 if device == "cuda" else -1
        )

        # NER (MuRIL fine-tuned for Indian entities)
        self.ner = pipeline(
            "ner",
            model="./muril-indian-ner",
            aggregation_strategy="simple",
            device=0 if device == "cuda" else -1
        )

        # Zero-shot classification for flexible intent detection
        self.zero_shot = pipeline(
            "zero-shot-classification",
            model="joeddav/xlm-roberta-large-xnli",
            device=0 if device == "cuda" else -1
        )

    def analyze(self, text, tasks=None):
        """Run all requested NLP tasks on input text."""
        if tasks is None:
            tasks = ["sentiment", "ner"]

        results = {"text": text}

        if "sentiment" in tasks:
            results["sentiment"] = self.sentiment(text)[0]

        if "ner" in tasks:
            entities = self.ner(text)
            results["entities"] = [
                {"text": e["word"], "type": e["entity_group"],
                 "score": round(e["score"], 3)}
                for e in entities
            ]

        if "intent" in tasks:
            candidate_labels = [
                "ticket booking", "complaint",
                "status inquiry", "general question"
            ]
            results["intent"] = self.zero_shot(
                text, candidate_labels
            )

        return results


# ─── Usage ───
nlp = IndianNLPPipeline()

result = nlp.analyze(
    "Reliance Industries ne Mumbai mein ₹500 crore ka naya plant "
    "kholne ka faisla kiya hai.",
    tasks=["sentiment", "ner"]
)

print(f"Sentiment: {result['sentiment']['label']}")
print(f"Entities:")
for e in result['entities']:
    print(f"  {e['type']:>5}: {e['text']} ({e['score']:.1%})")

Visual Diagrams

6.1 The Complete Indian NLP Landscape

6.2 BIO Tagging NER Pipeline

6.3 Extractive Summarization Architecture

Worked Example — End-to-End Flipkart Review Analysis

Let's trace a single Flipkart review through the entire NLP pipeline, step by step.

Step 1: Preprocessing

Original: "Realme ka ye phone bahut accha hai, camera quality bhi mast
           hai lekin battery jaldi khatam ho jaati hai. ₹12,999 mein theek hai."

After clean_text():
  → "realme ka ye phone बहुत अच्छा hai camera quality bhi मस्त
     hai lekin battery jaldi khatam ho jaati hai ₹12999 mein theek hai"

Language ratio: detect_language_ratio() → 0.31 (31% Hindi chars = Hinglish)

Step 2: MuRIL Tokenization

Tokens: ['[CLS]', 'real', '##me', 'ka', 'ye', 'phone', 'बहुत', 'अच्छा',
         'hai', 'camera', 'quality', 'bhi', 'मस्त', 'hai', 'le', '##kin',
         'battery', 'jal', '##di', 'khat', '##am', 'ho', 'ja', '##ati',
         'hai', '₹', '12', '##99', '##9', 'mein', 'the', '##ek', 'hai', '[SEP]']

Token IDs: [2, 8734, 1456, 342, 178, 4521, 6789, 7234, 156, ...]
Length: 34 tokens (within 128 max_length)

Step 3: Sentiment Classification

MuRIL Output Logits: [2.34, -1.12, 0.45]  ← [Positive, Negative, Neutral]

Softmax probabilities:
  Positive: e^2.34 / (e^2.34 + e^-1.12 + e^0.45)
          = 10.38 / (10.38 + 0.33 + 1.57)
          = 10.38 / 12.28
          = 84.5%

  Negative: 0.33 / 12.28 = 2.7%
  Neutral:  1.57 / 12.28 = 12.8%

Prediction: ✅ POSITIVE (84.5% confidence)

Step 4: NER Extraction

Token:      realme  ka  ye  phone  बहुत  ...  ₹    12999  mein  theek  hai
NER Tag:    B-ORG   O   O   O      O     ...  B-CUR I-CUR  O     O      O

Extracted Entities:
  ORG: "Realme"    (score: 0.934)
  CUR: "₹12,999"  (score: 0.912)

Step 5: Aspect-Based Analysis (Advanced)

Aspects detected:
  📱 "camera quality" → Positive ("mast hai")
  🔋 "battery"        → Negative ("jaldi khatam ho jaati")
  💰 "₹12,999 mein"   → Neutral  ("theek hai")

Overall: Mixed Positive — good product, battery concern, fair price

Case Study — Koo's Multilingual Content Moderation

Common Mistakes & Misconceptions

Comparison Tables

10.1 All Five Projects At a Glance

10.2 Indian Language Models Comparison

10.3 NLP Metrics Cheat Sheet

Exercises

Chapter Summary

References & Further Reading

Foundational Papers

1. Khanuja, S., et al. (2021). "MuRIL: Multilingual Representations for Indian Languages." Google Research. The paper behind Google's Indian language model, trained on 17 languages with transliteration.
2. Kakwani, D., et al. (2020). "IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages." AI4Bharat, EMNLP Findings. The IndicBERT paper from IIT Madras.
3. Malik, V., et al. (2021). "ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation." IIIT Hyderabad, ACL. The dataset used in our legal summarization project.
4. Liu, Y., & Lapata, M. (2019). "Text Summarization with Pretrained Encoders." EMNLP. The BertSumExt/BertSumExtAbs paper — foundation for our legal summarizer.
5. Baevski, A., et al. (2020). "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations." Meta AI, NeurIPS. The foundational ASR model we adapt with IndicWav2Vec.

Indian NLP Resources

6. Joshi, A., et al. (2022). "IndicWav2Vec: Exploring Wav2Vec 2.0 for Indian Languages." AI4Bharat. Pretrained on 40,000+ hours of Indian speech.
7. Kunchukuttan, A., et al. (2020). "AI4Bharat-IndicNLP Corpus." IIT Madras. Large-scale corpora for 12 Indian languages.
8. Aggarwal, P., & Rani, R. (2023). "Code-Mixed Sentiment Analysis for Hindi-English Social Media Text." Survey of techniques for Hinglish NLP.
9. Lample, G., et al. (2016). "Neural Architectures for Named Entity Recognition." NAACL. The original Bi-LSTM-CRF paper for NER.
10. Lin, C.-Y. (2004). "ROUGE: A Package for Automatic Evaluation of Summaries." ACL Workshop. The standard evaluation metric for summarization.

Platforms & Tools

11. AI4Bharat — ai4bharat.org — India's premier open-source language AI initiative (IIT Madras)
12. Bhashini — bhashini.gov.in — Government of India's language translation platform
13. HuggingFace Indian Models — Search "MuRIL", "IndicBERT", "IndicWav2Vec" on huggingface.co
14. IndicNLP Library — github.com/anoopkunchukuttan/indic_nlp_library — Preprocessing tools for Indian languages
15. iNLTK — github.com/goru001/inltk — Indian NLP Toolkit

Textbooks

16. Jurafsky, D. & Martin, J.H. (2024). "Speech and Language Processing." 3rd Edition (Draft). The definitive NLP textbook — Chapters on NER, summarization, and ASR.
17. Goldberg, Y. (2017). "Neural Network Methods for Natural Language Processing." Morgan & Claypool. Clear explanation of Bi-LSTM-CRF and attention mechanisms.