Phase 3 • EduArtha

Classical Machine Learning

Before deep learning, you must understand the simpler models. They build your intuition for how learning works. This book covers supervised learning, unsupervised learning, model evaluation, and production ML frameworks.

⏱ 3–5 months | 14 Chapters | 55+ Exercises

Part I

Supervised Learning

Learning from labeled data

Chapter 1

Linear & Logistic Regression

Learning Objectives

Understand linear regression: hypothesis, cost function, gradient descent
Master logistic regression for binary classification
Implement both from scratch and with scikit-learn
Interpret coefficients and understand assumptions

Linear Regression

Linear regression finds the best-fit line through data by minimizing the sum of squared errors. The model assumes a linear relationship: ŷ = w₁x₁ + w₂x₂ + ... + wₙxₙ + b.

Cost function (MSE): J(w) = (1/2m) Σᵢ (ŷᵢ - yᵢ)² → Minimize by adjusting weights w

Python
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.datasets import make_regression

# Generate synthetic data
X, y = make_regression(n_samples=500, n_features=3, noise=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Scikit-learn
model = LinearRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print(f"Coefficients: {model.coef_.round(2)}")
print(f"Intercept:    {model.intercept_:.2f}")
print(f"R² Score:     {r2_score(y_test, y_pred):.4f}")
print(f"RMSE:         {np.sqrt(mean_squared_error(y_test, y_pred)):.2f}")

From Scratch: Gradient Descent

Python
class LinearRegressionGD:
    def __init__(self, lr=0.01, epochs=1000):
        self.lr = lr
        self.epochs = epochs

    def fit(self, X, y):
        m, n = X.shape
        self.w = np.zeros(n)
        self.b = 0
        self.losses = []

        for _ in range(self.epochs):
            y_pred = X @ self.w + self.b
            error = y_pred - y
            self.w -= self.lr * (1/m) * (X.T @ error)
            self.b -= self.lr * (1/m) * np.sum(error)
            self.losses.append(np.mean(error**2))

    def predict(self, X):
        return X @ self.w + self.b

Logistic Regression

Logistic regression applies the sigmoid function σ(z) = 1/(1+e⁻ᶻ) to convert linear output into a probability between 0 and 1. It uses binary cross-entropy loss instead of MSE.

Python
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import accuracy_score, classification_report

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42)

clf = LogisticRegression(max_iter=5000)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)

print(f"Accuracy: {accuracy_score(y_test, y_pred):.2%}")
print(classification_report(y_test, y_pred, target_names=data.target_names))

Project: House Price Prediction

Python
from sklearn.datasets import fetch_california_housing
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge, Lasso

housing = fetch_california_housing()
X_train, X_test, y_train, y_test = train_test_split(
    housing.data, housing.target, test_size=0.2, random_state=42)

scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)

# Compare Linear, Ridge (L2), Lasso (L1)
for name, model in [("Linear", LinearRegression()),
                    ("Ridge", Ridge(alpha=1.0)),
                    ("Lasso", Lasso(alpha=0.1))]:
    model.fit(X_train_s, y_train)
    score = model.score(X_test_s, y_test)
    print(f"{name:8s} R²: {score:.4f}")

Exercises

Exercise 1.1: What is the difference between R² and adjusted R²?

R² measures proportion of variance explained, but always increases with more features (even irrelevant ones). Adjusted R² penalizes for additional features: it only increases if the new feature improves the model more than expected by chance. Use adjusted R² when comparing models with different numbers of features.

Exercise 1.2: Why must you scale features before logistic regression?

Logistic regression uses gradient descent. Features with large ranges (e.g., salary: 50000) dominate over small-range features (e.g., age: 30). Without scaling, gradients are uneven, causing slow convergence and suboptimal solutions. StandardScaler (mean=0, std=1) puts all features on equal footing.

Exercise 1.3: When would Ridge (L2) outperform Lasso (L1)?

Ridge: When all features are somewhat relevant — it shrinks coefficients but doesn't zero them out. Better for multicollinearity. Lasso: When you suspect many features are irrelevant — it drives coefficients to exactly zero, performing automatic feature selection. Use ElasticNet for a mix of both.

Exercise 1.4: Implement the sigmoid function and binary cross-entropy loss

def sigmoid(z): return 1 / (1 + np.exp(-z))
def bce_loss(y, y_hat):
    eps = 1e-15
    y_hat = np.clip(y_hat, eps, 1-eps)
    return -np.mean(y*np.log(y_hat) + (1-y)*np.log(1-y_hat))

Chapter Summary

Linear regression minimizes MSE to find the best-fit hyperplane
Logistic regression uses sigmoid + cross-entropy for classification
Ridge (L2) and Lasso (L1) add regularization to prevent overfitting
Always scale features before training gradient-based models