Introduction to Machine Learning

What Is Machine Learning?

Machine learning is the practice of building systems that improve through experience. Instead of writing explicit rules — “if the email contains these 500 spam words, classify as spam” — you feed the system examples and it learns the rules itself.

This shifts the engineering problem from “write the rules” to “collect the right data and choose the right algorithm.” The data is doing the heavy lifting.

Three Types of Machine Learning

Supervised Learning

You have input features X and known labels y. The model learns a mapping f: X → y.

• Classification — predict a discrete label: spam/not spam, cat/dog, fraud/legitimate
• Regression — predict a continuous value: house price, stock return, patient readmission risk

Unsupervised Learning

You only have input features X — no labels. The model finds structure on its own.

• Clustering — group similar data points: customer segments, document topics
• Dimensionality Reduction — compress features while preserving information: PCA, t-SNE, UMAP
• Anomaly Detection — identify unusual patterns: fraud, network intrusion

Reinforcement Learning

An agent takes actions in an environment and receives rewards. It learns to maximize cumulative reward: game playing (AlphaGo), robot control, recommendation systems.

The Bias-Variance Tradeoff

This is the most important concept in ML. Every model makes two types of errors:

Bias — systematic error from wrong assumptions. A linear model fitting non-linear data has high bias — it’s consistently wrong in the same way. High bias = underfitting.

Variance — sensitivity to noise in the training data. A deep decision tree memorizes training noise and performs poorly on new data. High variance = overfitting.

Error = Bias² + Variance + Irreducible Noise

Simple model (linear regression): High Bias, Low Variance
Complex model (deep tree):         Low Bias, High Variance
Goal: find the sweet spot          Moderate Bias, Moderate Variance

import numpy as np
import matplotlib
matplotlib.use('Agg')  # non-interactive backend
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(42)
n = 100
X = rng.uniform(0, 10, n).reshape(-1, 1)
y = np.sin(X.ravel()) + rng.normal(0, 0.3, n)   # true signal + noise

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

for degree in [1, 3, 10, 20]:
    model = Pipeline([
        ("poly",  PolynomialFeatures(degree=degree)),
        ("reg",   LinearRegression()),
    ])
    model.fit(X_train, y_train)

    train_mse = mean_squared_error(y_train, model.predict(X_train))
    test_mse  = mean_squared_error(y_test,  model.predict(X_test))

    print(f"Degree {degree:2d}: Train MSE={train_mse:.4f}  Test MSE={test_mse:.4f}  "
          f"{'⚠ OVERFIT' if test_mse > train_mse * 2 else '✓'}")
# Degree  1: Train MSE=0.1892  Test MSE=0.1910  ✓ (underfitting)
# Degree  3: Train MSE=0.0912  Test MSE=0.0945  ✓ (good fit)
# Degree 10: Train MSE=0.0814  Test MSE=0.1423  ✓ (slight overfit)
# Degree 20: Train MSE=0.0715  Test MSE=4.2301  ⚠ OVERFIT

The ML Workflow

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import classification_report

# 1. Load and understand the data
data = load_breast_cancer()
X, y = data.data, data.target

print(f"Dataset:    {X.shape[0]} samples, {X.shape[1]} features")
print(f"Target:     {data.target_names}")
print(f"Class dist: {dict(zip(*np.unique(y, return_counts=True)))}")

import numpy as np

# 2. Split first — BEFORE any preprocessing
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# 3. Preprocess — fit only on training data
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # fit + transform on train
X_test_scaled  = scaler.transform(X_test)       # transform only on test

# 4. Train
model = GradientBoostingClassifier(n_estimators=200, random_state=42)
model.fit(X_train_scaled, y_train)

# 5. Cross-validate on training data for unbiased performance estimate
cv_scores = cross_val_score(model, X_train_scaled, y_train, cv=5, scoring="f1_macro")
print(f"\nCV F1: {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")

# 6. Final evaluation on the held-out test set — only do this ONCE
y_pred = model.predict(X_test_scaled)
print("\nTest Set Performance:")
print(classification_report(y_test, y_pred, target_names=data.target_names))

Choosing an Algorithm

Preventing Data Leakage

Data leakage is when information from the test set influences the model, giving artificially inflated metrics. It’s the most common mistake in ML.

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

X = np.random.randn(1000, 20)
y = np.random.randint(0, 2, 1000)

# WRONG: Fit scaler on all data before splitting — test data leaks into training
scaler_wrong = StandardScaler()
X_scaled_all = scaler_wrong.fit_transform(X)  # uses test set statistics!
X_train_wrong, X_test_wrong = train_test_split(X_scaled_all, test_size=0.2)

# CORRECT: Split first, then fit only on train
X_train, X_test = train_test_split(X, test_size=0.2, random_state=42)
scaler_correct = StandardScaler()
X_train_correct = scaler_correct.fit_transform(X_train)  # fit on train only
X_test_correct  = scaler_correct.transform(X_test)        # apply to test