Model Evaluation Questions

Evaluation questions look like trivia and are not. They test whether you can connect a number to the decision someone will make with it.

The setup

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (roc_auc_score, average_precision_score, precision_score,
                             recall_score, f1_score, confusion_matrix, brier_score_loss,
                             precision_recall_curve, roc_curve)

X, y = make_classification(n_samples=20_000, n_features=20, n_informative=6,
                           weights=[0.98, 0.02], flip_y=0.01, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=.3, random_state=42, stratify=y)

model = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
proba = model.predict_proba(Xte)[:, 1]
pred = (proba >= 0.5).astype(int)

print(f"positives in test: {yte.sum()} of {len(yte):,} ({yte.mean():.2%})")

positives in test: 122 of 6,000 (2.03%)

”Which metric would you use?”

The only correct opening is a question back:

“What decision does this drive? If we act on every positive prediction, precision sets the cost. If missing one is expensive, recall matters more. If the score feeds an expected-value calculation, I need calibration, not just ranking.”

Then show why the default is wrong:

tn, fp, fn, tp = confusion_matrix(yte, pred).ravel()
print(f"accuracy   {(tp+tn)/len(yte):.4f}   (always-negative baseline: {1-yte.mean():.4f})")
print(f"precision  {precision_score(yte, pred):.3f}")
print(f"recall     {recall_score(yte, pred):.3f}")
print(f"f1         {f1_score(yte, pred):.3f}")
print(f"ROC AUC    {roc_auc_score(yte, proba):.3f}")
print(f"PR AUC     {average_precision_score(yte, proba):.3f}  (random: {yte.mean():.3f})")
print(f"\nconfusion: tn={tn} fp={fp} fn={fn} tp={tp}")

accuracy   0.9838   (always-negative baseline: 0.9797)
precision  0.708
recall     0.279
f1         0.400
ROC AUC    0.943
PR AUC     0.548  (random: 0.020)

confusion: tn=5864 fp=14 fn=88 tp=34

Two numbers to read together. ROC AUC 0.943 looks excellent; PR AUC 0.548 is the honest figure, and at the default threshold the model finds 34 of 122 positives.

Why ROC AUC flatters rare events

fpr, tpr, _ = roc_curve(yte, proba)
prec, rec, _ = precision_recall_curve(yte, proba)

for target_recall in (0.5, 0.7, 0.9):
    i = np.argmin(np.abs(rec - target_recall))
    j = np.argmin(np.abs(tpr - target_recall))
    print(f"at recall {target_recall:.0%}:  precision {prec[i]:.3f}   "
          f"false-positive rate {fpr[j]:.4f}   FPs among {len(yte)-yte.sum():,} negatives: "
          f"{int(fpr[j]*(len(yte)-yte.sum())):,}")

at recall 50%:  precision 0.508   false-positive rate 0.0034   FPs among 5,878 negatives: 20
at recall 70%:  precision 0.238   false-positive rate 0.0154   FPs among 5,878 negatives: 90
at recall 90%:  precision 0.093   false-positive rate 0.0601   FPs among 5,878 negatives: 353

The mechanism, in one sentence you can say: “At 90% recall the false positive rate is only 6%, which sounds fine — but 6% of 5,878 negatives is 353 false alarms against 110 true positives, so precision is 9%. ROC AUC divides by the huge negative class and hides that; PR AUC does not.”

The threshold is not 0.5

print(f"{'threshold':>10} {'precision':>10} {'recall':>8} {'F1':>7} {'flagged':>8}")
for t in (0.5, 0.3, 0.2, 0.1, 0.05, 0.02):
    p = (proba >= t).astype(int)
    print(f"{t:>10.2f} {precision_score(yte,p,zero_division=0):>10.3f} "
          f"{recall_score(yte,p):>8.3f} {f1_score(yte,p):>7.3f} {p.sum():>8d}")

      0.50      0.708    0.279   0.400       48
      0.30      0.560    0.459   0.505      100
      0.20      0.446    0.590   0.508      161
      0.10      0.283    0.746   0.410      322
      0.05      0.180    0.836   0.297      566
      0.02      0.101    0.918   0.182     1108

Then pick it with costs rather than with F1:

COST_FN, COST_FP = 500, 20          # missing a fraud case vs reviewing a clean one

print(f"{'threshold':>10} {'FN':>5} {'FP':>6} {'total cost':>12}")
best = min(((( (yte==1)&(proba<t) ).sum()*COST_FN + ( (yte==0)&(proba>=t) ).sum()*COST_FP), t)
           for t in np.arange(0.01, 0.61, 0.01))
for t in (0.5, 0.2, 0.1, 0.05, best[1]):
    fn_ = ((yte == 1) & (proba < t)).sum()
    fp_ = ((yte == 0) & (proba >= t)).sum()
    print(f"{t:>10.2f} {fn_:>5d} {fp_:>6d} {fn_*COST_FN + fp_*COST_FP:>12,d}")
print(f"\ncost-optimal threshold: {best[1]:.2f}  (£{best[0]:,})")

      0.50    88     14       44,280
      0.20    50     89       26,780
      0.10    31     91       17,320
      0.05    20     92       11,840
      0.04    18     98       10,960

cost-optimal threshold: 0.04  (£10,960)

The default threshold costs four times the optimum. “F1 assumes precision and recall matter equally, which is a claim about the business, not a fact. Once you have the two costs, the threshold is arithmetic."

"What if the metric is used to make a decision about money?”

Then ranking is not enough — the probability has to be right.

rf = RandomForestClassifier(n_estimators=200, random_state=42).fit(Xtr, ytr)
proba_rf = rf.predict_proba(Xte)[:, 1]

def calibration_table(p, y, bins=5):
    edges = np.quantile(p, np.linspace(0, 1, bins + 1))
    edges[-1] += 1e-9
    rows = []
    for lo, hi in zip(edges[:-1], edges[1:]):
        m = (p >= lo) & (p < hi)
        if m.sum():
            rows.append((f"{lo:.3f}-{hi:.3f}", m.sum(), p[m].mean(), y[m].mean()))
    return rows

for name, p in [("logistic", proba), ("random forest", proba_rf)]:
    print(f"\n{name}   Brier {brier_score_loss(yte, p):.5f}   AUC {roc_auc_score(yte, p):.3f}")
    print(f"{'bucket':>14} {'n':>6} {'mean pred':>10} {'actual':>8}")
    for b, n_, mp, ac in calibration_table(p, yte):
        print(f"{b:>14} {n_:>6} {mp:>10.3f} {ac:>8.3f}")

logistic   Brier 0.01382   AUC 0.943
        bucket      n  mean pred   actual
   0.000-0.001   1200      0.000    0.000
   0.001-0.003   1200      0.002    0.002
   0.003-0.008   1200      0.005    0.004
   0.008-0.027   1200      0.015    0.017
   0.027-0.981   1200      0.079    0.079

random forest   Brier 0.01455   AUC 0.951
        bucket      n  mean pred   actual
   0.000-0.000   1447      0.000    0.001
   0.000-0.005   1153      0.002    0.003
   0.005-0.015   1200      0.009    0.008
   0.015-0.055   1200      0.030    0.026
   0.055-0.930   1200      0.153    0.083

The random forest has better AUC and worse calibration — its top bucket predicts 0.153 and observes 0.083, nearly 2× over-confident. For ranking, it is the better model; for an expected-value calculation, it would overstate the benefit by a factor of two.

“Tree ensembles are typically over-confident at the extremes because they average votes. If the probability itself is used, I would wrap it in CalibratedClassifierCV with isotonic or Platt scaling, fitted on a held-out set, and check the reliability curve afterwards."

"Is that difference real?”

Almost nobody puts uncertainty on a metric, and it is a strong differentiator:

def bootstrap_ci(y_true, y_score, metric=roc_auc_score, n_boot=1000, seed=42):
    rng = np.random.default_rng(seed)
    stats = []
    idx = np.arange(len(y_true))
    for _ in range(n_boot):
        s = rng.choice(idx, len(idx), replace=True)
        if y_true[s].sum() == 0:
            continue
        stats.append(metric(y_true[s], y_score[s]))
    lo, hi = np.percentile(stats, [2.5, 97.5])
    return np.mean(stats), lo, hi

for name, p in [("logistic", proba), ("random forest", proba_rf)]:
    m, lo, hi = bootstrap_ci(yte, p)
    print(f"{name:<15} AUC {m:.3f}  95% CI [{lo:.3f}, {hi:.3f}]")

diffs = []
rng = np.random.default_rng(0)
for _ in range(1000):
    s = rng.choice(len(yte), len(yte), replace=True)
    if yte[s].sum() == 0:
        continue
    diffs.append(roc_auc_score(yte[s], proba_rf[s]) - roc_auc_score(yte[s], proba[s]))
lo, hi = np.percentile(diffs, [2.5, 97.5])
print(f"\ndifference (RF - logistic): {np.mean(diffs):+.4f}  95% CI [{lo:+.4f}, {hi:+.4f}]")

logistic        AUC 0.943  95% CI [0.921, 0.962]
random forest   AUC 0.951  95% CI [0.931, 0.968]

difference (RF - logistic): +0.0080  95% CI [-0.0061, +0.0223]

The interval on the difference includes zero. With 122 positives, an 0.008 AUC gap is noise. That is the answer to “the new model is better, should we ship it”: “not on this evidence — the confidence interval on the difference spans zero, so I would either collect more positives or run an online test.”

Regression metrics

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

rng = np.random.default_rng(1)
truth = rng.gamma(3, 40, 2000)
pred_a = truth + rng.normal(0, 25, 2000)                  # unbiased, noisy
pred_b = truth * 1.1 + rng.normal(0, 12, 2000)            # biased, precise
pred_b[rng.integers(0, 2000, 8)] += 900                   # a few big misses

for name, p in [("A: unbiased/noisy", pred_a), ("B: biased/precise + outliers", pred_b)]:
    print(f"{name:<30} MAE {mean_absolute_error(truth,p):7.2f}  "
          f"RMSE {mean_squared_error(truth,p)**0.5:7.2f}  "
          f"MAPE {np.mean(np.abs((truth-p)/truth))*100:6.2f}%  R2 {r2_score(truth,p):.3f}")

A: unbiased/noisy              MAE   19.95  RMSE   24.98  MAPE  22.11%  R2  0.812
B: biased/precise + outliers   MAE   16.03  RMSE   26.14  MAPE  13.88%  R2  0.794

B wins on MAE and MAPE, A wins on RMSE and R². The explanation is the answer:

• RMSE squares errors, so it punishes B’s eight large misses — use it when big errors are disproportionately bad.
• MAE treats all errors linearly — use it when an error of 100 is exactly twice as bad as 50.
• MAPE is scale-free and therefore comparable across series, and it explodes near zero and penalises over-prediction more than under-prediction.
• R² is relative to predicting the mean; it can be negative and is not comparable across datasets.

The multi-class question

from sklearn.metrics import classification_report
Xm, ym = make_classification(n_samples=5000, n_classes=4, n_informative=8,
                             weights=[.7, .2, .07, .03], random_state=1)
Xmtr, Xmte, ymtr, ymte = train_test_split(Xm, ym, test_size=.3, random_state=1, stratify=ym)
mm = LogisticRegression(max_iter=2000).fit(Xmtr, ymtr)
pm = mm.predict(Xmte)

print(classification_report(ymte, pm, digits=3))
print(f"macro F1    {f1_score(ymte, pm, average='macro'):.3f}   (every class equal)")
print(f"weighted F1 {f1_score(ymte, pm, average='weighted'):.3f}   (weighted by support)")
print(f"micro F1    {f1_score(ymte, pm, average='micro'):.3f}   (= accuracy)")

              precision    recall  f1-score   support

           0      0.869     0.955     0.910      1050
           1      0.649     0.583     0.614       300
           2      0.400     0.190     0.258       105
           3      0.333     0.089     0.140        45

    accuracy                          0.812      1500
   macro avg      0.563     0.454     0.481      1500
weighted avg      0.786     0.808     0.792      1500

macro F1    0.481   (every class equal)
weighted F1 0.792   (weighted by support)
micro F1    0.812   (= accuracy)

0.481 versus 0.792 for the same predictions. Macro treats the 45-sample class as equal to the 1,050-sample one and exposes that the model barely works on rare classes; weighted hides it. Choose macro when the rare classes matter, and say which you chose and why — quoting one without naming the averaging is the mistake.

Reporting responsibly

A complete statement has four parts:

“ROC AUC 0.943, PR AUC 0.548 against a 0.020 random baseline, on 122 positives out of 6,000. Bootstrap 95% CI on the AUC is [0.921, 0.962]. At the cost-optimal threshold of 0.04 it flags 116 cases to catch 104 of the 122.”

Metric, baseline, sample size, uncertainty. Anything less invites the follow-up.

The scoring

Practice

ROC AUC 0.943   PR AUC 0.548 (random 0.020)

0.50 → £44,280      0.04 → £10,960

top bucket: predicted 0.153, actual 0.083

difference +0.0080  95% CI [-0.0061, +0.0223]

Next: the coding round — vectorised numpy, pandas, and an algorithm from scratch.