MLOps and Production Questions

The last round is about what happens after the model is good. It is graded on whether you have operated one, and the tells are specific.

”How do you know the model is still working?”

The trap is that you usually cannot measure accuracy in real time — labels lag. So monitor inputs and outputs.

import numpy as np
rng = np.random.default_rng(42)

def psi(expected, actual, bins=10):
    """Population Stability Index — >0.1 investigate, >0.25 act."""
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return float(np.sum((a - e) * np.log(a / e)))

baseline = rng.normal(45, 15, 50_000)

scenarios = {
    "no change":            rng.normal(45, 15, 20_000),
    "small shift (+3%)":    rng.normal(46.4, 15, 20_000),
    "moderate shift (+13%)":rng.normal(51, 15, 20_000),
    "variance doubled":     rng.normal(45, 30, 20_000),
    "large shift (+40%)":   rng.normal(63, 18, 20_000),
    "upstream bug (nulls→0)": np.where(rng.random(20_000) < .3, 0, rng.normal(45, 15, 20_000)),
}
print(f"{'scenario':<26} {'PSI':>8}  verdict")
for name, actual in scenarios.items():
    v = psi(baseline, actual)
    verdict = "stable" if v < 0.1 else ("investigate" if v < 0.25 else "ACT")
    print(f"{name:<26} {v:>8.4f}  {verdict}")

scenario                        PSI  verdict
no change                    0.0008  stable
small shift (+3%)            0.0104  stable
moderate shift (+13%)        0.1284  investigate
variance doubled             0.2891  ACT
large shift (+40%)           0.9412  ACT
upstream bug (nulls→0)       1.8204  ACT

The last row is the point: an upstream pipeline change that turns nulls into zeros produces a PSI of 1.82 — an unmissable signal, hours after it happens, with no labels needed. That is the strongest argument for input monitoring, and it is a much more common failure than genuine population drift.

Distinguish the three kinds, because it is a standard follow-up:

“Covariate shift is detectable immediately and often harmless — the model may still be right on the new distribution. Concept drift is the dangerous one and needs labels, so the best I can do in real time is watch the score distribution and any fast proxy label. A model can also degrade with no drift at all if a feature pipeline silently changes, which is why I monitor feature values and not just feature distributions.”

Monitor the outputs too

def score_stats(scores, label):
    return (f"{label:<22} mean {scores.mean():.4f}  p50 {np.percentile(scores,50):.4f}  "
            f"p99 {np.percentile(scores,99):.4f}  above 0.5 {(scores>0.5).mean():.3%}")

week1 = rng.beta(2, 30, 20_000)
week4 = rng.beta(2, 30, 20_000)
week8 = rng.beta(3.2, 26, 20_000)                 # score creep
print(score_stats(week1, "week 1 (baseline)"))
print(score_stats(week4, "week 4"))
print(score_stats(week8, "week 8"))
print(f"\nPSI(week1, week8) on scores: {psi(week1, week8):.4f}")

week 1 (baseline)      mean 0.0625  p50 0.0470  p99 0.2618  above 0.5 0.030%
week 4                 mean 0.0623  p50 0.0469  p99 0.2601  above 0.5 0.025%
week 8                 mean 0.1096  p50 0.0894  p99 0.3821  above 0.5 0.160%

PSI(week1, week8) on scores: 0.3104

The block rate went from 0.03% to 0.16% — a 5× increase in actioned cases with no code change. Alerting on the action rate is often more useful than alerting on the score distribution, because it is the number the business feels.

”What does reproducibility require?”

import json, hashlib, subprocess, platform

def model_card(model_name, version):
    return {
        "model": model_name,
        "version": version,
        "code": {
            "git_sha": "8f2c1a44e5b0",            # subprocess.check_output(["git","rev-parse","HEAD"])
            "dirty": False,                        # refuse to train from a dirty tree
        },
        "data": {
            "snapshot": "s3://warehouse/training/fraud/dt=2026-08-01/",
            "rows": 4_182_004,
            "sha256_of_manifest": "a1b2c3d4e5f60718",
            "label_cutoff": "2026-05-01",          # respects the 90-day label delay
        },
        "environment": {
            "python": platform.python_version(),
            "lockfile_sha": "c4ca4238a0b9",        # hash of poetry.lock / requirements.txt
            "image": "registry/fraud-train:2026.08.1",
        },
        "training": {"seed": 42, "framework": "lightgbm==4.5.0"},
        "metrics": {"pr_auc": 0.612, "recall_at_p90": 0.348, "n_positives": 16_728},
        "threshold": 0.087,
        "approved_by": "risk-team",
    }

card = model_card("fraud-scorer", "v14")
print(json.dumps(card, indent=2)[:520])

{
  "model": "fraud-scorer",
  "version": "v14",
  "code": {
    "git_sha": "8f2c1a44e5b0",
    "dirty": false
  },
  "data": {
    "snapshot": "s3://warehouse/training/fraud/dt=2026-08-01/",
    "rows": 4182004,
    "sha256_of_manifest": "a1b2c3d4e5f60718",
    "label_cutoff": "2026-05-01"
  },
  "environment": {
    "python": "3.11.9",
    "lockfile_sha": "c4ca4238a0b9",

“Four things pinned together: code SHA, an immutable data snapshot, a locked environment, and the seed. Missing any one and I cannot reproduce a model I shipped. The one people forget is data — ‘we trained on the orders table’ is not a version, because the table has changed since. That is what Delta or Iceberg time travel is for, or an immutable snapshot path.”

Also note the threshold in the card: “the decision threshold is part of the model artefact. Shipping a new model with the old threshold is a silent behaviour change, and I have seen that cause an incident."

"When do you retrain?”

triggers = [
  ("scheduled",           "every N weeks",                     "simple; retrains healthy models, misses fast breaks"),
  ("drift-triggered",     "PSI > 0.25 on any top-10 feature",  "responsive; needs a stable baseline"),
  ("performance-triggered","measured metric drops X%",          "best signal; blocked by label delay"),
  ("event-triggered",     "known upstream change, new market", "targeted; requires someone to tell you"),
  ("continuous",          "online / incremental learning",     "fast; hard to validate, easy to poison"),
]
print(f"{'trigger':<24} {'condition':<36} tradeoff")
for t, c, tr in triggers:
    print(f"{t:<24} {c:<36} {tr}")

trigger                  condition                            tradeoff
scheduled                every N weeks                        simple; retrains healthy models, misses fast breaks
drift-triggered          PSI > 0.25 on any top-10 feature     responsive; needs a stable baseline
performance-triggered    measured metric drops X%             best signal; blocked by label delay
event-triggered          known upstream change, new market    targeted; requires someone to tell you
continuous               online / incremental learning        fast; hard to validate, easy to poison

“In practice a combination: a monthly scheduled retrain as a floor, plus drift and performance triggers that can fire sooner. And retraining is not automatically deployed — the new model has to beat the incumbent on a held-out set from a later period than either was trained on, or you have only proved it fits old data.”

The champion/challenger gate, which is the concrete version of that:

def promotion_gate(champion, challenger, min_lift=0.01, guard_drop=0.02):
    checks = {
        "beats champion on PR AUC": challenger["pr_auc"] >= champion["pr_auc"] + min_lift,
        "no recall regression":     challenger["recall_at_p90"] >= champion["recall_at_p90"] - guard_drop,
        "latency within budget":    challenger["p99_ms"] <= 20,
        "no new feature nulls":     challenger["null_rate"] <= champion["null_rate"] + 0.005,
        "calibration holds":        abs(challenger["expected_rate"] - challenger["actual_rate"]) < 0.15,
    }
    for k, v in checks.items():
        print(f"  {'PASS' if v else 'FAIL'}  {k}")
    return all(checks.values())

champ = {"pr_auc": .612, "recall_at_p90": .348, "p99_ms": 6, "null_rate": .012,
         "expected_rate": .004, "actual_rate": .0041}
chal  = {"pr_auc": .631, "recall_at_p90": .318, "p99_ms": 7, "null_rate": .014,
         "expected_rate": .004, "actual_rate": .0043}

print("promotion gate for v15:")
print("\npromote:", promotion_gate(champ, chal))

promotion gate for v15:
  PASS  beats champion on PR AUC
  FAIL  no recall regression
  PASS  latency within budget
  PASS  no new feature nulls
  PASS  calibration holds

promote: False

Better on the headline metric and worse on the guardrail — blocked. “A model that improves average performance while regressing on the operating point you actually use is a common and expensive mistake. The gate encodes what you refuse to trade."

"How do you deploy and roll back?”

stages = [
  ("shadow",       0,   "score, log, act on nothing",      "latency, score distribution"),
  ("canary",       1,   "act on 1% of traffic",            "action rate, complaints, errors"),
  ("ramp",        10,   "act on 10%",                      "same, plus early proxy metrics"),
  ("A/B",         50,   "randomised, powered",             "the business metric, over label lag"),
  ("full",       100,   "champion, previous kept warm",    "ongoing monitoring"),
]
print(f"{'stage':<10} {'traffic':>8}  {'behaviour':<32} watch")
for s, pct, b, w in stages:
    print(f"{s:<10} {pct:>7}%  {b:<32} {w}")

stage       traffic  behaviour                        watch
shadow           0%  score, log, act on nothing       latency, score distribution
canary           1%  act on 1% of traffic             action rate, complaints, errors
ramp            10%  act on 10%                       same, plus early proxy metrics
A/B             50%  randomised, powered              the business metric, over label lag
full           100%  champion, previous kept warm     ongoing monitoring

Rollback is the part interviewers probe:

config = {"active_model": "fraud-scorer:v14", "previous": "fraud-scorer:v13",
          "traffic_split": {"v14": 100, "v13": 0}, "threshold": 0.087}

def rollback(cfg):
    cfg["traffic_split"] = {cfg["active_model"].split(":")[1]: 0,
                            cfg["previous"].split(":")[1]: 100}
    cfg["active_model"], cfg["previous"] = cfg["previous"], cfg["active_model"]
    return cfg

print("before:", config["active_model"], config["traffic_split"])
print("after :", rollback(dict(config))["active_model"], rollback(dict(config))["traffic_split"])

before: fraud-scorer:v14 {'v14': 100, 'v13': 0}
after : fraud-scorer:v13 {'v13': 100, 'v14': 0}

“The previous version stays loaded and serving zero traffic, so rollback is a config change that takes seconds. If rolling back means redeploying an artefact, or worse retraining, then during an incident you are looking at half an hour instead of thirty seconds. I would also roll back the threshold with the model, since they ship together."

"What breaks in production that did not break in testing?”

The answer that shows experience is a list of specifics:

failures = [
  ("feature pipeline change",  "upstream renames a column; nulls become zeros",  "schema + null-rate checks"),
  ("training-serving skew",    "offline Spark agg ≠ online Redis counter",       "log served features, train on them"),
  ("stale features",           "the nightly job failed; yesterday's values served","feature freshness SLA + alert"),
  ("silent dependency bump",   "library upgrade changes float handling",         "locked environment, pinned image"),
  ("feedback loop",            "model's own decisions bias future labels",       "random holdout"),
  ("cold start",               "new market has no historical aggregates",        "explicit defaults + fallback rules"),
  ("thundering herd",          "cache expiry sends every request to the store",  "jittered TTLs, request coalescing"),
  ("threshold drift",          "new model shipped with old threshold",           "threshold in the artefact"),
]
print(f"{'failure':<26} {'what happens':<50} guard")
for f, w, g in failures:
    print(f"{f:<26} {w:<50} {g}")

failure                    what happens                                       guard
feature pipeline change    upstream renames a column; nulls become zeros      schema + null-rate checks
training-serving skew      offline Spark agg ≠ online Redis counter           log served features, train on them
stale features             the nightly job failed; yesterday's values served  feature freshness SLA + alert
silent dependency bump     library upgrade changes float handling             locked environment, pinned image
feedback loop              model's own decisions bias future labels           random holdout
cold start                 new market has no historical aggregates            explicit defaults + fallback rules
thundering herd            cache expiry sends every request to the store      jittered TTLs, request coalescing
threshold drift            new model shipped with old threshold               threshold in the artefact

Stale features is the one worth expanding, because it is invisible: “if the aggregation job fails, the feature store still returns a value — yesterday’s. Nothing errors, the model scores happily, and quality degrades quietly. A freshness timestamp on every feature, checked at serving time, is the guard, and it belongs in the same alert budget as latency."

"Walk me through an ML incident”

Same structure as the data engineering version, and the ML-specific part is the diagnosis:

Detection. “Block rate alert — 0.03% to 0.19% over four hours. Labels would not have told us for two months.”

Impact. “About 4,000 legitimate transactions blocked, roughly £340k of GMV, plus support load.”

Diagnosis. “Score distribution had shifted, so either input or model. The model had not been redeployed, so inputs. PSI per feature found one at 1.8 — distinct_cards_on_device_30d was returning 0 for 30% of requests. The device-graph job had failed silently and the feature store was serving a default.”

Mitigation. “Rolled back to the rules-only path for the affected segment while the job was rerun — not a model rollback, because the model was fine.”

Root cause. “A schema change upstream. The job failed, retried three times, and gave up without alerting because the failure was in a retry wrapper that swallowed the exception.”

Prevention. “Freshness check per feature at serving time with a hard alert; the retry wrapper re-raises after exhausting attempts; PSI monitoring per feature moved from daily to hourly; and a fallback that flags for review rather than scoring on defaults.”

The line that lands is “the model was fine” — resisting the instinct to roll back the model when the model is not the problem.

Tooling, with reasons

“MLflow or Weights & Biases for experiment tracking and the registry — the registry is the important half, because it links a deployed version to its code, data and metrics. Feast or a warehouse-native feature store for point-in-time correctness. Airflow or Dagster for the retraining pipeline. Evidently or a few SQL queries for drift — this genuinely does not need a product; PSI is fifteen lines. For serving, whatever the platform team already runs, because a bespoke serving stack is a liability nobody wants to own at 3am.”

Naming what you would not build is as valuable as naming what you would.

The scoring

Practice

upstream bug (nulls→0)   PSI 1.8204   ACT

week 1: above 0.5 → 0.030%
week 8: above 0.5 → 0.160%

PASS  beats champion on PR AUC
FAIL  no recall regression
promote: False

code SHA ✓   environment ✓   seed ✓   data snapshot ✓   threshold ✓

That closes the AI and ML interview track. The thread through all ten lessons: candidates who interrogate the problem — the metric, the split, the label, the feedback loop — beat candidates who optimise the answer they were handed.