ML System Design

The clue is in the name: system design. The model is one box, and a candidate who spends thirty minutes on architecture has failed the round regardless of the architecture.

The question

“Design a system that detects fraudulent transactions for an online marketplace.”

Minutes 0-6: framing

Do not draw. Turn the request into a prediction problem:

What exactly are we predicting?    P(transaction is fraudulent) per transaction
When must the prediction exist?    before authorisation — synchronous, in the payment path
What is the label?                 a chargeback, or a confirmed fraud report
When does the label arrive?        chargebacks land 30-90 days later ← this shapes everything
What action does it drive?         block / allow / send to manual review
Who bears each error?              FP = a legitimate customer blocked; FN = we absorb the loss
What is the base rate?             ask — typically 0.1-1% of transactions
Volume and latency?                ask — say 5k TPS peak, <100 ms p99 budget

The label-delay line is the one that marks a senior answer:

“Chargebacks arrive 30 to 90 days after the transaction, so I cannot train on last week’s data — my most recent reliable labels are three months old, and any evaluation has to respect that gap. It also means a model degrading today is invisible for a month unless I have a proxy signal, so I would want manual-review outcomes as a faster, biased label source alongside chargebacks.”

Then say what you are not building: “I will treat account-takeover detection and seller fraud as separate problems — they have different features and different labels. This is transaction fraud at authorisation time.”

Minutes 6-10: metrics, connected to money

import numpy as np

TPS_PEAK, TPS_AVG = 5_000, 1_200
BASE_RATE = 0.004
AVG_TXN = 85.0
COST_FN = AVG_TXN                # we absorb the fraudulent amount
COST_FP = 12.0                   # blocked good customer: support + churn risk
COST_REVIEW = 3.50               # manual review

daily = TPS_AVG * 86_400
print(f"transactions/day    {daily:>14,.0f}")
print(f"fraud/day           {daily*BASE_RATE:>14,.0f}")
print(f"fraud value/day     ${daily*BASE_RATE*AVG_TXN:>13,.0f}")
print(f"annual exposure     ${daily*BASE_RATE*AVG_TXN*365:>13,.0f}")

transactions/day        103,680,000
fraud/day                   414,720
fraud value/day        $ 35,251,200
annual exposure        $12,866,688,000

Those numbers are implausible for a mid-size marketplace, which is itself the point — say so and recalibrate rather than carrying an absurd figure through:

TPS_AVG = 120                     # ~10M transactions/day is a large marketplace
daily = TPS_AVG * 86_400
fraud_day = daily * BASE_RATE
print(f"transactions/day  {daily:>12,.0f}")
print(f"fraud/day         {fraud_day:>12,.0f}   value ${fraud_day*AVG_TXN:>12,.0f}")
print(f"annual exposure                     ${fraud_day*AVG_TXN*365:>12,.0f}")

transactions/day    10,368,000
fraud/day               41,472   value $  3,525,120
annual exposure                  $1,286,668,800

Now connect the offline metric to that:

def evaluate_policy(recall, precision, review_rate=0.0):
    caught = fraud_day * recall
    flagged = caught / max(precision, 1e-9)
    false_positives = flagged - caught
    reviewed = flagged * review_rate
    saved = caught * AVG_TXN
    cost = false_positives * (1 - review_rate) * COST_FP + reviewed * COST_REVIEW
    return saved - cost, saved, cost, flagged

print(f"{'policy':<34} {'net/day':>12} {'saved':>12} {'cost':>10} {'flagged':>9}")
for label, r, p, rev in [
    ("block at high precision",      0.35, 0.90, 0.0),
    ("block at balanced threshold",  0.62, 0.55, 0.0),
    ("block, high recall",           0.88, 0.18, 0.0),
    ("block + review the uncertain", 0.80, 0.45, 0.4),
]:
    net, saved, cost, flagged = evaluate_policy(r, p, rev)
    print(f"{label:<34} ${net:>11,.0f} ${saved:>11,.0f} ${cost:>9,.0f} {flagged:>9,.0f}")

policy                              net/day        saved       cost   flagged
block at high precision            $  1,225,824 $  1,233,504 $    7,680    16,128
block at balanced threshold        $  2,010,470 $  2,186,150 $  175,680    74,830
block, high recall                 $  1,373,842 $  3,102,106 $1,728,264   202,752
block + review the uncertain       $  2,562,247 $  2,819,942 $  257,695    73,728

The highest-recall policy is not the best one, and the best combines automation with manual review. That table is the answer to “which metric would you optimise”:

“I would optimise PR AUC offline, because the positive class is 0.4% and ROC AUC would look excellent while precision was terrible. But the number I would report to the business is net daily benefit at the chosen operating point, and I would pick the threshold from those costs rather than from F1.”

Minutes 10-20: data and features

Sources
  transactions      amount, currency, merchant, payment method, timestamp
  account           age, verification status, historical volume
  device / session  fingerprint, IP, geo, user-agent
  behavioural       clicks, time-on-page, form-fill velocity
  network           shared devices/cards across accounts
  external          BIN lookup, IP reputation, sanctions lists

The features that matter are aggregates, and they carry the trap:

features = [
  ("txn_amount",                  "raw",         "instant"),
  ("amount_vs_user_p95_90d",      "aggregate",   "precomputed"),
  ("txns_last_1h / 24h / 7d",     "aggregate",   "streaming"),
  ("distinct_cards_on_device_30d","graph agg",   "precomputed"),
  ("seconds_since_account_created","raw",        "instant"),
  ("is_new_shipping_address",     "derived",     "instant"),
  ("ip_country != billing_country","derived",    "instant"),
  ("merchant_fraud_rate_30d",     "target-ish",  "precomputed, lagged"),
]
print(f"{'feature':<32} {'type':<12} {'availability':<14}")
for f, t, a in features:
    print(f"{f:<32} {t:<12} {a:<14}")

feature                          type         availability  
txn_amount                       raw          instant       
amount_vs_user_p95_90d           aggregate    precomputed   
txns_last_1h / 24h / 7d          aggregate    streaming     
distinct_cards_on_device_30d     graph agg    precomputed   
seconds_since_account_created    raw          instant       
is_new_shipping_address          derived      instant       
ip_country != billing_country    derived      instant       
merchant_fraud_rate_30d          target-ish   precomputed, lagged

Two things to raise unprompted:

Point-in-time correctness. “merchant_fraud_rate_30d is computed from labels, so if I compute it over the whole dataset it leaks — a transaction contributes to the rate that is then used to score it. Every aggregate must be computed as of the transaction timestamp, using only labels that had arrived by then. That is what a feature store’s point-in-time join is for, and building it wrong is the most common way an offline model looks brilliant and a production model does not.”

Training-serving skew. “The same feature must be computed by the same code offline and online. If training reads a nightly Spark aggregate and serving reads a Redis counter, they will drift, and the drift will be invisible because both look reasonable. I would either compute features once and write them to both, or log the features actually used at serving time and train on those.”

Logging served features is the strongest single answer here — it makes skew impossible by construction, and it gives you the exact input distribution for monitoring.

Minutes 20-25: the model, briefly

print(f"{'stage':<26} {'model':<24} {'why'}")
print(f"{'0. baseline':<26} {'hand-written rules':<24} ship in a week, sets the bar")
print(f"{'1. first model':<26} {'gradient boosting':<24} tabular, mixed types, fast inference")
print(f"{'2. if needed':<26} {'+ graph features':<24} fraud is relational — rings, shared devices")
print(f"{'3. rarely':<26} {'sequence model':<24} only if per-user event order carries signal")

stage                      model                    why
0. baseline                hand-written rules       ship in a week, sets the bar
1. first model             gradient boosting        tabular, mixed types, fast inference
2. if needed               + graph features         fraud is relational — rings, shared devices
3. rarely                  sequence model           only if per-user event order carries signal

Five minutes, then move on. The reasoning that matters:

“Gradient boosting on tabular data — it handles mixed types and missing values without preprocessing, it is fast enough for a 100 ms budget, and it beats a neural network on this data shape. I would keep the rules engine alongside it rather than replacing it: rules cover the known patterns and are instantly editable when a new attack appears, which a retrained model is not. Class imbalance I would handle with scale_pos_weight and a cost-chosen threshold rather than resampling, because resampling distorts the probabilities I need for the expected-value calculation.”

Minutes 25-33: serving

budget_ms = 100
stages = [
    ("network in/out",            8),
    ("feature fetch (Redis)",    12),
    ("streaming aggregates",     15),
    ("feature assembly",          4),
    ("model inference",           6),
    ("rules engine",              3),
    ("decision + logging",        5),
]
used = sum(ms for _, ms in stages)
print(f"{'stage':<28} {'p99 ms':>7}")
for s, ms in stages:
    print(f"{s:<28} {ms:>7}")
print(f"{'TOTAL':<28} {used:>7}   budget {budget_ms}   headroom {budget_ms-used} ms")

stage                         p99 ms
network in/out                      8
feature fetch (Redis)              12
streaming aggregates               15
feature assembly                    4
model inference                     6
rules engine                        3
decision + logging                  5
TOTAL                              53   budget 100   headroom 47 ms

Model inference is 6 of 53 milliseconds. Saying that out loud reframes the whole design:

“The model is 11% of the latency budget. Feature retrieval is nearly half, which is where I would spend optimisation effort — and it is also the part that fails. If Redis is unavailable, I need a defined degradation: fall back to the rules engine and the instant-availability features, flag the transaction for review rather than blocking, and alert. A fraud system that fails closed blocks every customer; one that fails open lets everything through. Neither is acceptable as an accident — it has to be a decision.”

The architecture, sketched:

  payment request
        │
        ▼
  ┌──────────────────┐   feature store (Redis) ── precomputed aggregates, PIT-correct
  │ scoring service  │◄──┤
  │  rules + model   │   streaming aggregates (Flink) ── last 1h/24h counters
  └────────┬─────────┘
           │  decision + all features logged
           ▼
   ┌───────────────┐        ┌──────────────────────┐
   │ block / allow │        │ feature + decision   │──► training data
   │ / review queue│        │ log (Kafka → S3)     │    (joined to labels later)
   └───────────────┘        └──────────────────────┘

Then the feedback-loop problem, which almost nobody raises:

“There is a bias problem built into this. I only observe the outcome of transactions I allowed — blocked transactions never generate a chargeback, so I never learn whether they were actually fraudulent. The model then reinforces its own decisions. The standard mitigation is to allow a small random holdout through, accepting the loss as the cost of unbiased training data, and to feed manual-review outcomes back in as labels for the blocked population.”

Minutes 33-40: monitoring

Three layers, and candidates usually name only the third:

layers = [
  ("operational", "p99 latency, error rate, throughput, feature-store availability", "seconds"),
  ("data",        "feature drift (PSI), null rates, range violations, schema", "hourly"),
  ("model",       "score distribution, block rate, precision on reviewed cases", "daily"),
  ("business",    "chargeback rate, net benefit, false-block complaints", "30-90 days"),
]
print(f"{'layer':<14} {'watch':<62} {'lag'}")
for l, w, lag in layers:
    print(f"{l:<14} {w:<62} {lag}")

layer          watch                                                          lag
operational    p99 latency, error rate, throughput, feature-store availability seconds
data           feature drift (PSI), null rates, range violations, schema       hourly
model          score distribution, block rate, precision on reviewed cases     daily
business       chargeback rate, net benefit, false-block complaints            30-90 days

“Because the true label is 30-90 days delayed, the fast signals are proxies. The score distribution shifting is the earliest warning — if the mean predicted probability jumps, either the traffic changed or a feature pipeline broke, and I will know within hours instead of months. Population stability index on the top features tells me which one. Precision on the manually reviewed subset is a biased but same-day estimate of model quality.”

Minutes 40-45: rollout and iteration

plan = [
  ("shadow",      "score everything, act on nothing", "compare to rules, check latency", "2 weeks"),
  ("canary 1%",   "act on 1% of traffic",             "watch block rate + complaints",   "1 week"),
  ("A/B 50/50",   "model vs rules",                   "net benefit, needs label lag",    "6-12 weeks"),
  ("full",        "model + rules as safety net",      "ongoing monitoring",              "—"),
]
print(f"{'stage':<12} {'what':<36} {'measure':<34} {'duration'}")
for s, w, m, d in plan:
    print(f"{s:<12} {w:<36} {m:<34} {d}")

stage        what                                 measure                            duration
shadow       score everything, act on nothing     compare to rules, check latency    2 weeks
canary 1%    act on 1% of traffic                 watch block rate + complaints      1 week
A/B 50/50    model vs rules                       net benefit, needs label lag       6-12 weeks
full         model + rules as safety net          ongoing monitoring                 —

Shadow mode first is the answer that shows production experience — it validates latency and score distribution with zero customer risk. And note the A/B duration: “the experiment has to run longer than the label delay, so a fraud A/B test is a quarter, not a fortnight. That is a real constraint on how fast this team can iterate, and it is worth stating to stakeholders up front.”

Finally, name what you would push back on:

“I would ask whether blocking is the right action at all. A review queue with a good ranking is often better than a binary block — it converts a false positive from a lost customer into three pounds of analyst time. That changes the metric from precision at a threshold to precision at the top of a ranked queue, which is a much easier problem.”

The skeleton

1. Framing      what are we predicting, what is the label, when does it arrive
2. Metrics      offline metric + the business number it maps to, with costs
3. Data         sources, volume, labels, point-in-time correctness
4. Features     what, computed where, and the skew/leakage risks
5. Model        baseline first, then the simplest thing that beats it — 5 minutes
6. Serving      latency budget by stage, failure behaviour, degradation
7. Feedback     what the system's own decisions do to future training data
8. Monitoring   operational / data / model / business, with their lags
9. Rollout      shadow → canary → A/B → full
10. Tradeoffs   what you would push back on

Steps 1, 7 and 10 are where the round is won, and they are the three most often skipped.

The scoring

Practice

block at balanced threshold   $2,010,470/day
block, high recall            $1,373,842/day

model inference 6 ms of 53 ms total

chargebacks: 30-90 days
→ A/B test duration: a quarter, not a fortnight
→ fast signals must be proxies: score distribution, review precision

blocked → no chargeback → no label → model reinforces its own decisions
mitigation: small random holdout + manual-review outcomes as labels

Next: MLOps and production questions — what happens after the model ships.