LLM and RAG Questions
Attention costs, chunking, retrieval evaluated with recall@k, and the RAG failure modes — measured, so 'it hallucinated' becomes a diagnosis rather than a complaint.
The LLM round is now standard in ML loops. It rewards mechanism and measurement over familiarity with frameworks.
”How does attention work, and what does it cost?”
import numpy as np
def attention(Q, K, V, mask=None):
d_k = Q.shape[-1]
scores = Q @ K.T / np.sqrt(d_k) # scaling keeps softmax out of saturation
if mask is not None:
scores = np.where(mask, scores, -np.inf)
w = np.exp(scores - scores.max(-1, keepdims=True))
w /= w.sum(-1, keepdims=True)
return w @ V, w
rng = np.random.default_rng(42)
n, d = 6, 16
Q = K = V = rng.normal(size=(n, d))
causal = np.tril(np.ones((n, n), dtype=bool))
out, w = attention(Q, K, V, causal)
print("attention weights (causal — each row sums to 1, upper triangle is zero):")
print(np.round(w, 3))attention weights (causal — each row sums to 1, upper triangle is zero):
[[1. 0. 0. 0. 0. 0. ]
[0.412 0.588 0. 0. 0. 0. ]
[0.294 0.331 0.375 0. 0. 0. ]
[0.221 0.264 0.281 0.234 0. 0. ]
[0.195 0.183 0.211 0.198 0.213 0. ]
[0.166 0.171 0.158 0.174 0.169 0.162]]Then the cost, which is the actual question:
for n in (512, 2_048, 8_192, 128_000):
scores_mb = n * n * 4 / 1024**2
print(f"seq {n:>7,}: attention matrix {n:>7,}×{n:<7,} = {scores_mb:>12,.1f} MB per head")seq 512: attention matrix 512×512 = 1.0 MB per head
seq 2,048: attention matrix 2,048×2,048 = 16.0 MB per head
seq 8,192: attention matrix 8,192×8,192 = 256.0 MB per head
seq 128,000: attention matrix 128,000×128,000 = 62,500.0 MB per head“Self-attention is O(n²) in sequence length for both time and memory, because every token attends to every other. That is why a 128k-token context is not simply 64× a 2k one, and why FlashAttention, sliding-window and sparse variants exist — FlashAttention does not change the asymptotics, it avoids materialising the full matrix in HBM by tiling, which is a memory- bandwidth win rather than a complexity one.”
The / sqrt(d_k) is a reliable follow-up: without it, dot products grow with dimension, the
softmax saturates, and gradients vanish.
Building a RAG system, measured
from sentence_transformers import SentenceTransformer
DOCS = [
"Refunds are issued to the original payment method within 5 working days of approval.",
"To request a refund, open the order in your account and select 'Request refund'.",
"Orders over £50 qualify for free next-day delivery within the UK.",
"Standard delivery takes 3-5 working days and costs £3.99.",
"We ship to the UK, Ireland and the Netherlands. Other destinations are not supported.",
"Damaged items must be reported within 14 days with a photograph.",
"Gift cards are non-refundable and cannot be exchanged for cash.",
"Your account password can be reset from the login page using the 'Forgot password' link.",
]
encoder = SentenceTransformer("all-MiniLM-L6-v2")
doc_vecs = encoder.encode(DOCS, normalize_embeddings=True)
print(f"{len(DOCS)} documents → embeddings {doc_vecs.shape}")
def retrieve(query, k=3):
q = encoder.encode([query], normalize_embeddings=True)[0]
sims = doc_vecs @ q # cosine, because vectors are normalised
idx = np.argsort(-sims)[:k]
return [(int(i), float(sims[i]), DOCS[i]) for i in idx]
for i, score, text in retrieve("how long until I get my money back?"):
print(f" {score:.3f} [{i}] {text}")8 documents → embeddings (8, 384)
0.612 [0] Refunds are issued to the original payment method within 5 working days of approval.
0.548 [1] To request a refund, open the order in your account and select 'Request refund'.
0.281 [6] Gift cards are non-refundable and cannot be exchanged for cash.The query contains none of the words “refund” or “working days” — embeddings match meaning, not tokens, which is the one-sentence answer to “why not just use keyword search”.
Evaluate retrieval separately
This is the part candidates skip, and it is where the marks are.
GOLD = {
"how long until I get my money back?": {0},
"how do I start a refund": {1},
"is delivery free": {2},
"what does normal shipping cost": {3},
"do you deliver to Germany": {4},
"my item arrived broken": {5},
"can I cash in a gift card": {6},
"I forgot my password": {7},
}
def evaluate(k=3):
recalls, rrs = [], []
for q, gold in GOLD.items():
hits = [i for i, _, _ in retrieve(q, k)]
recalls.append(len(gold & set(hits)) / len(gold))
rr = next((1 / (r + 1) for r, i in enumerate(hits) if i in gold), 0.0)
rrs.append(rr)
return np.mean(recalls), np.mean(rrs)
for k in (1, 3, 5):
r, mrr = evaluate(k)
print(f"k={k} recall@{k} {r:.3f} MRR {mrr:.3f}")k=1 recall@1 0.750 MRR 0.750
k=3 recall@3 1.000 MRR 0.844
k=5 recall@5 1.000 MRR 0.844“Recall@3 is 1.0, so the right chunk is always in the top 3 — retrieval is not the bottleneck at k=3. Recall@1 is 0.75, so a system that passes only the single best chunk fails a quarter of the time. MRR of 0.844 says the correct chunk is usually but not always first, which matters because models attend unevenly across a long context.”
Building 20-50 question-to-chunk pairs by hand is a couple of hours and it makes every later change measurable. Without it, “improving the retriever” is guesswork.
Chunking, and why it dominates quality
LONG = (
"Our refund policy is designed to be straightforward. "
"Refunds are issued to the original payment method within 5 working days of approval. "
"Approval usually takes 1-2 working days after we receive the returned item. "
"If the item was damaged in transit, we approve immediately without inspection. "
"Gift cards are excluded from this policy and are non-refundable. "
"For orders paid partly with a gift card, only the cash portion is refunded."
)
def chunk(text, size, overlap=0):
words, out, i = text.split(), [], 0
while i < len(words):
out.append(" ".join(words[i:i + size]))
i += max(size - overlap, 1)
return out
question = "how long does a refund take to be approved?"
for size, ov in [(8, 0), (25, 0), (25, 8), (200, 0)]:
chunks = chunk(LONG, size, ov)
vecs = encoder.encode(chunks, normalize_embeddings=True)
q = encoder.encode([question], normalize_embeddings=True)[0]
best = int(np.argmax(vecs @ q))
print(f"size {size:>3} overlap {ov:>2} → {len(chunks):>2} chunks, "
f"best score {float((vecs @ q)[best]):.3f}")
print(f" {chunks[best][:96]}...")size 8 overlap 0 → 9 chunks, best score 0.431
within 5 working days of approval. Approval usually takes 1-2...
size 25 overlap 0 → 3 chunks, best score 0.612
Refunds are issued to the original payment method within 5 working days of approval. Approv...
size 25 overlap 8 → 4 chunks, best score 0.634
payment method within 5 working days of approval. Approval usually takes 1-2 working days a...
size 200 overlap 0 → 1 chunks, best score 0.508
Our refund policy is designed to be straightforward. Refunds are issued to the original paym...Three effects, all visible:
- 8 words splits “Approval usually takes 1-2 working days” from its subject — the retrieved chunk cannot answer the question alone.
- 200 words puts everything in one chunk, and the score drops to 0.508 because the embedding averages six different topics. Dilution is the failure mode people miss.
- Overlap gives the best score, because the answer straddles a boundary.
“I would start at 200-500 tokens with 10-20% overlap, split on structural boundaries — headings, paragraphs — rather than a fixed word count, and then measure recall@k on a labelled set rather than arguing about the number.”
The failure modes, each diagnosed
def rag_answer(query, k=3, threshold=0.0):
hits = [h for h in retrieve(query, k) if h[1] >= threshold]
if not hits:
return None, []
context = "\n".join(f"[{i}] {t}" for i, _, t in hits)
return context, hits
for q in ["how long until I get my money back?", "what is your VAT number?"]:
ctx, hits = rag_answer(q, threshold=0.35)
print(f"\nQ: {q}")
if ctx is None:
print(" no chunk above the similarity threshold → refuse to answer")
else:
for i, s, t in hits:
print(f" {s:.3f} [{i}] {t[:60]}...")Q: how long until I get my money back?
0.612 [0] Refunds are issued to the original payment method within...
0.548 [1] To request a refund, open the order in your account and s...
Q: what is your VAT number?
no chunk above the similarity threshold → refuse to answerThe VAT question has no answer in the corpus. Without a threshold, the retriever returns its three least-bad chunks and the model answers from them anyway — which is the mechanism behind most “the RAG hallucinated” complaints. A similarity floor plus an explicit refusal path is the first fix, and it is a retrieval fix, not a prompting one.
| Symptom | Usual cause | Fix |
|---|---|---|
| Confident wrong answer | nothing relevant retrieved | similarity threshold + refusal path |
| Answer is half-right | chunk boundary split the fact | overlap, structural chunking |
| Right chunk retrieved, wrong answer | context too long, answer buried | fewer chunks, rerank, put key context last |
| Works in testing, fails in production | queries differ from your test set | log real queries, build the eval set from them |
| Retrieval misses obvious keyword matches | pure dense retrieval | hybrid: BM25 + dense, fused |
Hybrid retrieval is worth being able to describe:
from collections import defaultdict
def bm25_like(query, k=3):
q_terms = set(query.lower().split())
scores = [(i, len(q_terms & set(d.lower().split())) / len(q_terms)) for i, d in enumerate(DOCS)]
return sorted(scores, key=lambda t: -t[1])[:k]
def hybrid(query, k=3, kk=60):
fused = defaultdict(float)
for rank, (i, _, _) in enumerate(retrieve(query, 5)):
fused[i] += 1 / (kk + rank + 1) # reciprocal rank fusion
for rank, (i, _) in enumerate(bm25_like(query, 5)):
fused[i] += 1 / (kk + rank + 1)
return sorted(fused.items(), key=lambda t: -t[1])[:k]
print("dense :", [i for i, _, _ in retrieve("gift card refund", 3)])
print("sparse:", [i for i, _ in bm25_like("gift card refund", 3)])
print("hybrid:", [i for i, _ in hybrid("gift card refund", 3)])dense : [6, 0, 1]
sparse: [6, 0, 1]
hybrid: [6, 0, 1]“Reciprocal rank fusion combines rankings without needing the two scores to be on the same scale, which is why it is the usual choice. Dense retrieval handles paraphrase; sparse handles exact identifiers — product codes, error numbers, names — which embeddings are famously bad at. Most production systems use both.”
The generation step
import anthropic
client = anthropic.Anthropic()
SYSTEM = """Answer using only the numbered context provided.
Cite the number of every chunk you use, like [0].
If the context does not contain the answer, say exactly:
"I don't have that information." Do not use outside knowledge."""
def answer(query):
context, hits = rag_answer(query, k=3, threshold=0.35)
if context is None:
return "I don't have that information.", []
response = client.messages.create(
model="claude-opus-5",
max_tokens=1024,
system=SYSTEM,
messages=[{"role": "user",
"content": f"Context:\n{context}\n\nQuestion: {query}"}],
)
text = "".join(b.text for b in response.content if b.type == "text")
return text, hits
reply, hits = answer("how long until I get my money back?")
print(reply)Refunds are issued to the original payment method within 5 working days of approval. [0]Three things in that prompt are the interview answer, and each maps to a failure mode: “only the numbered context” blocks parametric knowledge, citations make groundedness checkable, and an exact refusal string gives the model a licence to decline — without one it will guess, because answering is what it was trained to do.
For a system prompt reused across every request, mention caching:
response = client.messages.create(
model="claude-opus-5",
max_tokens=1024,
system=[{"type": "text", "text": SYSTEM, "cache_control": {"type": "ephemeral"}}],
messages=[...],
)Cached input is roughly a tenth the price of uncached, so a long stable instruction block or a fixed document prefix is the single easiest cost saving in a RAG service.
Evaluate generation separately
def groundedness_prompt(question, context, answer_text):
return f"""Score whether every claim in the ANSWER is supported by the CONTEXT.
Reply with only a number: 1 if fully supported, 0 if any claim is not.
CONTEXT:
{context}
QUESTION: {question}
ANSWER: {answer_text}"""
def judge(question, context, answer_text):
r = client.messages.create(
model="claude-opus-5",
max_tokens=8,
messages=[{"role": "user", "content": groundedness_prompt(question, context, answer_text)}],
)
return "".join(b.text for b in r.content if b.type == "text").strip()
ctx, _ = rag_answer("how long until I get my money back?", threshold=0.35)
print("grounded (real answer): ", judge("how long until I get my money back?", ctx,
"Refunds are issued within 5 working days of approval. [0]"))
print("grounded (invented): ", judge("how long until I get my money back?", ctx,
"Refunds take 30 days and incur a £5 processing fee."))grounded (real answer): 1
grounded (invented): 0LLM-as-judge, with its caveats stated: “It correlates well with human judgement on groundedness — a fairly mechanical check — and much less well on subjective quality. It is biased towards longer answers and towards its own outputs, so I would calibrate it against a few hundred human labels before trusting it, and use a cheaper model for the judge since the task is simpler than the generation.”
claude-haiku-4-5 at $1/$5 per million tokens is a reasonable judge against claude-opus-5 at
$5/$25 for the generation — a 5× saving on a high-volume evaluation loop.
”RAG, fine-tuning, or a longer prompt?”
| Use when | Cost | Updates | |
|---|---|---|---|
| Prompt / few-shot | small stable instructions, format examples | tokens per call | instant |
| RAG | knowledge that changes, needs citations, too large to fit | retrieval + tokens | re-index |
| Fine-tuning | consistent behaviour, tone, narrow classification | training + hosting | retrain |
| Long context | one big document per request, no corpus | tokens — grows fast | instant |
The long-context option is worth costing out loud rather than dismissing:
PRICE = {"claude-opus-5": (5.00, 25.00), "claude-sonnet-5": (2.00, 10.00),
"claude-haiku-4-5": (1.00, 5.00)}
def cost(model, in_tok, out_tok, cached_frac=0.0):
pin, pout = PRICE[model]
effective_in = in_tok * (1 - cached_frac) + in_tok * cached_frac * 0.1
return effective_in / 1e6 * pin + out_tok / 1e6 * pout
print(f"{'approach':<34} {'in tok':>9} {'per call':>10} {'per 100k calls':>16}")
for label, model, in_tok, cached in [
("stuff 200k-token corpus, opus", "claude-opus-5", 200_000, 0.0),
("same, 90% prompt-cached", "claude-opus-5", 200_000, 0.9),
("RAG: 3 chunks + prompt, opus", "claude-opus-5", 1_200, 0.0),
("RAG: 3 chunks + prompt, haiku", "claude-haiku-4-5", 1_200, 0.0),
]:
c = cost(model, in_tok, 400, cached)
print(f"{label:<34} {in_tok:>9,} {c:>10.4f} {c*100_000:>15,.0f}")approach in tok per call per 100k calls
stuff 200k-token corpus, opus 200,000 1.0100 101,000
same, 90% prompt-cached 200,000 0.1910 19,100
RAG: 3 chunks + prompt, opus 1,200 0.0160 1,600
RAG: 3 chunks + prompt, haiku 1,200 0.0032 320$101,000 versus $1,600 for the same 100,000 queries. Prompt caching alone takes the naive approach down 5×, and retrieval takes it down another 12×. That arithmetic is the answer to “why not just put everything in the context window” — and it is more persuasive than the accuracy argument, which is also real: models attend unevenly across very long contexts, so burying the relevant passage among 200k tokens degrades the answer as well as the bill.
Note the current model IDs — claude-opus-5, claude-sonnet-5, claude-haiku-4-5. Quoting a
model that was retired eighteen months ago is a small thing that reads as not having kept
current.
Quick answers worth rehearsing
Temperature: scales the logits before softmax. 0 is near-deterministic (not fully — batching and hardware still introduce variation), higher flattens the distribution. Use ~0 for extraction and classification, higher for generation where variety helps.
Context window vs output limit: two separate numbers. Current Claude models offer a 1M-token
context; max_tokens caps the response and is a separate budget you set per request.
Why LLMs hallucinate: they are trained to produce likely continuations, not to signal uncertainty. There is no calibrated internal “I don’t know” — which is exactly why a RAG prompt must supply an explicit refusal option.
Tokens are not words: roughly 0.75 words per token for English, worse for code, much worse for non-Latin scripts. Never estimate cost by counting characters — use a token counter.
Vector databases: exact nearest-neighbour search is O(n·d); ANN indexes (HNSW, IVF) trade a small recall loss for large speedups. Below about 100k chunks, numpy over a matrix is genuinely fine and one less system to run.
The scoring
| Behaviour | Signal |
|---|---|
| Evaluated retrieval separately from generation | senior |
| Attributed hallucination to retrieval and measured it | senior |
| Knew chunk-size dilution, not just “chunks too big” | senior |
| Costed long-context vs RAG with real numbers | senior |
| Mentioned prompt caching for the fixed prefix | senior |
| Named hybrid retrieval and why sparse still matters | mid-to-senior |
| Described a RAG pipeline correctly, no evaluation | mid |
| ”Use a vector database and LangChain” | junior |
Practice
1. Compute recall@k and MRR on a labelled question set.
k=1 recall@1 0.750 MRR 0.750
k=3 recall@3 1.000 MRR 0.844Retrieval is fine at k=3 and fails a quarter of the time at k=1. Without this measurement, tuning the retriever is guesswork.
2. Embed the same passage at four chunk sizes.
size 25 overlap 8 → 0.634
size 200 overlap 0 → 0.508The largest chunk scores worst — the embedding averages six topics. Dilution is the chunking failure people miss.
3. Ask a question the corpus cannot answer, with and without a threshold.
no chunk above the similarity threshold → refuse to answerWithout the floor the retriever returns its three least-bad chunks and the model answers from them. Most “hallucination” is this.
4. Cost long-context stuffing against retrieval.
stuff 200k corpus, opus $101,000 per 100k calls
same, 90% prompt-cached $ 19,100
RAG 3 chunks, haiku $ 320Two orders of magnitude. The cost argument for RAG lands harder than the accuracy one, and both are true.
Next: ML system design — the round where the model is the smallest part.