Cost, Latency and Caching for LLM Features

You already know how to call a model (LLMs, agents and RAG, Vercel AI SDK). This lesson is about what happens when real traffic arrives: the bill, the wait, and the failures. I do the arithmetic for one request, explain time to first token and streaming, then cover prompt design, prompt caching, response caching, model routing, batching, timeouts and budgets. At the end I run a small simulation that shows how much a cache and a router can save.

For vector search, prompt caching and tracing in short form see Full-Stack Lesson 35. For a normal Redis cache see Lesson 18. Measuring quality, so you know a cheaper setup is not worse, is Lesson 2.

Words used in this lesson

• Token: a small piece of text the model reads or writes. It can be a word, part of a word, or punctuation.
• Input tokens / output tokens: tokens you send (the prompt) and tokens the model writes back.
• Context window: the maximum number of tokens the model can handle in one call (input plus output). It differs by model and provider.
• Time to first token (TTFT): how long until the first piece of the answer arrives.
• Total time: how long until the whole answer is finished.
• Streaming: the server sends the answer in small pieces as they are made, not all at the end.
• Prompt (prefix) cache: the provider reuses its work for the start of a prompt that it has seen before, and usually charges less for it.
• Response cache: your own store of finished answers, keyed by the question.
• Semantic cache: a response cache that matches questions by meaning (similar), not by exact text.
• Router: code that picks which model handles a request.
• Escalation: sending a request to a stronger, more expensive model after the cheap one fails or is unsure.
• Backoff: waiting longer after each failed try (for example 200 ms, 400 ms, 800 ms). Jitter adds a small random amount so many clients do not retry at the same moment.
• Budget: a hard limit, per user or per day, on spend or tokens.

What a token is, and why cost follows it

A model does not read letters or words. A tokenizer program first cuts the text into tokens, and the model works on those. Short common words are often one token. Long or rare words, code, and non-English text are often several tokens. The only safe way to know is to count with the provider's own tokenizer, or to read the counts the API returns with each response.

Providers usually bill input tokens and output tokens at different rates, and output tokens usually cost more per token. Some also offer a lower rate for input that was served from a prompt cache. So the shape of the formula is:

cost = (fresh input tokens x input rate)     + (cached input tokens x cached rate)     + (output tokens x output rate)

The numbers in the formula change by provider and by date. I use example numbers, not real prices, in "credits per 1,000 tokens":

Two facts follow from any such table, whatever the real numbers are:

1. Long prompts cost money on every call. A 300-token system prompt sent with each request is paid again for each request.
2. Output is the expensive and slow part. Each output token costs more and is made one after another. Asking for "a short answer, max 3 sentences" saves both money and time.

Cost of one request, with the fake model

The program below (full code later) has a 309-token system prompt and a 6-token question. The first request has a cold prefix. The second sends the same prefix, which the fake provider now has cached:

tokens: system 309 question 6request 1 (cold prefix): { inTok: 315, cachedTok: 0, outTok: 28, cost: 4.27, ttft: 600, ms: 1440 }request 2 (warm prefix): { inTok: 315, cachedTok: 309, outTok: 28, cost: 1.489, ttft: 600, ms: 1440 }

(I put each object on one line here; the real output has them over several lines.) Check by hand for request 1: 315 x 10 / 1000 = 3.15 for input, plus 28 x 40 / 1000 = 1.12 for output, total 4.27. For request 2: 6 x 10 / 1000 = 0.06, plus 309 x 1 / 1000 = 0.309, plus 1.12, total 1.489. Note that output was 26% of the tokens but 26% to 75% of the cost in these example numbers. And note that in my fake model, the cache changed the price but not the time. Real providers say that a prefix cache can also reduce time; I did not model that. Check the provider's documentation.

Time to first token versus total time

Users feel two different delays.

• Time to first token (TTFT) is mostly the wait before the model starts. It grows with input length, because the model reads the whole prompt first, and with how busy the provider is.
• Total time is TTFT plus (output tokens x time per token). A 500-token answer takes much longer to finish than a 20-token one.

In the fake large tier: 600 ms + 28 tokens x 30 ms = 1,440 ms. If the answer were 500 tokens, it would be 600 + 15,000 = 15,600 ms.

Streaming

With streaming, you show text as it arrives. The user waits only for TTFT before seeing something, even if the total time is long. This is the biggest felt speed-up for chat. It does not reduce cost or total time. It has costs of its own:

• You cannot validate a full JSON answer until it ends. For structured output, either stream only the human text, or buffer.
• You cannot easily run an output check (moderation, schema) on text the user has already seen. Decide what you do if a check fails halfway.
• If the user closes the page, make sure you cancel the model call. Otherwise you pay for output nobody reads.

The Vercel AI SDK has helpers for streaming; see Lesson 11 rather than repeating them here.

Prompt design for cost

Before any cache, make the prompt smaller. These are the usual levers:

A common mistake is to put the whole knowledge base into every prompt "to be safe". It is slow, costly, can hit the context limit, and often makes answers worse because the relevant part is buried. Retrieval costs a little engineering and saves tokens on every request.

Prompt (prefix) caching

When many requests start with the same long text (system prompt, tool descriptions, a big shared document), the provider can reuse the work it did on that start. This is prompt caching or prefix caching. The details (how long entries live, the minimum size, whether you must mark the prefix, the discount) differ by provider. Check the provider's current documentation. The shape is the same everywhere I know of, so here is the concept:

• The cache matches the start of the prompt, token by token. It is a prefix match. One changed token early on breaks the match for everything after it.
• So build prompts as stable part first, changing part last.

GOOD (prefix is the same for every user)  [system rules][tool descriptions][shared policy text] | [user question]BAD (a timestamp at the top changes every call, so nothing is reused)  [Today is 2026-10-11 14:03:22][system rules][tool descriptions] | [user question]

What makes a prefix cacheable:

1. It is byte-for-byte the same across requests. Watch for timestamps, request ids, user names, random ordering of tools, or a map printed in a different key order.
2. It is long enough to be worth it (providers set their own minimum).
3. It comes first. Per-user and per-request text goes at the end.
4. Requests come close enough in time that the entry has not expired.

Prompt caching saves money (and sometimes time) on the input side only. The model still writes a fresh answer every time. For the second kind of caching, see the next section.

Response caching

A response cache stores finished answers in your own store (for example Redis, see Lesson 18). A hit costs no model call: no tokens, near-zero delay.

Exact keys

The key is a hash of everything that can change the answer: the normalized question, the model id string, the prompt version, and any settings. Normalizing (lowercase, trim spaces, remove filler) raises the hit rate. This is safe because the text you match on is the text you answered.

Semantic keys

A semantic cache turns the question into an embedding (a list of numbers that captures meaning, see Lesson 35) and returns a stored answer if a past question is close enough (a similarity threshold). "How do I reset my password" and "I forgot my password, how to change it" can share one answer.

This is where risk starts. I did not build a semantic cache in the demo (an embedding needs a real embedding model; a fake would prove nothing about quality). The risks are conceptual:

The demo at the end shows the wrong-user bug with a simple key.

Model routing

Not every request needs the strongest model. A router sends easy requests to a cheap, fast model and uses the strong model only when needed.

Three simple styles:

1. Rules first: short question with no risky words goes small; anything with money, legal or medical terms goes large.
2. Cascade (cheap first, escalate): ask the small model. If it says it is unsure, or its answer fails a check, ask the large model. You pay for both on escalated requests, so this only wins when most requests stay on the small model.
3. A classifier: a small model or a cheap function labels the request, then you choose. Another call is added, so count its cost.

The danger is quality, not money. A router that sends hard questions to the cheap model saves cost and hurts users. So: run your evaluation set per route (Lesson 2), log which route each request took (Lesson 4), and watch the escalation rate. If the escalation rate climbs, the cheap model is no longer enough, or the traffic changed.

Batching

Two different meanings, so be clear which one you mean:

• Batch many independent jobs into one non-urgent run, for example labelling 50,000 old tickets overnight. Some providers offer a separate batch interface with a lower price and a longer wait. Check the provider's documentation for whether and how. Use it for work no user is waiting for.
• Put several small items in one prompt ("classify these 20 titles") to share the instruction text. It saves repeated input tokens. The risks: one bad item can confuse the others, the output is longer, and one failure fails the whole batch. Validate every item.

For live chat, neither helps much. The live-chat savings come from caching, routing and short prompts.

Timeouts, retries and budgets

Model calls are slow and sometimes fail. Three controls keep that from becoming an outage or a surprise bill.

Timeouts. Set one for each attempt and one total deadline for the request. The attempt timeout must be shorter than the total deadline, or the deadline cannot work (my demo shows this mistake on purpose).

Retries with backoff. Retry only errors that may go away: overloaded, timeout, rate-limited. Do not retry a bad request, because the same input fails again. Wait longer after each failure and add jitter. Cap the number of attempts. Each retry costs tokens if the first try did any work, and many clients retrying together make an overload worse.

Budgets. Limit what one user, one tenant and the whole app can spend per day. Check before the call with an estimate (input tokens are known; output is capped by your setting), and record the real cost after. This ties to "unbounded consumption" in Lesson 0, LLM10. Also cap the number of agent steps and the output size per call.

Build it: a cache and a router, with numbers

The file below has the fake model, a whitespace tokenizer (stand-in), the example price table, and five ways to serve the same 200 requests. The traffic is only 8 distinct questions in different casing, repeated, so it is very repetitive on purpose. The small tier says UNSURE for questions with words such as refund, legal, contract, dispute. That rule is my fake stand-in for "the cheap model cannot handle this". It is not how real models behave.

cost.ts (run with node cost.ts; Node 22 strips the types itself):

// Stand-in tokenizer: one token per whitespace-separated word.// Real tokenizers split text differently (often into sub-word pieces).const countTokens = (text: string): number =>  text.trim() === "" ? 0 : text.trim().split(/\s+/).length;// EXAMPLE NUMBERS, NOT REAL PRICES. Units: "credits" per 1,000 tokens.type Tier = { name: string; inPer1k: number; outPer1k: number; cachedInPer1k: number;              msToFirstToken: number; msPerOutToken: number };const SMALL: Tier = { name: "small", inPer1k: 1, outPer1k: 4, cachedInPer1k: 0.1, msToFirstToken: 200, msPerOutToken: 10 };const LARGE: Tier = { name: "large", inPer1k: 10, outPer1k: 40, cachedInPer1k: 1, msToFirstToken: 600, msPerOutToken: 30 };type Usage = { inTok: number; cachedTok: number; outTok: number; cost: number; ms: number; ttft: number };// Fake model. Deterministic. "Answer quality" is a rule: the small tier only// handles questions that contain no word from HARD_WORDS; otherwise it says UNSURE.const HARD_WORDS = ["refund", "legal", "contract", "dispute"];class FakeModel {  private prefixSeen = new Set<string>(); // prefixes the "provider" has cached  calls = 0;  tier: Tier;  constructor(tier: Tier) { this.tier = tier; }  call(prefix: string, question: string): { text: string; unsure: boolean; usage: Usage } {    this.calls++;    const prefixTok = countTokens(prefix);    const qTok = countTokens(question);    const cached = this.prefixSeen.has(prefix) ? prefixTok : 0;    this.prefixSeen.add(prefix);    const hard = HARD_WORDS.some((w) => question.toLowerCase().includes(w));    const unsure = this.tier.name === "small" && hard;    const text = unsure ? "UNSURE" : `Answer (${this.tier.name}) to: ${question.trim().split(/\s+/).slice(0, 4).join(" ")} ... ` + "word ".repeat(20).trim();    const inTok = prefixTok + qTok;    const outTok = countTokens(text);    const cost = ((inTok - cached) * this.tier.inPer1k + cached * this.tier.cachedInPer1k + outTok * this.tier.outPer1k) / 1000;    const ttft = this.tier.msToFirstToken;    return { text, unsure, usage: { inTok, cachedTok: cached, outTok, cost, ttft, ms: ttft + outTok * this.tier.msPerOutToken } };  }}// ---- Part 1: arithmetic for one request ----const SYSTEM = ("You are a support assistant. Be short and polite. " + "Policy line. ".repeat(150)).trim();const q0 = "How do I reset my password";{  const m = new FakeModel(LARGE);  const a = m.call(SYSTEM, q0).usage;  const b = m.call(SYSTEM, q0).usage;  console.log("tokens: system", countTokens(SYSTEM), "question", countTokens(q0));  console.log("request 1 (cold prefix):", a);  console.log("request 2 (warm prefix):", b);}// ---- Part 2: traffic and strategies ----const base = ["How do I reset my password", "Where is my invoice", "How do I change my email", "Can I get a refund for last month",  "I want to dispute a charge", "What are your opening hours", "How do I delete my account", "Explain the contract terms"];const traffic: string[] = [];for (let i = 0; i < 200; i++) {  const q = base[(i * 7 + (i >> 2)) % base.length]; // deterministic, repeated mix  traffic.push(i % 5 === 0 ? q.toLowerCase() + " please" : i % 3 === 0 ? q.toUpperCase() : q);}type Report = { name: string; cost: number; avgMs: number; modelCalls: number };const report = (name: string, cost: number, ms: number, calls: number): Report =>  ({ name, cost: +cost.toFixed(2), avgMs: Math.round(ms / traffic.length), modelCalls: calls });// A: always the large model, no cachefunction runA(): Report {  const m = new FakeModel(LARGE); let c = 0, t = 0;  for (const q of traffic) { const r = m.call(SYSTEM, q); c += r.usage.cost; t += r.usage.ms; }  return report("A large, prefix cache only", c, t, m.calls);}// B: exact-match response cache (key = exact string)function runB(): Report {  const m = new FakeModel(LARGE); const cache = new Map<string, string>(); let c = 0, t = 0;  for (const q of traffic) {    if (cache.has(q)) { t += 1; continue; }    const r = m.call(SYSTEM, q); cache.set(q, r.text); c += r.usage.cost; t += r.usage.ms;  }  return report("B + exact response cache", c, t, m.calls);}// C: normalized key (lowercase, strip 'please', collapse spaces). Still exact, just cleaner.const norm = (q: string) => q.toLowerCase().replace(/\bplease\b/g, "").replace(/\s+/g, " ").trim();function runC(): Report {  const m = new FakeModel(LARGE); const cache = new Map<string, string>(); let c = 0, t = 0;  for (const q of traffic) {    const k = norm(q);    if (cache.has(k)) { t += 1; continue; }    const r = m.call(SYSTEM, q); cache.set(k, r.text); c += r.usage.cost; t += r.usage.ms;  }  return report("C + normalized key", c, t, m.calls);}// D: C plus a router: small first, escalate to large when small says UNSUREfunction runD(): Report {  const small = new FakeModel(SMALL), large = new FakeModel(LARGE);  const cache = new Map<string, string>(); let c = 0, t = 0;  for (const q of traffic) {    const k = norm(q);    if (cache.has(k)) { t += 1; continue; }    let r = small.call(SYSTEM, q); c += r.usage.cost; t += r.usage.ms;    if (r.unsure) { r = large.call(SYSTEM, q); c += r.usage.cost; t += r.usage.ms; }    cache.set(k, r.text);  }  return report("D + router (small first)", c, t, small.calls + large.calls);}// E: router with NO cache, to show the router alonefunction runE(): Report {  const small = new FakeModel(SMALL), large = new FakeModel(LARGE); let c = 0, t = 0, esc = 0;  for (const q of traffic) {    let r = small.call(SYSTEM, q); c += r.usage.cost; t += r.usage.ms;    if (r.unsure) { esc++; r = large.call(SYSTEM, q); c += r.usage.cost; t += r.usage.ms; }  }  console.log("router only: escalations", esc, "of", traffic.length);  return report("E router only, no response cache", c, t, small.calls + large.calls);}console.table([runA(), runB(), runC(), runE(), runD()]);// ---- Part 3: wrong-user risk with a cache key that forgets the user ----const cacheBad = new Map<string, string>();const answerFor = (user: string, q: string) => `Your balance is ${user === "ana" ? 120 : 9} (for ${user}) -- ${q}`;function badLookup(user: string, q: string) { const k = norm(q); if (!cacheBad.has(k)) cacheBad.set(k, answerFor(user, q)); return cacheBad.get(k)!; }console.log("bad key, ana :", badLookup("ana", "What is my balance"));console.log("bad key, ben :", badLookup("ben", "what is my balance"));const cacheOk = new Map<string, string>();function okLookup(user: string, q: string) { const k = user + "|" + norm(q); if (!cacheOk.has(k)) cacheOk.set(k, answerFor(user, q)); return cacheOk.get(k)!; }console.log("good key, ben:", okLookup("ben", "what is my balance"));

Output (real, from my run):

tokens: system 309 question 6request 1 (cold prefix): {  inTok: 315,  cachedTok: 0,  outTok: 28,  cost: 4.27,  ttft: 600,  ms: 1440}request 2 (warm prefix): {  inTok: 315,  cachedTok: 309,  outTok: 28,  cost: 1.489,  ttft: 600,  ms: 1440}router only: escalations 75 of 200┌─────────┬────────────────────────────────────┬────────┬───────┬────────────┐│ (index) │ name                               │ cost   │ avgMs │ modelCalls │├─────────┼────────────────────────────────────┼────────┼───────┼────────────┤│ 0       │ 'A large, prefix cache only'       │ 300.24 │ 1440  │ 200        ││ 1       │ 'B + exact response cache'         │ 38.51  │ 174   │ 24         ││ 2       │ 'C + normalized key'               │ 14.68  │ 59    │ 8          ││ 3       │ 'E router only, no response cache' │ 136.52 │ 919   │ 275        ││ 4       │ 'D + router (small first)'         │ 8.4    │ 38    │ 11         │└─────────┴────────────────────────────────────┴────────┴───────┴────────────┘bad key, ana : Your balance is 120 (for ana) -- What is my balancebad key, ben : Your balance is 120 (for ana) -- What is my balancegood key, ben: Your balance is 9 (for ben) -- what is my balance

How to read it:

Do not read "97% saved" as a promise. It came from 8 distinct questions among 200 requests. If your users rarely repeat themselves, the response cache hits rarely and the router and short prompts do most of the work. Measure your own hit rate and escalation rate.

The same file's last lines show the wrong-user bug. A cache key made from the question alone served Ana's balance to Ben. Adding the user id to the key fixed it. This is not a hypothetical: it is the most common way a response cache leaks data.

Retries, backoff, deadline and budget

retry.ts uses a simulated clock (no real waiting) and a fake model that follows a script:

// Simulated clock: no real waiting, so the run is instant and repeatable.let now = 0;const sleep = (ms: number) => { now += ms; };class ProviderError extends Error {  retryable: boolean;  constructor(msg: string, retryable: boolean) { super(msg); this.retryable = retryable; }}// Fake model: fails with a scripted pattern, then answers. Each call takes 300 ms.function makeFake(script: Array<"ok" | "overloaded" | "badrequest" | "slow">) {  let i = 0;  return () => {    const step = script[Math.min(i++, script.length - 1)];    if (step === "slow") { sleep(5000); throw new ProviderError("timeout after 5000 ms", true); }    sleep(300);    if (step === "overloaded") throw new ProviderError("overloaded", true);    if (step === "badrequest") throw new ProviderError("bad request", false);    return "answer";  };}// Deterministic "jitter" (a real one uses Math.random()).const jitter = (n: number) => (n * 37) % 50;function callWithRetry(fn: () => string, o: { maxAttempts: number; baseMs: number; deadlineMs: number }) {  const start = now;  for (let attempt = 1; ; attempt++) {    try { return { ok: true, value: fn(), attempt, spent: now - start }; }    catch (e) {      const err = e as ProviderError;      const wait = o.baseMs * 2 ** (attempt - 1) + jitter(attempt);      const outOfTime = now - start + wait > o.deadlineMs;      console.log(`  attempt ${attempt} failed: ${err.message}; retryable=${err.retryable}; next wait ${wait} ms; elapsed ${now - start} ms`);      if (!err.retryable || attempt >= o.maxAttempts || outOfTime) return { ok: false, value: err.message, attempt, spent: now - start };      sleep(wait);    }  }}const opts = { maxAttempts: 4, baseMs: 200, deadlineMs: 3000 };console.log("case 1: two overloads then success");console.log(callWithRetry(makeFake(["overloaded", "overloaded", "ok"]), opts));console.log("case 2: bad request is not retried");console.log(callWithRetry(makeFake(["badrequest", "ok"]), opts));console.log("case 3: slow calls hit the overall deadline");console.log(callWithRetry(makeFake(["slow", "slow", "ok"]), opts));// Per-user daily budget (credits). Check BEFORE the call with an estimate, record AFTER with the real cost.const spent = new Map<string, number>();function withBudget(user: string, estimate: number, limit: number, run: () => number) {  const used = spent.get(user) ?? 0;  if (used + estimate > limit) return `refused: ${user} used ${used.toFixed(1)} of ${limit}`;  const real = run();  spent.set(user, used + real);  return `ok: ${user} now at ${(used + real).toFixed(1)} of ${limit}`;}for (let i = 0; i < 4; i++) console.log(withBudget("ana", 3, 10, () => 3.4));

case 1: two overloads then success  attempt 1 failed: overloaded; retryable=true; next wait 237 ms; elapsed 300 ms  attempt 2 failed: overloaded; retryable=true; next wait 424 ms; elapsed 837 ms{ ok: true, value: 'answer', attempt: 3, spent: 1561 }case 2: bad request is not retried  attempt 1 failed: bad request; retryable=false; next wait 237 ms; elapsed 300 ms{ ok: false, value: 'bad request', attempt: 1, spent: 300 }case 3: slow calls hit the overall deadline  attempt 1 failed: timeout after 5000 ms; retryable=true; next wait 237 ms; elapsed 5000 ms{ ok: false, value: 'timeout after 5000 ms', attempt: 1, spent: 5000 }ok: ana now at 3.4 of 10ok: ana now at 6.8 of 10ok: ana now at 10.2 of 10refused: ana used 10.2 of 10

Three lessons from the output:

• Case 1: two overloads, then success on attempt 3, after 1,561 simulated ms. The waits were 237 and 424 ms (base x 2^n plus a fixed stand-in for jitter).
• Case 2: a bad request is not retried. Retrying would only waste money.
• Case 3 is a bug I left in on purpose. The first attempt took 5,000 ms, which is longer than the whole 3,000 ms deadline, because I never gave the attempt its own timeout. The retry loop stopped, but only after the damage. Fix: give each attempt a timeout smaller than the remaining deadline.
• The budget check let ana reach 10.2 of 10, because the estimate (3) was lower than the real cost (3.4). Budgets based on estimates can overshoot a little. Use a conservative estimate, or a hard stop mid-stream.

Try it

1. A prompt has 2,000 input tokens and the answer has 300. Using the example "large" rates in this lesson (10 in, 40 out per 1,000), what is the cost?
2. Your prompt starts with Current time: <ISO timestamp>. Why does your prompt cache never hit, and what is the fix?
3. You add a semantic cache for "What is my order status?". What can go wrong, and what do you add to the key?
4. Your router sends 80% of requests to the small tier and 20% escalate. Each small call costs 1 unit, each large call costs 10. Cost per 100 requests versus all-large?

Answers.

1. Input: 2,000 x 10 / 1000 = 20. Output: 300 x 40 / 1000 = 12. Total 32 credits. Output is 15% of tokens and 37.5% of the cost.
2. The timestamp changes every call, so no two prompts share a prefix. Move it to the end (or drop it, or round it to the day if the answer does not need seconds).
3. It can return Ana's order status to Ben. Include the user or tenant id in the key, or do not cache personal answers at all. Also add a short TTL, since order status changes.
4. In a cascade every request first goes to the small model: 100 x 1 = 100. The 20 that escalate also cost 20 x 10 = 200. Total 300, versus 1,000 for all-large. If the small model had to be called for all and 60% escalated, it would be 100 + 600 = 700, still less, but the margin shrinks as escalation grows.

Interview questions

Key takeaways

• Cost and delay follow tokens. Input and output are billed differently, and output is usually the slower, pricier part.
• Count tokens with the provider's tokenizer, and take prices from the provider's current documentation. My whitespace tokenizer and prices in this lesson are stand-ins.
• Shorten prompts and retrieve instead of stuffing before you add any cache.
• Put stable text first and changing text last so a prefix can be cached.
• Response caches need careful keys (user, prompt version, content version), TTLs, and validated answers only.
• Route cheap first, escalate when unsure, and measure quality per route.
• Set a per-attempt timeout, a total deadline, capped retries with backoff and jitter, and per-user budgets.
• Measure your own hit rate and escalation rate. The demo's savings came from repetitive made-up traffic.

What's next

Lesson 4: Observability and guardrails for LLM features in production covers what to log per request (and what never to log), tracing an agent, online checks, circuit breakers, kill switches and an incident playbook.