Cheat sheet · AI Engineering for Full-Stack Developers
AI Engineering cheat sheet
What a full-stack developer needs to ship an LLM feature safely: the OWASP risks, prompt injection, evals, cost and speed, and logging with guardrails. Treat the model as an untrusted user and enforce rules in code.
blogs · jp-blogs.vercel.app/cheatsheets/series/ai-engineering
The big ideaLesson 0 →
- An LLM app is a normal web app plus a part that reads text from anyone and can be talked into almost anything.
- Model input and output both cross a trust boundary. Validate, encode and authorize on both sides.
- A prompt is not a security control. Rules live in code, auth systems and downstream APIs.
- Most fixes are old engineering: least privilege, validation, rate limits, logging.
OWASP LLM Top 10 (2025)Lesson 0 →
| ID | Risk | In a phrase |
|---|---|---|
| LLM01 | Prompt Injection | Text changes what the model does |
| LLM02 | Sensitive Info Disclosure | Private data comes out |
| LLM03 | Supply Chain | Bad model, data or package comes in |
| LLM04 | Data and Model Poisoning | Training or embedding data is tampered with |
| LLM05 | Improper Output Handling | Output is used unsafely downstream |
| LLM06 | Excessive Agency | The model can do too much |
| LLM07 | System Prompt Leakage | Hidden instructions get read |
| LLM08 | Vector and Embedding Weaknesses | RAG store leaks or is poisoned |
| LLM09 | Misinformation | Confident wrong answers |
| LLM10 | Unbounded Consumption | Runaway cost, load or copying |
The fix for each riskLesson 0 →
- Output (05)
- Encode, parameterize, validate. Never
innerHTMLmodel text - Agency (06)
- Few tools, the user's own permissions, approval for risky steps
- Consumption (10)
- Input and output caps, per-user budget, timeout, step limit
- Vectors (08)
- Filter by tenant and permission inside the query
- Leakage (07)
- Nothing secret in the system prompt
- Disclosure (02)
- Best: do not send the data. Regex redaction is one layer
- Supply chain (03/04)
- Pin and check models, adapters, datasets, packages
Tip: First three controls on a new feature: tool allow-list, output validation and encoding, per-user budgets plus logging.
Prompt injectionLesson 1 →
- Instructions and data share one text channel, so the model cannot reliably tell them apart.
- Direct: the user types the attack. Indirect: it hides in an email, page or file the model reads. Indirect is the serious case: the attacker borrows the victim's permissions.
- A better system prompt lowers risk but never fixes it. Filters and detector prompts are extra layers only.
Watch out: Lethal trifecta: private data + untrusted content + a way to send data out. Cut at least one leg.
Safe agent defencesLesson 1 →
- Least-privilege tools: read-only where possible, the user's own rights.
- Allow-list tools per task (not a block-list).
- Validate tool arguments and model output with schemas.
- Human approval for risky or irreversible actions.
- Separate trusted and untrusted content; track tainted data.
- Sandbox any model-written code.
- Log every decision the policy layer makes.
Watch out: A policy layer limits actions, not words. Wrong or manipulated text still needs output checks, citations and evals.
Evals: a test suite for fuzzy outputLesson 2 →
| Method | Good for | Weakness |
|---|---|---|
| Exact match | Label, number, ID | Breaks on harmless rewording |
| Regex / must-not-contain | Required phrases, leaks | Misses meaning |
| Schema check | Structured output, tool calls | Says nothing on quality |
| Code oracle | Run the SQL, compile the code | Needs a checker per task |
| LLM judge | Tone, faithfulness | Cost, noise, bias |
- Cheap code checks first; a judge only where code cannot score. Calibrate it with human labels.
- Golden set: versioned, grown from production failures, with hard cases and refusals.
- Gate in CI: exit non-zero, print failures, keep the baseline in the repo.
Judges and small test setsLesson 2 →
| Judge bias | Mitigation |
|---|---|
| Position: favours the first answer | Judge both orders; count a win only if both agree |
| Verbosity: longer scores higher | Put length in the rubric; compare at similar length |
| Self-preference: likes its own style | Use a different model family, or code checks |
- 17 of 20 passes is not a true 85%. Report a Wilson interval; it shrinks with the square root of the cases.
- Overlapping intervals mean no claim. Read the paired flips (which cases changed).
- Above temperature 0, run each case several times. Pick the main metric before the run.
Tokens, cost and speedLesson 3 →
- Cost and delay follow tokens. Output is usually the slower, pricier side.
- Time to first token (feel) differs from total time. Streaming helps the first, not the second.
- Order of savings: shorter prompt, retrieve instead of stuffing, trim history, cap output, then caches.
- Prices and tokenizers differ by provider. Read their current docs; never quote from memory.
GOOD [rules][tools] | [question]
BAD [14:03:22][rules][tools] | [q]Prefix cache matches the start, token by token. Stable first, changing last.
Response cache, routing, retriesLesson 3 →
| Cache risk | Defence |
|---|---|
| Stale answer | Short TTL; prompt and content version in the key |
| Wrong-user answer | User or tenant id in the key, or do not cache |
| Near but not same | High threshold; semantic cache for low-risk FAQ only |
| Poisoning, private text | Cache only validated answers; redact; per tenant |
- Normalise the key (case, filler words): in the lesson's demo, 24 model calls fell to 8.
- Route cheap first, escalate when unsure, and measure quality per route.
- Retry only temporary errors (timeout, overload, rate limit), with backoff and jitter, a max attempts, a per-attempt timeout shorter than the total deadline.
Log and guard every requestLesson 4 →
- request / user id
- Link logs, traces, support tickets
- prompt version
- Which prompt produced which result
- model + settings
- Compare after a model change
- token counts
- Cost and size
- latency per step
- Find the slow span
- attempts, outcome
- ok / fallback / blocked / failed
Watch out: Log facts, not private content. Redaction is a layer, not a promise.
Guardrail layersLesson 4 →
- Input limits and rate limits; redact private data.
- Versioned prompts, rolled out gradually and testable by evals.
- Timeouts; validate output shape, retry a few times, then fall back.
- Content checks as extra layers, never guarantees.
- Least-privilege tools; safe fallback answer or a human.
- Circuit breaker: after repeated failures, stop calling and fail fast.
- Kill switch: flip a setting to turn the feature off in seconds.
Interview one-linersLesson 1 →
| Question | Short answer |
|---|---|
| Fix prompt injection with a prompt? | No. Limit what a tricked model can do, in code. |
| Why is the system prompt not a secret? | It can be extracted. Keep secrets and rules out of it. |
| What is denial of wallet? | Abuse that runs up your bill. Caps, budgets, timeouts. |
| Model returns bad JSON? | Validate, retry a few times, fall back, log, add to evals. |
| Scores up, users unhappy? | Golden set drifted from traffic, overfit prompt, or biased judge. |
| Costs doubled, traffic did not? | Check tokens per request, cache hit rate, escalation rate. |
| Incident with harmful output? | Stop the harm first, then investigate, then add an eval. |
A cheat sheet is a reminder, not a teacher. Each card links to the lesson that explains it step by step.