Design a URL Shortener

Step 1: Requirements

Functional:

1. Given a long URL, create a short URL.
2. Visiting the short URL redirects to the long URL.
3. Optional: custom aliases (sho.rt/diwali-sale).
4. Optional: expiry (the link stops working after a date).
5. Optional: basic click analytics (counts, referrers, countries).

Out of scope: user accounts and dashboards, link editing, QR codes.

Non-functional:

• Very high availability for redirects (the service is almost always up). A broken short link breaks every place where it was shared.
• Low latency: redirects should feel instant. p99 under about 50 ms on the server means 99 out of 100 requests are faster than 50 ms.
• Durability: once a link is created, it must not be lost.
• Unpredictable codes: codes should not be easy to guess or to list one by one (enumerate). This protects private links.
• Read-heavy: there are far more redirects (reads) than new links (writes).

Step 2: Estimation

Assumptions:

• 100 million new URLs per month.
• Read:write ratio 100:1, so 10 billion redirects per month.
• Keep links for 10 years.
• About 500 bytes per record (long URL + metadata).

Traffic (there are about 2.6 million seconds in a month):

Storage:

Cache: a cache is fast memory that keeps copies of popular data. If 20% of links get 80% of the traffic, the hottest links from recent traffic need only tens of GB. This fits easily in a Redis cluster (Redis is an in-memory data store).

So this means:

• Writes are small. One primary database (the main one that accepts writes) can handle them.
• Reads need caching (and maybe a CDN or edge redirects) to keep latency low.
• Storage is moderate: a few TB fits in a sharded relational database or in a key-value store like DynamoDB or Cassandra. (A key-value store finds a value by its key, like a dictionary.)
• Code length: we need room for at least 12 billion codes, with a lot of space left over.

How long should the code be?

Base62 means writing numbers with 62 different characters (a–z, A–Z, 0–9). With them, each position can hold 62 values:

7 characters gives about 3.5 trillion codes, roughly 290 times our 10-year need. That is enough room to keep codes spread out and hard to guess.

Step 3: API and data model

API:

POST /api/v1/urlsAuthorization: (API key, optional)Idempotency-Key: (optional){ "longUrl": "https://…", "customAlias": "diwali-sale", "expiresAt": "2027-12-31T23:59:59Z" }201 Created  { "code": "aZ3kQ9x", "shortUrl": "https://sho.rt/aZ3kQ9x", "expiresAt": "…" }409 Conflict      (custom alias taken)400 Bad Request   (invalid or disallowed URL)GET /{code}301 Moved Permanently / 302 Found    Location: https://…original…404 Not Found / 410 Gone (expired)

301 or 302? These are HTTP status codes, the numbers a server sends to say what happened. 301 means "moved permanently" and 302 means "found, but only for now". 409 means "conflict", 410 means "gone" and 404 means "not found".

• 301 (permanent): browsers and CDNs cache (remember) the redirect, so repeat visits may never reach your servers. This means less load, but you lose analytics, and you cannot change or expire the link reliably for users who have cached it.
• 302 / 307 (temporary): every visit comes back to you. You get correct analytics and control, but you get more traffic.
• Many shorteners use 302 with short cache headers (instructions that tell caches how long to keep the answer). They accept a bit more load to keep control and analytics.

Data model:

url_mappings
┌────────────┬──────────────────────────────┬─────────────┬─────────────┬──────────┐
│ code (PK)  │ long_url                     │ created_at  │ expires_at  │ owner_id │
├────────────┼──────────────────────────────┼─────────────┼─────────────┼──────────┤
│ aZ3kQ9x    │ https://www.example-shop…    │ 2027-04-26  │ NULL        │ 881      │
└────────────┴──────────────────────────────┴─────────────┴─────────────┴──────────┘

The main access pattern (how the data is read) is get by code. This is a perfect key-value lookup. Your options are:

• A relational DB (PostgreSQL/MySQL) with code as the primary key (the field that identifies each row). It is simple and reliable, and you can shard it later by code (post 25).
• A key-value / wide-column store (DynamoDB and Cassandra are databases that store each value under a key): made for exactly this kind of lookup at huge scale. They are easy to scale horizontally (by adding more machines).

Either works at this scale. Many teams start with a relational database and move later if they need to (post 29).

Step 4: High-level design

                               ┌───────────────── CREATE PATH ─────────────────┐
Client ──POST /urls──► LB ──► API servers ──► ID/code generator ──► Database (primary)
                                  │                                   │ replicas
                                  └── validate URL (blocklist / safe-browsing check)

                               ┌───────────────── REDIRECT PATH ───────────────┐
Browser ──GET /aZ3kQ9x──► CDN/edge (optional) ──► LB ──► Redirect servers
                                                            │ 1. Redis cache hit? → 302
                                                            │ 2. miss → DB replica → cache → 302
                                                            └── 3. publish click event ──► Kafka ──► Analytics workers ──► Analytics store

• Separate the create path and the redirect path. Redirects are about 100 times more frequent and must be very fast, so they should be able to scale on their own.
• Redirect servers check the cache first and use the database if the cache has no answer. Then they publish a click event asynchronously, which means they send it and do not wait for it (post 18). The event goes to Kafka, a system that stores a stream of messages so that other programs can read them later. They never wait for analytics.
• Analytics (counting and studying the clicks) is processed away from the main redirect path, so if analytics stops working, redirects still work.

Step 5: Deep dives

Deep dive 1: Generating unique short codes

Option A: Hash the long URL. A hash function turns any input into a fixed-size fingerprint (like MD5 or SHA-256). Compute the hash, write it in Base62, and keep the first 7 characters.

base62(sha256("https://www.example-shop.com/…"))[:7] → "aZ3kQ9x"

• ✅ The same URL gives the same code (this removes duplicates by itself, if you want that).
• ❌ Collisions: two different URLs can get the same first 7 characters. You must check the database and try again with a salt (extra random text added to the input). This adds lookups and race conditions (two requests clash at the same time).
• ❌ It is predictable if someone knows the URL. Also, you may not want deduplication, because different users may want separate links and separate analytics.

Option B: A unique counter, Base62-encoded. Base62 encoding turns a big number into a short text. For example, the number 125 needs only two Base62 characters. Give every new URL a unique number, then encode it:

How do you get unique numbers across many servers?

• A single database sequence or auto-increment: a counter that the database raises by one each time. It is simple, but every writer must wait for this one counter (a single point of contention). This is fine at about 40 writes per second.
• Ticket servers / ranges: each API server takes a block of IDs (for example, 1,000 at a time) from a central counter (in a database or ZooKeeper, a service that helps servers agree), then gives them out by itself. This is very fast, and if a block is lost, it only leaves a harmless gap.
• Snowflake-style IDs: 64-bit IDs built from timestamp + machine ID + sequence number. Every server makes IDs on its own, with no need to talk to the others (post 25).

• ✅ No collisions, no retries, and it is very fast.
• ❌ Sequential codes are guessable (2BcP9aF, 2BcP9aG…). Anyone could enumerate (list one by one) the links and see private ones.
• Fix: scramble the ID before encoding it. Use a permutation (a reversible shuffle) or encryption of the number, with a secret key. The codes stay unique but look random.

Option C: Random codes. Generate 7 random Base62 characters, and insert with a uniqueness check. A unique constraint on code is a database rule that rejects a second row with the same code. If a collision happens (rare, because there is so much space), try again with a new code.

• ✅ Unpredictable and simple.
• ❌ The chance of a collision grows as the space fills up. But with 3.5 trillion slots and 12 billion used, it stays low.

Option D: A pre-generated key service. A background service pre-generates random unique codes and stores them in an "unused keys" table. API servers take batches of keys. It is fast when a request arrives and has no collisions. But it is an extra service, and it must stay available too.

A good choice here: random codes with a unique constraint (simple and hard to guess), or range-based / Snowflake IDs with scrambling (no collision checks). Custom aliases use the same table, and a unique constraint rejects duplicates with 409 Conflict.

Deep dive 2: Making redirects fast

• Cache-aside with Redis (post 16): the app checks the cache first, and on a miss it reads the database and saves the answer in the cache. The cache stores code → long_url, with a TTL (time to live, the time after which an entry is removed). Most redirects are served from memory in about 1 ms.
• Cache negative results ("code not found") for a short time. This protects the database from people who guess random codes (this attack is called cache penetration).
• Pre-warm or protect hot links: a viral link can get millions of hits per minute. To pre-warm means to load the cache before the traffic arrives. Use local caches inside each server's own process for the very hottest keys (post 16).
• Edge redirects: CDNs or edge functions (small programs that run near the user) can serve redirects for popular links close to users (post 14). Use short cache times so that expiry and analytics still work well enough.
• Read replicas (extra copies of the database for reading) handle cache misses (post 24).

Deep dive 3: Scaling storage

At a few TB, one well-sized primary database with replicas can work for a while. After that:

• Shard by code (use a hash of the code to pick the shard, post 25). A shard is one part of the data on its own machine. Every redirect has the code, so every lookup goes to exactly one shard.
• Or use a key-value store (DynamoDB, Cassandra) that splits the data into partitions by itself.
• Expired links: a background job deletes them or moves them to cheap storage (archives them). Or use the database's built-in TTL feature if it has one.

Deep dive 4: Analytics without slowing redirects

• Redirects never wait for analytics.
• Use at-least-once delivery (every event arrives, but some may arrive twice) with approximate counts (post 37). An occasional double-counted click is acceptable here.
• Count unique visitors with approximate algorithms (like HyperLogLog). They give a close answer and use very little memory.

Deep dive 5: Reliability and security

• Availability: stateless redirect servers (they keep no data between requests) across several AZs, which are separate data centres (post 40), a cache and database with copies, and possibly multi-region read copies for users around the world.
• Graceful degradation: the system keeps its most important job when a part fails. If analytics is down, keep redirecting (post 44). If the cache is down, use the replicas, and drop some requests on purpose (load shedding) to protect the database.
• Abuse prevention:
  • rate limit URL creation per API key and IP (post 43),
  • check long URLs against lists of known malware and phishing sites (blocklists), and let people report bad links so you can turn them off,
  • do not allow redirects to dangerous schemes (the start of a URL, like javascript:),
  • watch for open-redirect misuse. An open redirect is a link on a trusted site that sends users anywhere, and phishers use it to look trustworthy.
• Privacy: use codes that are hard to guess for private links, and offer links with a password or an expiry date.

Wrap-up

• Key decisions: 7-character Base62 codes (random, or scrambled Snowflake IDs). A key-value access pattern on a relational or key-value store sharded by code. A separate redirect path with a Redis cache (and optional edge caching). 302 redirects with analytics sent asynchronously through Kafka (a system that moves streams of events between programs).
• Trade-offs: 302 gives analytics and control, but it means more traffic than 301. Random codes are hard to guess but need uniqueness checks. Analytics is eventually consistent (the numbers catch up a little later).
• Next steps: multi-region active-active redirects (several regions all serve traffic at the same time), per-user dashboards, link editing, and QR codes.