Design File Upload (Including Large Files)

A file upload looks like one form field. At scale it is a whole pipeline, with security, reliability and cost problems. The AWS S3 side is covered in the DevOps series. Here is the design.

Object storage is a service (like Amazon S3) that stores files as flat "objects". Each object has a key (its name or path, for example uploads/42/report.pdf). You read and write whole objects through an HTTP API, not through a folder on a disk.

A few more terms used below:

• Upload pipeline is the chain of steps a file goes through, from the browser to its final, safe, usable form.
• API server is the program that answers your app's HTTP requests (log in, save a comment, and so on).
• Metadata is data about a file (its name, size, owner, type), not the file's bytes.
• Background processing is work done by a separate program after the request has ended, so the user does not wait for it.

1. Requirements

Functional (what the system does)

• Users upload files (images, documents, videos) from a web app.
• Show progress, allow cancel, and let large uploads resume after a failure.
• Files become available (download, thumbnails, previews) after processing.

Non-functional (how well it must work)

• Handle large files (hundreds of MB to several GB) and many uploads at once without wearing out the API servers.
• Security: check type and size, block malware, and stop files from attacking other users.
• Reliability: weak mobile networks and retries must work, and half-finished uploads must not stay forever.
• Cost: use as little bandwidth and compute on your own servers as you can.

2. Option A: upload through your API (multipart/form-data)

Browser ──(entire file)──▶ API server ──(entire file)──▶ Object storage

// Express + multer: simple, and fine for small filesapp.post("/upload", upload.single("file"), async (req, res) => { /* save req.file to S3 */ });

multipart/form-data is the standard way a browser form sends a file in an HTTP request: the body is cut into labelled parts (one per field), and one part holds the file bytes. A reverse proxy (such as nginx) is a server that sits in front of your app and passes requests to it. A load balancer is a server that shares incoming requests between several copies of your app.

Pros: it is simple, you can look at the bytes as they pass, and there is one endpoint.

Cons (why it breaks at scale):

• Your server receives every byte and sends it again. Your bandwidth use doubles, and the CPU and memory cost grows with the number of uploads in progress.
• A few large uploads can hold connections and workers for minutes, so normal traffic has to wait (Backend, Lesson 10).
• There are timeouts and limits at every step:
  • the load balancer's idle or request timeout
  • the reverse proxy setting client_max_body_size
  • API Gateway payload limits (about 10 MB)
  • serverless function limits (about 6 MB for synchronous Lambda calls)
  • Node memory, if you buffer the file
• If it fails at 95%, you start again from zero.

This is fine for small files (avatars, a few MB) when you want the simplest thing. Do not use it for large files.

3. Option B: presigned URL, direct-to-storage (the standard answer)

1. Browser ─ "I want to upload report.pdf, 48 MB, application/pdf" ──▶ API2. API: authenticate, authorise, validate declared name/type/size, create a DB row (status "pending"),        generate a presigned URL for a unique key ──▶ Browser3. Browser ──────────────── PUT bytes directly ───────────────▶ Object storage (S3)4. Storage ── "ObjectCreated" event ──▶ queue/Lambda/worker: verify, scan, process ──▶ DB row "ready"5. Browser polls (or is notified): file is ready

A presigned URL is a web address that already contains a time-limited signature, so storage accepts one exact request without asking for a login. You make the signature on your server with your own secret credentials. Whoever holds the URL can do only that one action (for example, PUT one object key) until the URL expires. Think of it as a ticket for one single entry: the holder does not need a key to the building.

PUT is the HTTP method that says "store this body at this address". A signature is a value computed from the request details and a secret, so storage can check that the URL was not changed.

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";import { getSignedUrl } from "@aws-sdk/s3-request-presigner";import { randomUUID } from "node:crypto";const s3 = new S3Client({ region: "ap-south-1" });const ALLOWED = new Map([["image/jpeg", "jpg"], ["image/png", "png"], ["application/pdf", "pdf"]]);const MAX_BYTES = 100 * 1024 * 1024;app.post("/v1/uploads", requireAuth, async (req, res) => {  const { filename, contentType, size } = uploadRequestSchema.parse(req.body);       // validate (Backend, Lesson 03)  if (!ALLOWED.has(contentType)) throw new ValidationError("File type not allowed");  if (size < 1 || size > MAX_BYTES) throw new ValidationError("File too large");  const key = `uploads/${req.user.id}/${randomUUID()}.${ALLOWED.get(contentType)}`;   // never trust the client's filename as the key  const file = await prisma.file.create({    data: { id: randomUUID(), userId: req.user.id, key, originalName: filename, contentType, size, status: "PENDING" },  });  const url = await getSignedUrl(    s3,    new PutObjectCommand({ Bucket: process.env.BUCKET!, Key: key, ContentType: contentType, ContentLength: size }),    { expiresIn: 300 },                                                               // valid for 5 minutes  );  res.status(201).json({ fileId: file.id, uploadUrl: url, method: "PUT", headers: { "Content-Type": contentType } });});

Key points:

• The signature is tied to the key, the method and (when you include them) the content type and length. So the client cannot upload a different type or write somewhere else.
• Use a short expiry (minutes). Use a unique, server-made key, never the user's file name. A user's file name can cause path traversal (using ../ to reach other folders), name clashes, and leaks of private names.
• For stricter limits, use a presigned POST. This is a presigned form upload that comes with a policy (a set of rules signed together with the URL). The rules can include content-length-range (a minimum and maximum size), a Content-Type prefix and a key prefix. S3 enforces them during the upload.
• The bucket (the top-level container that holds your objects) is private. The browser needs CORS (a setting that tells the browser which websites may call the bucket) on the bucket for your app's origin (PUT, and the headers you send) (Browser Contract, Lesson 04).
• Integrity (the file arrives unchanged): send a checksum (a short fingerprint worked out from the bytes, for example in the x-amz-checksum-sha256 header). Storage compares it with its own result and rejects damaged uploads.

Client with progress and cancel

The fetch function has no upload-progress event. XMLHttpRequest (the older browser API for HTTP requests) does have one, so it is the usual way to show progress. An AbortSignal is an object that lets you cancel an operation from outside:

function uploadWithProgress(url: string, file: File, onProgress: (pct: number) => void, signal?: AbortSignal) {  return new Promise<void>((resolve, reject) => {    const xhr = new XMLHttpRequest();    xhr.open("PUT", url);    xhr.setRequestHeader("Content-Type", file.type);    xhr.upload.onprogress = (e) => e.lengthComputable && onProgress(Math.round((e.loaded / e.total) * 100));    xhr.onload = () => (xhr.status >= 200 && xhr.status < 300 ? resolve() : reject(new Error(`Upload failed: ${xhr.status}`)));    xhr.onerror = () => reject(new Error("Network error"));    signal?.addEventListener("abort", () => { xhr.abort(); reject(new DOMException("Aborted", "AbortError")); });    xhr.send(file);                                   // streams the File without loading it all into memory  });}

After the PUT works, call POST /v1/uploads/{id}/complete (or rely on the storage event). Then the server checks that the object really exists and that its size matches (a HeadObject call returns an object's size and type without downloading it), and marks the row UPLOADED. Do not trust the declared size alone.

4. Large files: multipart and resumable uploads

One PUT of 2 GB over a mobile network will fail sooner or later. So split the file into parts (chunks).

S3 multipart upload is an S3 feature that lets you upload one object as many separate parts and then join them. Its steps are:

1. Start (CreateMultipartUpload). You get an UploadId.
2. Upload the parts (UploadPart). Each part is between 5 MB and 5 GB (the last part may be smaller than 5 MB). Parts are numbered 1 to 10,000, and one object can be up to 5 TB in total (10,000 parts of 5 GB would be more, so the object limit is the cap). You can upload parts at the same time, and retry one part alone if it fails.
3. Finish (CompleteMultipartUpload) with the list of part numbers and ETags (an ETag is a tag, usually a hash of the part's bytes, that S3 gives back for each part). S3 joins the parts into one object.
4. Abort if the upload is abandoned. Also add a lifecycle rule (an automatic storage rule, here AbortIncompleteMultipartUpload) that aborts unfinished multipart uploads after some days. Unfinished parts cost money for storage, and you cannot see them easily.

With presigned URLs, the API signs each part. The client asks "give me URLs for parts 1 to N", and later it calls complete:

client ─▶ API: start upload (size 2 GB)  → uploadId, partSize 16 MB, N = 128 partsclient ─▶ API: presign parts 1..N        → N URLsclient ─▶ S3: PUT part k (parallel, 3–5 at a time), retry on failure, collect ETagclient ─▶ API: complete (uploadId, [{part, etag}…]) → API calls CompleteMultipartUpload

Resuming after a page refresh or a crash: save the uploadId and the finished part numbers (on the device, and/or ask S3 with ListParts). Then upload only the missing parts. Presigned part URLs also expire, so ask the API for fresh URLs for the parts that are left. Cutting a file is cheap in the browser: file.slice(start, end).

async function uploadParts(file: File, partSize: number, urls: string[], concurrency = 4) {  const etags: { PartNumber: number; ETag: string }[] = [];  let next = 0;  async function worker() {    while (next < urls.length) {      const i = next++;      const chunk = file.slice(i * partSize, (i + 1) * partSize);      for (let attempt = 1; ; attempt++) {        try {          const res = await fetch(urls[i], { method: "PUT", body: chunk });          if (!res.ok) throw new Error(String(res.status));          etags.push({ PartNumber: i + 1, ETag: res.headers.get("ETag")! });          break;        } catch (e) {          if (attempt === 4) throw e;          await new Promise((r) => setTimeout(r, 2 ** attempt * 500 + Math.random() * 250)); // backoff + jitter        }      }    }  }  await Promise.all(Array.from({ length: concurrency }, worker));  return etags.sort((a, b) => a.PartNumber - b.PartNumber);}

Backoff means waiting longer after each failed try. Jitter means adding a small random amount to that wait, so many clients do not retry at the same moment (see Lesson 07). file.slice(start, end) returns a piece of a File without copying all of it into memory.

The bucket's CORS setting must expose the ETag header to the browser, or JavaScript cannot read it. The tus protocol is an open standard (a set of agreed HTTP rules) for resumable uploads. Uppy is a JavaScript upload library. It supports S3 multipart and tus, with progress, retries and a ready-made UI.

5. After the bytes land: asynchronous processing

Asynchronous means in the background, not while the user waits. Never make the user wait for heavy work inside a request. Use storage events instead. An event is a small message that says "something happened" (here: "an object was created"). A queue (such as Amazon SQS) is a waiting line where messages are stored until a worker (a program that does jobs) takes them. A derivative is a new file made from the original, such as a thumbnail. Transcoding means converting a video or audio file to another format or size. A webhook is an HTTP request that your server sends to another URL when something happens. SSE (Server-Sent Events) is a browser feature where the server keeps one HTTP response open and streams messages to the page (Lesson 05). ClamAV is an open-source antivirus scanner.

S3 ObjectCreated ──▶ SQS queue ──▶ worker(s):    1. Fetch metadata; verify size and detected content type (magic bytes), not just the extension    2. Scan for malware (ClamAV or a managed scanning service)    3. Generate derivatives: thumbnails, transcodes, text extraction    4. Move/copy to the final location; set status READY (or REJECTED with a reason)    5. Notify the client (webhook, SSE, push) or let it poll GET /v1/files/{id}

The states are: PENDING (URL issued) → UPLOADED (object is there) → SCANNING → READY or REJECTED. A cleanup job deletes PENDING rows and objects that never finished. Make workers idempotent. Idempotent means that running the same job twice gives the same result as running it once. You need this because queues deliver at least once, so a message can arrive twice (Lesson 07). Use a dead-letter queue (a separate queue where messages go after they fail too many times) for "poison" files that always fail. To poll means to ask the server again and again, "is it ready yet?".

6. Validating files safely

Checks in the browser are only a nice touch for the user. The server is the authority.

Serving user uploads safely

User files can carry XSS (cross-site scripting: an attack where a page runs a script written by the attacker). An uploaded SVG (an image format written as XML text, which can contain scripts) or HTML file served from your main domain can run script with your site's rights.

• Serve uploads from a separate domain (for example usercontent.example.net) or a CDN origin (a CDN is a network of servers around the world that keep copies of files close to users) with no cookies and no shared origin. An origin is the combination of scheme, host and port, such as https://example.com; browsers keep different origins apart. Then script inside a file cannot touch your session (Browser Contract, Lesson 07).
• Send X-Content-Type-Options: nosniff (a header that stops the browser from guessing a file's type), a correct Content-Type, and for downloads Content-Disposition: attachment (a header that tells the browser to save the file instead of showing it).
• For files with access control, prefer private buckets with short-lived signed GET URLs (or CloudFront signed URLs/cookies). Use public CDN URLs only for assets that are really public.
• Add a Content-Security-Policy header with the sandbox directive for risky types. CSP is a header that tells the browser what a page may do; sandbox switches off scripts and other powers.

7. Scenario: uploads time out behind the load balancer

Diagnosis. The upload goes through the API. A slow client that uploads 500 MB takes minutes. Something cuts the connection: the load balancer's idle timeout (the time it waits with no data before it closes the connection; 60 seconds by default on AWS Application Load Balancer), or the request timeout in nginx, API Gateway or the function platform. Or the app server runs out of memory or connections.

Fixes (best first):

1. Move to presigned direct-to-storage uploads. Then the bytes never touch the load balancer or your app.
2. Use multipart/chunked uploads, so each request is small and can be retried.
3. As a quick fix for now: raise the timeouts and body-size limits at each step (proxy, load balancer, app). Stream the request to storage instead of buffering it. Keep connections alive.
4. Move long processing to background workers.

8. Scale and cost

• Bandwidth and compute for incoming files move to the storage provider, which scales out by itself.
• Transfer acceleration or edge endpoints (S3 Transfer Acceleration, which sends the upload to a nearby AWS edge location first, or regional upload endpoints) make uploads faster for users who are far away.
• Lifecycle policies: move cold files (files rarely used) to cheaper storage tiers. Delete temporary and PENDING objects. Abort old multipart uploads.
• Deduplication (not storing the same file twice): work out a content hash (a fingerprint of the file's bytes, such as SHA-256, on the client or from the storage checksum), if it fits your case. Think about privacy first.
• Observability (being able to see what the system does from its logs, numbers and traces): upload success rate, abandoned upload rate, processing time, reasons for rejection, and storage growth.

Common mistakes

1. Streaming all uploads through the API server.
2. Trusting the client's Content-Type, extension or file name.
3. Presigned URLs that live too long or allow too much. The goal is: minutes, one key, one method.
4. Public buckets, and serving user files from the main application origin.
5. No cleanup of pending uploads and unfinished multipart uploads.
6. No malware scan or image re-encode for user content.
7. No integrity check or retry for chunks, and no way to resume.
8. Heavy processing done synchronously in the request that confirms the upload.

Interview questions

Key takeaways

• Control plane vs data plane: the control plane makes decisions and keeps records; the data plane moves the bytes. Your API gives permission (the presigned URL) and tracks state. Object storage handles the bytes.
• Multipart + parallel parts + a retry for each part gives reliability for large files and a way to resume. Abort old uploads with lifecycle rules.
• Validate on the server: allow-list types, check magic bytes, enforce size, scan, re-encode images, and make your own keys.
• Process in the background from storage events, with idempotent workers, and show a clear set of states.
• Serve safely: separate domain, nosniff, attachment, and signed URLs for private files.
• Timeouts behind the load balancer are a sign that you are passing bytes through it. Take your servers out of the data path.

What's next

Lesson 05: Design a Notification System. Uploads push data in. Notifications push data out to users in real time, across channels, without duplicates.