Processes, Threads & Concurrency for Backend Engineers

Process vs thread

Context switching. A CPU core runs one thing at a time. The operating system (OS) quickly switches between threads, so it looks like they all run together. This switch is called a context switch. Each switch costs time, because the OS must save one thread's state and load another's. With a few threads, this cost is tiny. With 10,000 threads, the CPU can spend a big share of its time just switching.

CPU-bound vs I/O-bound work

• CPU-bound work is work where the processor is the slow part, because it is busy all the time: resizing images, encrypting data, video encoding, machine learning. More CPU cores help.
• I/O-bound work is work where most of the time is spent waiting for input or output (I/O): database queries, calls to other APIs, reading files. Handling more waiting requests at once helps. More CPU does not.

Most web backends are I/O-bound. This is why many modern tools (Node.js, NGINX, Go, async Python) are built to wait efficiently.

Blocking vs non-blocking I/O

• Blocking I/O. The thread calls db.query() and stops until the answer comes back. It is simple to write. To serve 1,000 waiting requests at once, you need about 1,000 threads.
• Non-blocking I/O. The thread says "start this query and tell me when it is done", then moves on to other work. The operating system tells it when the data is ready, using tools like epoll on Linux or kqueue on macOS. One thread can manage thousands of connections. An event loop is a loop that waits for such "ready" events and runs a small piece of code for each one.

Around 1999, engineers asked, "How can one server handle 10,000 connections at once?" This became known as the C10K problem (C10K means 10 thousand connections). Non-blocking I/O and event loops were the answer.

How popular servers handle concurrency

1. Process per request (old CGI, early PHP setups). Each request starts a new process. It is safe and simple, but very heavy. It is rarely used this way today.

2. Thread per request with a thread pool (Java Spring/Tomcat, older Ruby and Python servers). A thread pool is a fixed group of ready threads that are reused. With a pool of, say, 200 threads, each request borrows a thread until it finishes.

• Easy to write and debug, since the code reads top to bottom.
• If all 200 threads are waiting on a slow database, request 201 must wait in a queue.

3. Event loop (Node.js, NGINX, Redis). A single thread runs a loop: take the next ready event, run a short piece of code, repeat. Slow work is started and the result comes back later. It comes back as a callback (a function that is called when the work is done), a promise (an object that stands for a future result) or an await (a keyword that waits for a promise without blocking the loop).

• Excellent for I/O-heavy work with many connections.
• The golden rule: never block the event loop. One heavy calculation in Node.js stops every user's request until it finishes.

// ❌ Blocks every user for secondsapp.get("/report", (req, res) => {  const result = heavyCalculation();   // CPU work on the event loop  res.json(result);});// ✅ Waiting is fine – the loop serves other users meanwhileapp.get("/user/:id", async (req, res) => {  const user = await db.findUser(req.params.id);  // I/O, non-blocking  res.json(user);});

4. Lightweight threads (Go goroutines, Java 21 virtual threads, Erlang/Elixir processes). You write simple blocking-style code, but the runtime (the system that runs your program) uses only a few real OS threads and switches cheaply between very lightweight tasks. A goroutine (Go's lightweight task) starts with only about 2 KB of memory, so one server can run hundreds of thousands of them.

• You get both: easy code and high concurrency.

5. Async/await (Python asyncio, Rust Tokio, C#). This works like the event loop, but the code is easier to read. You mark each waiting point with await, and the function pauses there without blocking other work.

Multiple processes for multiple cores. A single-threaded event loop uses only one CPU core (a core is one processing unit inside the CPU). So in production you run several copies:

• Node.js runs one process per core (for example, with PM2 or cluster mode).
• NGINX runs one worker process per core.
• Python web apps run several Gunicorn or Uvicorn workers.

Race conditions: when threads collide

When threads share memory, you cannot predict the order in which they run. A race condition is a bug where the result depends on which thread happens to run first. Here is the classic example:

Each thread's code was "correct", but together they give a wrong result. This is a race condition. The same bug appears as:

• two people booking the last seat,
• overselling the last item in stock,
• a counter losing updates.

Fixes:

• Locks (mutexes). A lock is a "key" that only one thread can hold at a time. Only the thread with the key may enter the "critical section" (the code that touches the shared data). This is correct but reduces parallelism.
• Atomic operations. An atomic operation is one that cannot be cut in the middle by another thread. Do read-check-write as one unbreakable step. Examples: counter.incrementAndGet(), or Redis INCR.
• Let the database do it. For example, UPDATE accounts SET balance = balance - 80 WHERE id = 1 AND balance >= 80. The database runs this as one safe step, and you check how many rows changed. If zero rows changed, the balance was too low.
• Avoid shared state. Give each request its own data, and let parts of the system talk through messages or queues.

Important: race conditions do not only happen inside one server. Say you have 10 servers behind a load balancer (a server that shares requests between them). In-memory locks in your code no longer help, because each server has its own memory. You need protection in the database, or a distributed lock (a lock that all servers share). This is a big theme in later lessons.

Deadlocks

A deadlock happens when two threads each hold something the other needs, and both wait forever:

Thread A: holds Lock 1, waits for Lock 2Thread B: holds Lock 2, waits for Lock 1→ both stuck forever

The simplest prevention is to always take locks in the same order. Adding timeouts to lock waits also helps. Difference in one line: in a race condition the result is wrong, in a deadlock nothing moves at all.

Pools: bounded is better than unlimited

Servers use pools. A pool is a fixed set of ready-to-use resources that are borrowed and given back: thread pools, database connection pools, worker pools. Why not just create a new thread or connection whenever you need one?

• Creating them is slow.
• If creation has no limit, a traffic spike makes you create thousands. You run out of memory and crash, and every user is down.

A bounded pool (one with a maximum size) with a queue works better under heavy load. Some requests wait or are rejected, but the server survives.

Pool size is a real design decision. A useful rule is Little's Law (more in a later lesson): the average number in use = arrival rate × time each one is held. If you get 200 requests per second and each holds a database connection for 50 ms, then on average 200 × 0.05 = 10 connections are in use.

A note on Python's GIL

Standard Python (CPython, the usual Python program) has a Global Interpreter Lock (GIL). The GIL is a lock that lets only one thread run Python code at a time. So:

• Threads help Python with I/O-bound work, because the lock is released while waiting.
• Threads do not help much with CPU-bound work.

That is why Python apps use multiple processes (Gunicorn workers, multiprocessing) or async to scale. Newer Python versions (3.13 and later) offer an optional "free-threaded" build without the GIL, but the GIL is still the default and most production code still assumes it.