Streams and Backpressure

In lesson 2 we read files with readFile and saw it loads everything into memory. That is fine for a config file and a disaster for a log file, a video or a database export. Streams are Node's answer, and they sit under fs, http, zlib, crypto and child processes. In this lesson we first feel the problem with real memory numbers, then learn the four kinds of streams, then measure backpressure with and without respect for it, then use pipeline, the safe way to connect streams. The next lesson, An HTTP Server From Scratch, uses streams for the request and response.

Words used in this lesson

• Stream is an object that sends or receives data in pieces over time.
• Chunk is one piece of data in a stream. For files it is a Buffer; for text streams with an encoding it is a string; in object mode it is any JavaScript value.
• Readable is a source of data you read from (a file, an HTTP request body).
• Writable is a destination you write to (a file, an HTTP response, a socket).
• Duplex is both readable and writable, like a network socket.
• Transform is a duplex stream whose output is a changed version of its input, like gzip or an upper-case filter.
• Pipe means connecting the output of one stream to the input of the next.
• Backpressure is the signal from a slow reader or writer to a fast one that says "wait, I am full".
• highWaterMark is the size at which a stream's internal buffer counts as full. It is a hint, not a hard limit.
• Buffering means holding chunks in memory while waiting for the other side.
• drain event is what a writable emits when its buffer has emptied and it is ready for more.
• Object mode means a stream carries any JavaScript value instead of bytes or strings.
• RSS (resident set size) is the amount of RAM the operating system has given your process right now.

The problem: reading everything at once

The task is simple: count the lines in the CSV that end with ,2026-01-15. I wrote the same task two ways. The first reads the whole file as a string and splits it:

import { readFile } from "node:fs/promises";const text = await readFile(process.argv[2], "utf8");let count = 0;for (const line of text.split("\n")) { if (line.endsWith(",2026-01-15")) count++; }

The second reads a stream, line by line, with the built-in readline module:

import { createReadStream } from "node:fs";import { createInterface } from "node:readline";const rl = createInterface({ input: createReadStream(process.argv[2]), crlfDelay: Infinity });let count = 0;for await (const line of rl) { if (line.endsWith(",2026-01-15")) count++; }

Both scripts sampled the process memory in the background and printed the peak. I ran each twice:

readFile + split: 357143 matches, 8041 ms, peak rss 881 MBstream + readline: 357143 matches, 4052 ms, peak rss 82 MBreadFile + split: 357143 matches, 6943 ms, peak rss 853 MBstream + readline: 357143 matches, 3874 ms, peak rss 82 MB

Same answer: 357,143 matches. But the whole-file version needed about 850 to 880 MB of memory for a 302 MB file, and the stream version needed 82 MB, and that is nearly the same as an empty Node process plus some buffers (an empty process was about 42 MB on this machine). The stream version was also faster here, because splitting a 300 MB string into ten million line strings is heavy work. Do not take the speed ratio as a law; take the memory result seriously.

It gets worse. A server usually has a memory limit. I ran the whole-file script with the heap limited to 64 MB (node --max-old-space-size=64):

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

It crashed. The stream version, with the same 64 MB limit, finished normally: stream + readline: 357143 matches, 3673 ms, peak rss 83 MB.

If you have 100 users each downloading a file at the same time and you read each file whole, you need 100 times the file size in RAM. With streams you need roughly 100 times a few chunks. That is the reason streams exist.

The four kinds of streams

A good picture is a river of water pipes. A Readable is a tap. A Writable is a sink. A Transform is a filter in the middle of the pipe. Water flows, but only as fast as the narrowest pipe can take it.

Reading: a stream gives you chunks

The easiest way to read a stream today is for await:

import { createReadStream } from "node:fs";let bytes = 0, chunks = 0, peak = 0;const s = createReadStream("/tmp/lab-nodejs/big.csv");const timer = setInterval(() => { peak = Math.max(peak, process.memoryUsage().rss); }, 5);const t0 = performance.now();for await (const chunk of s) { bytes += chunk.length; chunks++; }clearInterval(timer);console.log(`chunks: ${chunks}, first chunk size ${s.readableHighWaterMark} bytes (highWaterMark), total ${(bytes/1048576).toFixed(0)} MB, ${(performance.now()-t0).toFixed(0)} ms, peak rss ${(peak/1048576).toFixed(0)} MB`);

chunks: 4833, first chunk size 65536 bytes (highWaterMark), total 302 MB, 639 ms, peak rss 76 MB

The 302 MB file arrived as 4,833 chunks of up to 64 KiB each (65,536 bytes). The Node docs say 64 KiB is the default highWaterMark for streams of bytes in this version, and 16 for object mode streams. I checked in code:

Writable 65536 Readable 65536 objectMode 16 fs read 65536 fs write 65536

Peak memory was 76 MB: the loop touches one chunk at a time. You can pick another chunk size. With highWaterMark: 1024 * 1024 and an end limit of 5 MiB I got exactly 5 chunks of 1 MiB.

There is also the older event style, still common in tutorials and library code:

import { createReadStream } from "node:fs";const r = createReadStream("/etc/hostname", { encoding: "utf8" });r.on("data", (c) => console.log("data event, typeof", typeof c, JSON.stringify(c)));r.on("end", () => console.log("end event"));r.on("close", () => console.log("close event"));

data event, typeof string "vm\n"end eventclose event

data fires for each chunk, end fires when there is no more data, and close fires when the underlying file is released. Adding a data listener switches the stream to "flowing" mode: it starts pushing chunks to you as fast as it can. Note { encoding: "utf8" } turned chunks into strings. That also fixes the split-character problem from lesson 2, because the stream keeps unfinished bytes for the next chunk.

Writing: write() answers a question

A writable has a method write(chunk). Its return value matters more than most beginners think. I made a writable with a tiny highWaterMark of 10 bytes and a slow write (it finishes each chunk in a later turn of the loop):

const w = new Writable({ highWaterMark: 10, write(c, e, cb) { setImmediate(cb); } });console.log("write 4 bytes ->", w.write("1234"),            "| write 4 more ->", w.write("1234"),            "| write 4 more (12 > 10) ->", w.write("1234"));console.log("writableLength", w.writableLength, "needDrain", w.writableNeedDrain);w.on("drain", () => console.log("drain event, length now", w.writableLength));

write 4 bytes -> true | write 4 more -> true | write 4 more (12 > 10) -> falsewritableLength 12 needDrain truedrain event, length now 0

write() returns true while the stream's internal buffer is below highWaterMark. When the buffer goes past it (12 bytes queued, mark is 10), it returns false. That false means: "I have taken your data, but please stop writing until I tell you I am ready". When the buffer is empty again, the stream emits drain. That is the whole backpressure mechanism on the writing side: a false, then a pause, then a drain.

Important: a false from write() does not lose your data. The chunk is accepted and queued. The danger is what happens if you keep writing anyway.

Backpressure, measured

Here is an experiment with a fast producer and a slow consumer. The consumer is a writable that finishes each chunk in a later loop turn (a stand-in for a slow disk or a slow network client). The producer writes 20,000 chunks of 32 KiB each, 625 MiB in total, in a loop. The two versions differ in one thing: whether the loop obeys write().

import { Writable } from "node:stream";import { once } from "node:events";const mode = process.argv[2];           // "ignore" or "respect"const CHUNKS = 20000, SIZE = 32 * 1024;let consumed = 0;const slowSink = new Writable({  write(chunk, enc, cb) { consumed += chunk.length; setImmediate(cb); }});for (let i = 0; i < CHUNKS; i++) {  const ok = slowSink.write(Buffer.alloc(SIZE, 97));  if (!ok && mode === "respect") await once(slowSink, "drain");   // the only difference}

$ node bp.mjs ignoreignore: write() returned false 19999 times, waited for drain 0 timesignore: all chunks handed over after 4547 ms, peak rss so far 685 MB, bytes the sink had finished: 0 MB, buffered inside the writable: 625 MBignore: finished at 4634 ms, peak rss 685 MB$ node bp.mjs respectrespect: write() returned false 10000 times, waited for drain 10000 timesrespect: all chunks handed over after 224 ms, peak rss so far 83 MB, bytes the sink had finished: 625 MB, buffered inside the writable: 0 MBrespect: finished at 225 ms, peak rss 83 MB

This is the whole lesson in two blocks.

• Ignoring false: the loop handed over all 625 MB first. The sink had processed 0 MB by then, because the loop never gave it a turn. Everything, 625 MB, sat in the writable's internal queue. Peak memory: 685 MB. This is the "memory leak" people describe when a stream-based program uses more and more RAM.
• Respecting false: the loop paused 10,000 times and waited for drain. The sink kept up. Peak memory: 83 MB, only a little more than an empty process, and the sink finished all 625 MB.

(Why did the ignoring version take 4.5 seconds and the respecting one only 0.2 seconds? My guess is that allocating and holding 625 MB of buffers costs time, but I did not profile it, and I would not claim this speed gap holds on every machine. The memory gap is the reliable lesson.)

One more detail from the output: in the ignoring run write() returned false 19,999 times, that is for every chunk except the first, because the queue was past the mark from then on. In the respecting run it returned false 10,000 times and the loop waited for drain every time. I did not dig into why the second count is exactly half.

Now the same thing with a real reader. Here the source is a file stream and the destination a slow writable that takes 2 ms per chunk. First with pipeline (which respects backpressure for you):

const src = createReadStream("/tmp/lab-nodejs/big.csv", { end: 50 * 1024 * 1024 - 1 });   // first 50 MiBconst slow = new Writable({  write(chunk, enc, cb) { /* record src.readableLength and slow.writableLength */ setTimeout(cb, 2); },});await pipeline(src, slow);

wrote 50 MB in 800 chunks, 1889 msmost bytes ever waiting inside the reader: 0, inside the writer: 65536peak rss 66 MB

The file reader paused whenever the writer was busy. At most 64 KiB sat inside the writer. The reading side held 0 bytes at the sampled moments. Memory stayed at 66 MB, and the speed was set by the slow writer (about 800 chunks at 2 ms each, 1.9 s). The flow of the whole pipeline matched its slowest member.

And now the broken version, where I read with a data event and call write without checking:

src.on("data", (chunk) => { slow.write(chunk); });   // ignores the return value, never pausessrc.on("end", () => console.log(`reader finished while the writer still had ${(slow.writableLength / 1048576).toFixed(0)} MB queued`));

reader finished while the writer still had 43 MB queued; rss 99 MB

The reader finished its 50 MB long before the writer; 43 MB were still queued in memory, and resident memory had already risen to 99 MB. Scale this to a 3 GB file and a slow client and you get an out-of-memory crash.

Rule: a data handler that writes somewhere must stop when write() returns false, or you should let pipeline do it for you. The easiest correct design is to not write this code yourself at all.

pipe versus pipeline

readable.pipe(writable) is the classic way to connect streams. It does handle backpressure, but it has a problem with errors. Here is a test: the destination fails on the first chunk, and I look at the source afterwards.

// pipe(): when the destination fails, the source is NOT cleaned upconst src1 = createReadStream(FILE);const bad1 = new Writable({ write(c, e, cb) { cb(new Error("disk full")); } });bad1.on("error", (e) => console.log("pipe(): dest error:", e.message));src1.pipe(bad1);await sleep(300);console.log("pipe(): source destroyed?", src1.destroyed, "| closed?", src1.closed);// pipeline(): everything is destroyedconst src2 = createReadStream(FILE);const bad2 = new Writable({ write(c, e, cb) { cb(new Error("disk full")); } });try { await pipeline(src2, bad2); } catch (e) { console.log("pipeline(): rejected with:", e.message); }await sleep(100);console.log("pipeline(): source destroyed?", src2.destroyed, "| closed?", src2.closed);

pipe(): dest error: disk fullpipe(): source destroyed? false | closed? falsepipeline(): rejected with: disk fullpipeline(): source destroyed? true | closed? true

With pipe, the destination errored, but the source file stayed open (destroyed: false). In a server that serves many requests, each failed pipe leaves a file descriptor open. After a while you hit EMFILE, too many open files. With pipeline, the error went to one place (a rejected promise), and every stream in the chain was destroyed and closed.

Use pipeline from node:stream/promises. It does three jobs: it respects backpressure, it forwards errors to one handler, and it cleans up all streams when anything fails or finishes. The stream.finished() helper is useful too, but the Node docs warn that it leaves some event listeners hanging after the callback, so use pipeline where you can.

You can cancel a pipeline with an AbortSignal. I started a pipeline over the 302 MB file with a slow writer and aborted after 50 ms:

AbortError | bytes handled before abort: 2490368 | source destroyed: true

It stopped after about 2.4 MB and the source was destroyed. This is useful to stop work when a client disconnects.

Compressing a file with a pipeline

Here is the typical real use: read, transform, write.

import { createReadStream, createWriteStream } from "node:fs";import { pipeline } from "node:stream/promises";import { createGzip } from "node:zlib";await pipeline(  createReadStream("/tmp/lab-nodejs/big.csv"),  createGzip(),  createWriteStream("/tmp/lab-nodejs/big.csv.gz"),);

gzip done in 7026 ms; peak rss 73 MBoriginal 302 MB -> gz 53 MB

A 302 MB file was compressed to 53 MB with a peak memory of 73 MB. The compression work runs off the main JavaScript thread, so the event loop stays free while it works. I did not measure loop lag during this run, so take that as the documented design, not as my measurement. Reading it back is the mirror image with createGunzip().

Writing your own Transform

A Transform takes input chunks, does something, and pushes output chunks. Three ways to write one. The class way:

import { Readable, Writable, Transform } from "node:stream";import { pipeline } from "node:stream/promises";const source = Readable.from(["apple\n", "banana\n", "cherry\n"]);const upper = new Transform({  transform(chunk, enc, cb) { cb(null, chunk.toString().toUpperCase()); }});const out = [];const sink = new Writable({ write(chunk, enc, cb) { out.push(chunk.toString()); cb(); } });await pipeline(source, upper, sink);console.log(out);

[ 'APPLE\n', 'BANANA\n', 'CHERRY\n' ]

Readable.from(iterable) turns an array or generator into a readable stream. In transform, you call cb(error, output) when done with the chunk.

The modern way is an async generator inside pipeline. It is much shorter and respects backpressure automatically (the generator is only asked for the next value when downstream is ready):

async function* numbered(src) { let n = 1; for await (const chunk of src) yield `${n++}: ${chunk}`; }const out2 = [];await pipeline(Readable.from(["a\n", "b\n"]), numbered, async function (src) { for await (const c of src) out2.push(c); });console.log(out2);

[ '1: a\n', '2: b\n' ]

Object mode streams carry JavaScript objects instead of bytes. Set objectMode: true on the sides that need it:

const objs = Readable.from([{ id: 1 }, { id: 2 }, { id: 3 }]);const doubled = new Transform({ objectMode: true, transform(o, e, cb) { cb(null, { ...o, twice: o.id * 2 }); } });const got = [];await pipeline(objs, doubled, new Writable({ objectMode: true, write(o, e, cb) { got.push(o); cb(); } }));console.log(got);

[ { id: 1, twice: 2 }, { id: 2, twice: 4 }, { id: 3, twice: 6 } ]

In object mode the highWaterMark counts objects (16 by default, as we saw), not bytes.

The classic bug: chunks are not lines

A chunk boundary can fall in the middle of a line. If you split each chunk on \n and count, you get the wrong answer. I tested it on the 10,000,000-line file:

// WRONG: treat each chunk as a list of whole lineslet naive = 0;for await (const chunk of createReadStream(FILE, { encoding: "utf8" })) naive += chunk.split("\n").filter(Boolean).length;// RIGHT: keep the unfinished last piece for the next chunkclass LineSplitter extends Transform {  constructor() { super({ readableObjectMode: true }); this.rest = ""; }  _transform(chunk, enc, cb) {    const parts = (this.rest + chunk.toString()).split("\n");    this.rest = parts.pop();                 // the last piece may be unfinished    for (const p of parts) this.push(p);    cb();  }  _flush(cb) { if (this.rest) this.push(this.rest); cb(); }}

true line count (wc -l): 10000000naive chunk.split count : 10004536LineSplitter count      : 10000000

The naive count was off by 4,536. The file has 4,832 chunk boundaries, and almost every one fell in the middle of a line. A cut line is counted as two pieces, so each cut adds one. The LineSplitter, which saves the unfinished tail and finishes it with the next chunk, got exactly 10,000,000. In real code, use readline (as in the first test) or a tested library for lines. The lesson is the idea: a chunk is a piece of the stream, not a unit of your data. The same is true for JSON: you cannot JSON.parse a chunk.

Streams in HTTP

An HTTP server's request is a Readable and its response is a Writable. So you can stream a big file to a client without loading it:

http.createServer(async (req, res) => {  if (req.url === "/whole") { const data = await readFile(FILE); res.setHeader("content-length", data.length); res.end(data); }  else if (req.url === "/stream") { res.setHeader("content-type", "text/csv"); await pipeline(createReadStream(FILE), res).catch(() => {}); }  // ...}).listen(3102);

I started a fresh server for each route, downloaded the 302 MB file with curl, then asked the server for the peak memory it had seen (the base before any request was 54 MB):

== whole (fresh server)peak rss so far: 54 MBdownloaded 316686660 bytes in 0.886968speak rss so far: 358 MB== stream (fresh server)peak rss so far: 54 MBdownloaded 316686660 bytes in 0.788890speak rss so far: 97 MB

Both sent all 316,686,660 bytes in about the same time. The whole-file route grew the server by 304 MB (one copy of the file) per request in flight. The streaming route grew by 43 MB. With ten simultaneous downloads the first would need over 3 GB.

When the client disconnects midway, pipeline is built to destroy the file stream, as in the failing-writable test above, so no file handle is left open. I did not run a disconnect test against this server. That is why I used pipeline and not pipe there. The .catch(() => {}) is only there because I do not want an unhandled rejection for a client that left. In a real server, you would log the error.

Also notice that the streaming response has no content-length header: Node sends it with Transfer-Encoding: chunked. The headers I got from curl -D - on the /stream route:

HTTP/1.1 200 OKcontent-type: text/csvDate: Sun, 11 Oct 2026 18:51:42 GMTConnection: keep-aliveKeep-Alive: timeout=5Transfer-Encoding: chunked

No Content-Length, because the size is not known when the headers go out. The client reads chunk by chunk until the server says it is finished. We saw this in action in lesson 4, where I read a response with three parts spaced 200 ms apart and got them as they arrived.

When not to use streams

Streams add code and complexity. Do not use them everywhere.

• Small data. A 2 KB config file: readFile is simpler and just as good.
• Formats that need all the data. JSON.parse needs the full text. For big JSON, switch the format to one JSON object per line (NDJSON) and read it with the line approach, or use a streaming parser from npm.
• Random access. If you need the middle of a file, use FileHandle.read with a position (lesson 2), not a full stream.

And remember the size of the problem: if the whole data fits easily in memory, and there are few users, the simple code is the correct choice. If the data is unbounded or user-controlled (uploads, exports, logs), use streams.

Web streams, in one paragraph

Browsers have their own stream API (ReadableStream, WritableStream) and Node has it too. fetch() returns a body that is a web ReadableStream; in lesson 4 I read it with for await (const chunk of r.body) and it worked. The Node docs list Readable.fromWeb, Readable.toWeb and similar helpers to convert between the two kinds. The ideas (chunks, backpressure) are the same. This series uses Node streams, because that is what fs, http and zlib give you.

Try it

1. Generate a file with 5 million lines (use a writable and respect drain). Then count the lines with readline and with readFile, and compare peak memory using process.memoryUsage().rss.
2. Write a pipeline that reads a text file, upper-cases it with a Transform, gzips it and writes out.txt.gz. Then write a second pipeline that reads it back and prints the first 100 characters.
3. Create a Writable with highWaterMark: 100 that takes 10 ms per chunk. Write 1000 small strings in a loop with and without waiting for drain. Print writableLength at the end of the loop in both cases.
4. Replace pipeline with pipe in the "destination fails" experiment and add src.on("error"). What do you still not get?
5. Write an async generator transform that removes empty lines from a stream of lines and use it in a pipeline.
6. Add an AbortController to a long pipeline and abort it with Ctrl+C (process.on("SIGINT", ...)). Check that the output file is closed.

Answers

1. In my 10-million-line test, readFile gave about 850 MB peak and readline about 82 MB. Your numbers differ, but the 10x ratio should look similar.
2. Use createReadStream, an async generator async function* (src) { for await (const c of src) yield c.toString().toUpperCase(); }, createGzip() and createWriteStream("out.txt.gz"). To read back, createReadStream, createGunzip(), then in the last step print chunk.toString().slice(0, 100) for the first chunk only and stop.
3. Without waiting you will see a large writableLength (all strings, close to the total size you wrote). With await once(w, "drain") after each false, it stays near 100.
4. Even with an error listener on the source, pipe does not destroy the other streams when one fails, so the source stays open (as in the output above). Only pipeline (or manual destroy calls) cleans up.
5. async function* dropEmpty(src) { for await (const line of src) if (line.trim() !== "") yield line; }. Place it between a line source and the destination.
6. Calling ac.abort() in the signal handler makes pipeline reject with AbortError and destroy all streams. The output file is closed but contains whatever was written before the abort, so delete or rename a partial file in a catch.

Interview questions

Key takeaways

• Streams process data in chunks, so memory use depends on the chunk size, not the data size: 82 MB versus about 850 MB for a 302 MB file, and a crash under a 64 MB heap limit for the non-stream version.
• write() returning false means "wait for drain". Ignoring it makes the data pile up: 625 MB queued and a 685 MB peak in my test, against 83 MB when respected.
• Use pipeline from node:stream/promises. It handles backpressure, forwards errors, destroys all streams on failure and supports AbortSignal. pipe() left the source open when the destination failed.
• Chunks are not lines and not JSON values. Use readline, a LineSplitter, or an encoding on the stream. The naive version miscounted by 4,536 lines.
• Async generators are the shortest way to write a transform inside pipeline.
• In HTTP, stream big responses: about 43 MB extra versus about 304 MB extra for one 302 MB download.
• Do not use streams for tiny data or formats that need all of the data at once.

What's next

Next: An HTTP Server From Scratch, Lesson 4. You will build a small JSON API with only node:http: routing by hand, reading JSON bodies with a size limit, status codes, curl output for every case, then keep-alive connections and the timeouts that protect a server. Practise now: pick the largest file you have and try the readline line count on it. Watch memory with ps or Activity Monitor while it runs.