TCP and UDP: Connections, Reliability and Congestion

Every time you call fetch, open a database connection or load a web page, TCP is under it. Most developers only meet TCP when something goes wrong: "connection refused", "connection timed out", "connection reset by peer", a request that hangs for two minutes, or a server with thousands of sockets stuck in a strange state. Each of those messages has a precise meaning, and once you have seen the packets behind them, they stop being scary.

This is lesson 2 of a nine-lesson series. In lesson 1 we saw how an IP packet finds its way across networks. IP is "best effort": a packet can be lost, copied, delayed or arrive out of order, and IP does not care. This lesson shows how TCP builds something dependable on top of that unreliable layer, and how UDP chooses not to. The next lesson, DNS, is a good example of a service that mostly uses UDP.

Words used in this lesson

• TCP (Transmission Control Protocol) is the transport protocol that gives you a reliable, ordered byte stream between two programs.
• UDP (User Datagram Protocol) is the simple transport protocol that sends independent messages with no connection and no guarantees.
• A segment is one TCP packet's worth of data. A datagram is one UDP message.
• A port is a number (0 to 65535) that tells the computer which program gets the data. A socket is one end of a conversation: an IP address plus a port.
• The three-way handshake is the SYN, SYN-ACK, ACK exchange that opens a TCP connection.
• SYN, ACK, FIN, RST are flags in the TCP header. SYN starts a connection, ACK confirms data, FIN says "I am finished sending", RST says "this connection is dead, stop".
• A sequence number counts bytes. Each side numbers the bytes it sends, so the other side can put them in order and notice gaps.
• An acknowledgement (ACK) says "I have received everything up to this byte number".
• Retransmission means sending the same data again because no ACK came back.
• RTT (round trip time) is how long a packet takes to go there and back.
• RTO (retransmission timeout) is how long the sender waits for an ACK before it sends again.
• The receive window is how many more bytes the receiver says it can accept right now. This is flow control.
• The congestion window (cwnd) is how many bytes the sender allows itself to have in flight. This is congestion control.
• Slow start is the early phase where the window grows fast (it doubles each round trip). Congestion avoidance is the later phase of slow, careful growth.
• TIME_WAIT is a state the side that closes first stays in for a while after the connection ends.
• The listen backlog is the queue of connections that finished the handshake and wait for the program to call accept.

Two ways to use the network

Both protocols sit on top of IP and use port numbers. They make very different promises.

A good way to remember it: TCP is a phone call (you dial, the other person answers, you talk in order, and if the line goes bad the phone company tries to fix it). UDP is a postcard (you write the address and drop it in the box, and you never hear back).

The three-way handshake, captured

Before any data moves, TCP opens a connection. Let me do the smallest possible example. A server that waits for a connection, reads a few bytes, and answers pong:

# echo_server.pyimport sockets = socket.socket(); s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)s.bind(("127.0.0.1", 9001)); s.listen()conn, addr = s.accept()data = conn.recv(100)print("server got", data, "from", addr, flush=True)conn.sendall(b"pong")conn.close()

# client1.pyimport socketc = socket.create_connection(("127.0.0.1", 9001))print("client local", c.getsockname(), "remote", c.getpeername())c.sendall(b"ping"); print("client got", c.recv(100)); c.close()

I started tcpdump first, so it would see everything. The -nn flag stops it from turning numbers into names, and -S prints the real (absolute) sequence numbers instead of small relative ones:

tcpdump -i lo -nn -S -w h.pcap 'tcp port 9001'     # in one terminalpython3 echo_server.py &  python3 client1.py        # in anothertcpdump -nn -S -r h.pcap                            # read the capture back

client local ('127.0.0.1', 46104) remote ('127.0.0.1', 9001)server got b'ping' from ('127.0.0.1', 46104)client got b'pong'1  127.0.0.1.46104 > 127.0.0.1.9001: Flags [S],  seq 3785438839, win 65495, options [mss 65495,sackOK,TS val 3096116337 ecr 0,nop,wscale 10], length 02  127.0.0.1.9001 > 127.0.0.1.46104: Flags [S.], seq 882628887, ack 3785438840, win 65483, options [mss 65495,sackOK,TS val 2558415559 ecr 3096116337,nop,wscale 10], length 03  127.0.0.1.46104 > 127.0.0.1.9001: Flags [.],  ack 882628888, win 64, options [nop,nop,TS val 3096116338 ecr 2558415559], length 04  127.0.0.1.46104 > 127.0.0.1.9001: Flags [P.], seq 3785438840:3785438844, ack 882628888, win 64, length 45  127.0.0.1.9001 > 127.0.0.1.46104: Flags [.],  ack 3785438844, win 64, length 06  127.0.0.1.9001 > 127.0.0.1.46104: Flags [P.], seq 882628888:882628892, ack 3785438844, win 64, length 47  127.0.0.1.46104 > 127.0.0.1.9001: Flags [.],  ack 882628892, win 64, length 08  127.0.0.1.9001 > 127.0.0.1.46104: Flags [F.], seq 882628892, ack 3785438844, win 64, length 09  127.0.0.1.46104 > 127.0.0.1.9001: Flags [F.], seq 3785438844, ack 882628893, win 64, length 010 127.0.0.1.9001 > 127.0.0.1.46104: Flags [.],  ack 3785438845, win 64, length 0

(I added the line numbers and removed the timestamp column and some repeated options so it fits.) tcpdump writes a dot for ACK, S for SYN, P for PUSH (the sender says "pass this to the program now") and F for FIN. Read it line by line.

Lines 1 to 3 are the three-way handshake.

1. SYN. The client says: "I want to talk. My first byte number is 3785438839." That starting number is called the initial sequence number (ISN). It is random on purpose, so old packets from an earlier connection are not mistaken for new ones. The client also lists options: its MSS (the biggest segment it will take), that it understands selective acknowledgements (sackOK), timestamps (TS), and a window scale (wscale 10).
2. SYN-ACK. The server says: "OK. My first byte number is 882628887, and I have got everything up to yours: ack 3785438840." Notice the ack is the client's number plus one. A SYN uses up one sequence number, even though it carries no data.
3. ACK. The client says: "Got it, ack 882628888." The connection is now open. Each side has proved it can send and receive, and each side knows the other's starting number.

The cost of this is one round trip before the first byte of your data can be sent (well, the client can send data right with the third packet, but the server cannot answer before it has seen the client's SYN, so a request and response need at least two round trips from a cold start). On the same machine this is microseconds. Between two continents it is 150 ms or more, which is why reusing connections matters so much (see keep-alive in lesson 4).

Lines 4 to 7 are the data. The client sends 4 bytes: seq 3785438840:3785438844. That range means "bytes numbered 3785438840 up to, but not including, 3785438844". The server answers ack 3785438844, meaning "I have everything before byte 3785438844". The server then sends its own 4 bytes (pong) and the client acknowledges them. Every byte has a number and every byte gets confirmed. This is the whole trick of TCP.

The win value is the receive window, which we will use in a moment. tcpdump prints the raw field. The wscale 10 option from the handshake says "multiply this number by 2^10". So win 64 really means 64 x 1024 = 65,536 bytes. Without scaling, a window could never be bigger than 65,535 bytes, which is far too small for fast, long links.

You can see the same numbers in plain relative form if you leave out -S. Then the first line shows seq 3785438839 and later lines count from 1 (ack 1), which is easier on the eyes:

IP 127.0.0.1.46104 > 127.0.0.1.9001: Flags [S], seq 3785438839, win 65495, options [mss 65495,...], length 0IP 127.0.0.1.9001 > 127.0.0.1.46104: Flags [S.], seq 882628887, ack 3785438840, win 65483, ...IP 127.0.0.1.46104 > 127.0.0.1.9001: Flags [.], ack 1, win 64, ...

Closing a connection, and TIME_WAIT

Lines 8 to 10 close the connection. The server called conn.close(), so it sent a FIN ("I have no more data to send"), the client sent its own FIN, and each FIN was acknowledged. Closing takes four messages in general because each direction is closed separately. (Here, line 9 combines the client's ACK and FIN, and the server's FIN was already in flight.)

Now the part that surprises people. The tool ss lists sockets and their states. I looked right after the run:

$ ss -tan 'sport = :9001 or dport = :9001'State     Recv-Q Send-Q Local Address:Port  Peer Address:PortTIME-WAIT 0      0          127.0.0.1:9001    127.0.0.1:46104

The server side is still there, in TIME-WAIT, even though both programs have ended. The side that closes first stays in this state for a fixed time (60 seconds on Linux). Why? Two reasons that I know from background reading, not from this lab. First, the last ACK might get lost, so the side that sent the last FIN must be ready to resend it. Second, late copies of old packets could still be wandering around the network, and the system must not let a new connection with the same four numbers (source IP, source port, destination IP, destination port) accept them by mistake.

To see the other states, I held a connection open and closed one side at a time. First the whole picture while both sides are open, then after only the client closed, then after both closed:

LISTEN     0      128        127.0.0.1:9030       0.0.0.0:*ESTAB      0      0          127.0.0.1:51968    127.0.0.1:9030ESTAB      0      0          127.0.0.1:9030     127.0.0.1:51968(client calls close())FIN-WAIT-2 0      0          127.0.0.1:51968    127.0.0.1:9030CLOSE-WAIT 1      0          127.0.0.1:9030     127.0.0.1:51968(server calls close())TIME-WAIT  0      0          127.0.0.1:51968    127.0.0.1:9030

• LISTEN: a server waiting for connections.
• ESTAB: an open connection. Two lines, one for each end.
• FIN-WAIT-2: "I closed my side, and I am waiting for the other side to close theirs".
• CLOSE-WAIT: "The other side closed. I must now call close() too." The Recv-Q 1 is the end-of-stream marker waiting to be read.
• TIME-WAIT: described above.

TCP is a stream, not messages

This is a classic beginner bug. TCP delivers bytes in order, but it does not keep the boundaries of your send calls. I sent three separate pieces and read once:

c.sendall(b"hello "); c.sendall(b"world"); c.sendall(b"!")   # three sends# server side, a moment later:print("server recv #1:", c.recv(100))

server recv #1: b'hello world!'

All three arrived glued together in one read. The reverse also happens: one big send can arrive in several reads. TCP is allowed to split and join your data any way it likes, because the network decides the packet sizes. This means any protocol on top of TCP must say where a message ends. HTTP does it with the Content-Length header or with chunked encoding. Many binary protocols put a length number before each message. If you write socket.on("data", handler) and assume that each event is one complete message, your code works on your laptop and breaks in production.

Now the same three pieces with UDP:

u = socket.socket(socket.AF_INET, socket.SOCK_DGRAM); u.bind(("127.0.0.1", 9004))t = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)for m in (b"hello ", b"world", b"!"): t.sendto(m, ("127.0.0.1", 9004))for i in range(3): print("udp recv", i+1, u.recvfrom(100)[0])

udp recv 1 b'hello 'udp recv 2 b'world'udp recv 3 b'!'

UDP keeps one sendto = one recvfrom. A datagram arrives whole or not at all.

What "connection refused" and "timed out" really mean

These two error messages are different events on the wire. You can tell what is wrong from the message alone.

Connection refused. I connected to a port where nobody listens:

ConnectionRefusedError [Errno 111] Connection refusedIP 127.0.0.1.49232 > 127.0.0.1.9999: Flags [S], seq 3390641222, win 65495, ...IP 127.0.0.1.9999 > 127.0.0.1.49232: Flags [R.], seq 0, ack 3390641223, win 0, length 0

The computer answered at once with a RST (reset): "nothing here, go away". Refused is fast and means: the machine is up and reachable, but no program listens on that port (the server crashed, wrong port, wrong host, or it listens only on 127.0.0.1 while you connect from outside).

Timed out. Now I made the SYN disappear, with a firewall rule that silently drops it. A real firewall that "drops" does the same. I captured what the client did:

connect failed after 8.0s: TimeoutError1792243998.975  49004 > 9040: Flags [S], seq 31163257161792243999.985  49004 > 9040: Flags [S], seq 3116325716     (+1.0 s)1792244001.009  49004 > 9040: Flags [S], seq 3116325716     (+1.0 s)1792244002.033  49004 > 9040: Flags [S], seq 3116325716     (+1.0 s)1792244003.057  49004 > 9040: Flags [S], seq 3116325716     (+1.0 s)1792244004.081  49004 > 9040: Flags [S], seq 3116325716     (+1.0 s)1792244006.097  49004 > 9040: Flags [S], seq 3116325716     (+2.0 s)

Same SYN, same sequence number, sent again and again, and no answer ever came. The kernel documentation says that this Linux version sends the first few retries at one-second steps and then doubles (the pattern is 1, 1, 1, 1, 1, 2, 4 seconds, controlled by tcp_syn_linear_timeouts, which defaults to 4), and that with the default of 6 SYN retries it finally gives up after about 131 seconds. My capture matches the start of that pattern. I stopped after 8 seconds because my own script had a timeout of 8 seconds. (Older kernels doubled from the start: 1, 2, 4, 8. That is from my memory, not tested.)

So timeout means: the packets vanish. Common causes are a firewall or security group that drops instead of rejecting, a wrong IP address, a server that is down with no one to send an RST, or a route that leads nowhere.

The listen backlog: connections waiting to be accepted

There is one more queue to know about. When a client's handshake finishes, the connection is not given to your program yet. It waits in the accept queue until your code calls accept(). The size is the argument to listen(). I made a server that listens with a backlog of 2 and never calls accept, and connected 3 times:

$ ss -ltn 'sport = :9005'          (before connecting)State  Recv-Q Send-Q Local Address:PortLISTEN 0      2          127.0.0.1:9005$ ss -ltn 'sport = :9005'          (after 3 clients connected)LISTEN 3      2          127.0.0.1:9005

For a listening socket, ss shows Recv-Q = how many connections wait in the queue and Send-Q = the maximum size. Here three connections waited, with a limit of 2, so Linux let one more in than the number I asked for (I did not look up why; I only report what ss said). If the queue is full, new handshakes are slowed or dropped, and clients see slow connects or timeouts. When a busy server shows Recv-Q growing on its listening port, the application is not calling accept fast enough: it is blocked, busy, or out of threads. The kernel page also mentions tcp_abort_on_overflow, off by default: when on, it resets connections instead of letting them wait.

Reliability: numbering, acknowledging and sending again

We saw the numbers. Now what if a packet is lost? The sender keeps every unacknowledged byte, and starts a timer. If no ACK covers those bytes before the timer ends, it sends them again.

I tested it. The client connects, and then I added a firewall rule that drops every packet to the server that carries data (the handshake packets are small, so they get through; the length filter 54: matches only packets of 54 bytes or more). The client sends hello. After 1.3 seconds I removed the rule. Here is what the capture on the sender side saw:

1792243855.353728  41720 > 9007: Flags [P.], seq 1:6, ack 1, win 64, length 51792243855.561970  41720 > 9007: Flags [P.], seq 1:6, ack 1, win 64, length 5    (+0.208 s)1792243855.765152  41720 > 9007: Flags [P.], seq 1:6, ack 1, win 64, length 5    (+0.203 s)1792243856.181191  41720 > 9007: Flags [P.], seq 1:6, ack 1, win 64, length 5    (+0.416 s)1792243857.009135  41720 > 9007: Flags [P.], seq 1:6, ack 1, win 64, length 5    (+0.828 s)1792243857.009149  9007 > 41720: Flags [.],  ack 6, win 64, length 0              <- finally got through

The same five bytes (seq 1:6) were sent five times. The waits between tries were about 0.2 s, 0.2 s, 0.4 s and 0.8 s. The timer doubles after each failure. This is called exponential backoff, and it is there so a sick network is not made worse by every sender shouting faster. The first wait of about 200 ms is the lower limit that Linux uses for the retransmission timeout.

While the packets were being dropped, I asked ss for the connection's details (ss -tin, the i means "show internals"):

ESTAB 0      5     127.0.0.1:41720    127.0.0.1:9007	 bbr wscale:10,10 rto:800 backoff:2 rtt:0.229/0.114 mss:32741 cwnd:1 bytes_sent:20	 bytes_retrans:15 bytes_acked:1 segs_out:6 data_segs_out:4 unacked:1 retrans:1/3 lost:1 ...

Here is how to read the useful fields. Send-Q 5 means 5 bytes were sent but not acknowledged. rto:800 is the current timer (800 ms), backoff:2 says it has doubled twice, retrans:1/3 says one segment is waiting to be resent and three resends happened in total, unacked:1 and lost:1 say what the kernel believes, and cwnd:1 shows something we will meet below: after a timeout, TCP shrinks its sending allowance to one segment. The rtt:0.229 is the smoothed round trip time measured earlier. The sender keeps measuring RTT and builds its timer from it, so on a slow link the timer is longer.

The program on the other side never saw anything odd. The server printed server got b'hello' after 1.66s. The data arrived once, complete and in order, only later. That is the promise of TCP: it hides loss by trading time for it. A lost packet shows up in your application as a pause.

Some more facts from the kernel page that explain the "two minutes" delays. A connection that already works but whose data is not getting acknowledged is not abandoned quickly. With the default tcp_retries2 = 15, the docs say the connection is killed after about 924 seconds (more than 15 minutes) of unacknowledged retransmissions. This is why a pulled network cable can leave a program waiting for a very long time unless it has its own timeouts or TCP keep-alive.

Flow control: the receiver says "slow down"

Reliability is half the story. What if the sender is fast and the receiving program is slow? The receiver has a buffer of limited size for data that has arrived but your code has not read. In every ACK, the receiver includes its window: how many more bytes it has room for. The sender must never have more unacknowledged data in flight than that. If the buffer is full, the window becomes 0, and the sender must stop.

I tested it with a server that accepts a connection and then does not read for three seconds. I made the buffers small (4 KB) so it fills fast. The client sends until send() would block:

client could queue 9000 bytes then send() would blockState  Recv-Q Send-Q Local Address:Port  Peer Address:PortESTAB  5096   0      127.0.0.1:9006      127.0.0.1:37432     <- the server: 5096 bytes waiting for the appESTAB  0      3904   127.0.0.1:37432     127.0.0.1:9006      <- the client: 3904 bytes it cannot send yet

You can read the pressure from ss. On the receiver, Recv-Q is data that arrived but the program has not read. On the sender, Send-Q is data the program wrote but the other side has not accepted. A growing Recv-Q on one machine and a growing Send-Q on the other tells you which side is slow. Then the capture:

41.013  9006 > 47044: Flags [.], ack 3001, win 2096, length 041.013  47044 > 9006: Flags [P.], seq 3001:5097, ack 1, win 64, length 209641.033  47044 > 9006: Flags [.], seq 5049:5097, ack 1, win 64, length 4841.057  9006 > 47044: Flags [.], ack 5097, win 0, length 0              <- window is ZERO41.277  47044 > 9006: Flags [.], ack 1, win 64, length 0                <- probe (+0.22 s)41.277  9006 > 47044: Flags [.], ack 5097, win 0, length 0              <- still zero41.713  47044 > 9006: Flags [.], ack 1, win 64, length 0                <- probe (+0.44 s)42.577  47044 > 9006: Flags [.], ack 1, win 64, length 0                <- probe (+0.86 s)42.577  9006 > 47044: Flags [.], ack 5097, win 0, length 043.972  9006 > 47044: Flags [.], ack 5097, win 4096, length 0           <- app read; window opens43.972  47044 > 9006: Flags [P.], seq 5097:9193, ack 1, win 64, length 4096

(The times are the seconds part of the timestamp, so 41.057 to 41.277 is 0.22 s. I dropped the rest of the number to fit.)

Look at the window. In the first packets of this run it shrank (3096, then 2096, as the buffer filled), and here it reaches 0. The receiver told the sender "stop". The sender then sent small probe packets with no data, at growing gaps (0.22 s, 0.44 s, 0.86 s), to ask "is there room yet?" This matters, because the packet that says "the window is open again" could itself be lost, and without probes both sides would wait forever. At 43.972 the program finally read data. The window jumped to 4096 and the sender went at full speed again. At the end the server printed server finally read 59000 bytes: all of it arrived.

This is backpressure, and it is built into TCP. In Node.js, socket.write() returns false when the buffer is full and you should wait for the drain event. That false is this window mechanism reaching your code. If you ignore it and keep writing, your process keeps the data in memory, and a slow client can make your server run out of memory.

Congestion control: the sender protects the network

Flow control protects the receiver. Congestion control protects the network in the middle. The sender does not know how fast the path can go. If everyone sends at full speed, routers' queues overflow, packets are lost, and everybody resends, which makes it worse. In the 1980s the early internet did suffer from exactly this "congestion collapse". The cure is that every TCP sender keeps a second limit, the congestion window (cwnd), and treats packet loss (or growing delay) as a sign that the path is full.

The classic approach, in plain words:

1. Slow start. Begin with a small window (about 10 segments on modern Linux, from my background knowledge). Each time an ACK returns, increase it. In practice the window doubles every round trip. It is "slow" only compared with sending everything at once.
2. Congestion avoidance. After a while, grow by only about one segment per round trip.
3. On loss, back off. Cut the window (the old Reno algorithm halves it) and carry on. After a timeout, as we saw with cwnd:1, restart from tiny.

The result is a sawtooth: climb slowly until loss, drop, climb again. The sender is always probing for the limit of the path. Newer algorithms change the details. CUBIC (the usual default on Linux) grows along a curve and recovers faster. BBR tries to measure the path's bandwidth and delay instead of reacting to loss. You can see what is available and which is used:

$ sysctl net.ipv4.tcp_congestion_control net.ipv4.tcp_available_congestion_controlnet.ipv4.tcp_congestion_control = bbrnet.ipv4.tcp_available_congestion_control = reno bbr cubic

The kernel docs say the default can be changed for the whole machine, and a program can pick one for a single connection with the TCP_CONGESTION socket option. I used that to compare algorithms in the same test.

A real bottleneck

To see the sawtooth I needed a link that is slower than the sender. I made two virtual network cards (a veth pair) in two network namespaces, one for the server and one for the client, with a normal 1500 byte packet size. Then I put a token bucket filter on the client's card, which limits the speed to 20 Mbit/s and has a small queue (60 KB). Anything that does not fit in the queue is dropped. This is the same situation as a slow office uplink. The client sent 25 MB, and every 0.25 seconds I read ss -tin for the connection:

== cubic: 18.3 Mbit/s, 10.9s, retransmitted segments 21  t=0.45s cwnd=30 rtt=8.041ms  retrans_total=3  t=0.71s cwnd=36 rtt=18.819ms retrans_total=3  t=0.96s cwnd=28 rtt=12.651ms retrans_total=3  t=1.22s cwnd=34 rtt=15.879ms retrans_total=4  t=1.47s cwnd=40 rtt=18.375ms retrans_total=4  t=1.73s cwnd=30 rtt=14.282ms retrans_total=5  t=1.98s cwnd=37 rtt=17.233ms retrans_total=5  t=2.24s cwnd=29 rtt=17.994ms retrans_total=5  t=2.49s cwnd=33 rtt=15.776ms retrans_total=6  t=2.75s cwnd=39 rtt=18.23ms  retrans_total=6

Two things to notice. First, the sawtooth: cwnd goes 30, 36, 28, 34, 40, 30. It climbs, a loss happens, it drops, it climbs again. Second, the round trip time rose from under a millisecond to 8 to 19 ms. A ping across the empty cable took 0.59 ms. The extra time is the packets waiting in the queue of the slow link. This effect, where a full buffer adds delay, is called bufferbloat. It is why a big upload can make a video call lag on the same connection.

The three algorithms, same test (25 MB, one run each, so treat the numbers as an example and not as a ranking):

All three filled the 20 Mbit/s link to about the same level, which shows that congestion control works: each one found the speed limit without being told. The big difference is how they find it. BBR kept the queue and the delay lower (its RTT readings were 5 to 10 ms against CUBIC's 13 to 19 ms), but with this very small queue it caused many more drops, because it keeps pushing at the speed it estimates. I do not take this as a verdict on BBR, since one short test with one tiny queue is not enough. I take it as proof that the algorithm is a real setting with real effects, and that ss -tin shows you which one a connection uses (the word bbr, cubic or reno at the start of the details line).

What loss costs depends on the round trip time

I also ran a plain 50 MB transfer on loopback with 0%, 0.1%, 1% and 5% of data packets dropped by iptables. Every byte arrived every time (bytes_received=50000000), the time stayed at about 0.05 s, and retransmitted segments went from 0 to 47 (CUBIC, 5% loss). That does not mean loss is harmless. On loopback the round trip is a few microseconds, so a lost segment is noticed and resent almost at once. On a real path with a 100 ms RTT every recovery costs at least one 100 ms round trip, and the window is cut each time. I could not show that, because netem (which adds delay) is missing here. From background knowledge, TCP throughput falls roughly with the square root of the loss rate and with growing RTT, so a small loss on a long path hurts a lot. That is one reason CDNs put servers near users (see lesson 6).

Two things that hit developers

• Cold connections are slow. A new connection starts with a small window and must grow it. A connection that has moved a lot of data is "warm" and fast. The MDN page on connection management says the same thing in HTTP words: short-lived connections do not benefit from TCP's ability to adapt to load. Linux even shrinks the window again after the connection has been idle for a while (tcp_slow_start_after_idle, which is on by default on this machine; I read the sysctl value, I did not measure the effect). Connection pools and keep-alive keep connections warm.
• Big downloads fill the pipe. One long transfer can occupy the whole queue of a slow link, and every other request on that link waits behind it. If your app is slow only while a backup runs, think bufferbloat.

UDP: no connection, no promises

UDP has no handshake, no sequence numbers, no acknowledgements, no windows. A datagram is sent and that is all. The header is only 8 bytes (source port, destination port, length, checksum). I tested what that means in practice.

Loss is silent. I sent 100 numbered datagrams to a receiver while iptables dropped about 30% of them at random:

sent 100, received 59 ; sender saw no error at allfirst received: [3, 4, 8, 13, 15, 16, 17, 19, 22, 24, 25, 28]some missing: [0, 1, 2, 5, 6, 7, 9, 10, 11, 12]in order: True

41 messages vanished and neither side got any warning. (The rate came out higher than 30% in this run by chance. Order was kept here only because loopback has one path; on the real internet, datagrams can arrive in a different order, and I did not test that.) If you need to know whether it arrived, your program has to ask.

Errors can come back, sometimes. If you send UDP to a port where nothing listens, the receiving computer replies with an ICMP message "port unreachable". I captured it:

IP 127.0.0.1.60233 > 127.0.0.1.9021: UDP, length 7IP 127.0.0.1 > 127.0.0.1: ICMP 127.0.0.1 udp port 9021 unreachable, length 43

What your program sees depends on how you wrote the code. With a connected UDP socket (you called connect() on it, which for UDP only remembers the address) the error shows up on the next read:

connected UDP to closed port   -> ConnectionRefusedError [Errno 111] Connection refusedunconnected UDP to closed port -> TimeoutError

With a normal unconnected socket (sendto), the error was ignored by the kernel and my code only saw a timeout. So even "is anyone there?" is unreliable with UDP.

Size limits. A UDP datagram can be at most 65,507 bytes of data (65,535 minus the 20-byte IP header and the 8-byte UDP header). I tried it:

65507 bytes: sent65508 bytes: [Errno 90] Message too long

In real networks a datagram larger than the path's packet size (often 1500 bytes) has to be cut into fragments by IP, and if any fragment is lost, the whole datagram is lost. So protocols that use UDP, like DNS and QUIC, try to keep datagrams small. I did not test fragmentation here.

Building reliability on top of UDP

What if you want UDP's speed and a little reliability? You write it yourself. This is the smallest version, "stop and wait": send a numbered message, wait for an ACK, and if the timer runs out, send again. The receiver always acknowledges, and it ignores repeats:

# rudp.py (simplified)srv = socket.socket(socket.AF_INET, socket.SOCK_DGRAM); srv.bind(("127.0.0.1", 9022))def server():    seen = set()    while True:        d, a = srv.recvfrom(100); seq = int(d.split()[0])        srv.sendto(b"ACK %d" % seq, a)                 # always ack        if seq not in seen: seen.add(seq); print("  server delivered seq", seq)c = socket.socket(socket.AF_INET, socket.SOCK_DGRAM); c.settimeout(0.05)for seq in range(8):    while True:        c.sendto(b"%d payload" % seq, ("127.0.0.1", 9022))        try:            if c.recv(100) == b"ACK %d" % seq: break        except socket.timeout: print("  timeout, resend seq", seq)

With a firewall rule that drops half of all datagrams to the server:

  timeout, resend seq 0  timeout, resend seq 0  server delivered seq 0  timeout, resend seq 1  server delivered seq 1  ...  timeout, resend seq 3  (six times in a row)  server delivered seq 3  ...8 messages delivered using 24 sends

All 8 messages were delivered once, in order, at the price of 24 sends. This tiny program has the same ideas as TCP: sequence numbers, ACKs, a timer and duplicate detection. A real design also needs the things we saw above: a window so many messages are in flight (stop-and-wait wastes the link), an adaptive timer, and congestion control. That is hard work, which is why protocols such as QUIC (under HTTP/3) are built in user space on UDP by specialists, and why you should almost never invent your own. We look at QUIC in lesson 4.

When to choose which

Use TCP for web pages, APIs, files and databases, anything where every byte matters. Use UDP when a late packet is worse than a lost one (live voice and video, game state sent 60 times a second), for tiny question-and-answer traffic like DNS (with TCP as a fallback, see lesson 3), for local discovery and broadcast, and as the carrier for QUIC (HTTP/3), which wants TCP's reliability without TCP's head-of-line blocking.

Rule of thumb: use TCP unless you have a clear reason and the skill to handle loss yourself. "TCP is slow" is rarely the real problem. More often the problem is the number of round trips, which better connection reuse fixes.

A short checklist for TCP problems

Read the error first (refused, timed out, reset and "hangs" are four different problems). Then ss -ltnp on the server (is anything listening, and on which address?), ss -tn on both ends (read the queues), ss -s (count the states), ss -tin (RTT, RTO, cwnd, retransmissions) and finally tcpdump -i any -nn 'host 10.0.0.5 and tcp port 443' -w cap.pcap to follow one connection from SYN to FIN. Lesson 8 puts these into a full workflow.

Try it

You need Python 3 and tcpdump (and root for the last two).

1. Run the echo server and client from this lesson with tcpdump -i lo -nn 'tcp port 9001'. Without -S, what is the ack number in the third packet of the handshake? Why?
2. Run the client with the server not started. What error do you get? Then change the client to connect to 192.0.2.1 (a documentation address that nobody answers) with a 3 second timeout. What error do you get now, and why is it different?
3. Write a server that accepts a connection and reads once with recv(100). Make the client call sendall three times with a short time.sleep(0.2) before the server reads. How many bytes does recv return? Now make the client send 1,000,000 bytes in one call and count how many recv calls the server needs if it reads 4096 bytes at a time.
4. (Root needed.) With iptables -A INPUT -i lo -p tcp --dport 9007 -m length --length 54: -j DROP, connect a client, send hello, and remove the rule after 2 seconds with iptables -D and the same arguments. Capture with tcpdump and write down the gaps between the resends.

Answers

1. ack 1. The server's SYN used sequence number 0 (relative), and a SYN counts as one byte, so the client acknowledges "I have everything up to byte 1". With absolute numbers (-S) it is the server's ISN plus 1.
2. Without a server: ConnectionRefusedError (an immediate RST). With an address that drops everything: TimeoutError after your 3 seconds. Refused comes from an answer, timeout comes from silence.
3. Usually all 3 pieces arrive in a single recv (for example 12 bytes if you send hello , world and !), because the kernel has joined them by the time you read. For 1,000,000 bytes you need at least 245 reads of 4096 bytes, and often more because some reads return less. This is why you must loop until you have the whole message.
4. Expect gaps of roughly 0.2 s, 0.2 s, 0.4 s, 0.8 s, 1.6 s, and so on: the timer doubles. Your numbers will differ a little.

Interview questions

Key takeaways

• TCP opens with SYN, SYN-ACK, ACK. It costs one round trip, so reuse connections.
• Every byte has a number. The receiver's ACK says what it has. The sender resends after a timer that doubles on each failure (about 0.2, 0.4, 0.8 s in my capture).
• TCP is a byte stream. It does not keep message boundaries, so your protocol needs framing.
• Flow control: the receiver's window. A zero window makes the sender stop and probe. In Node, write() returning false is this reaching your code.
• Congestion control: the sender's congestion window grows, then drops after loss. It makes a sawtooth and also builds queues (bufferbloat).
• Error meanings: refused = fast RST (nothing listens). Timeout = silence (packets dropped). Reset = connection killed mid-way.
• ss -tan, ss -tin and ss -ltn show states, queues, RTT, cwnd and retransmissions. Many CLOSE-WAIT means a bug in your code.
• UDP has no connection, no retransmission, no order guarantee and no error reporting. Loss is silent. If you need reliability on UDP you must build it.
• Always set connect and read timeouts. The operating system's defaults are measured in minutes.

What's next

Next: DNS: How Names Become Addresses. Before TCP can connect to example.com, something must turn that name into an IP address, and that something is mostly a few tiny UDP packets. We will build a small DNS system of our own (a root server, a top-level server and an authoritative server on your own machine), watch a resolver follow the referrals step by step, see TTLs and caches in action, and learn the dig commands for debugging. Practise now: do the "Try it" items 2 and 3, because the difference between refused and timed out is the first thing to check when a name resolves to the wrong place.