Linux & Shell
SSH, Networking Commands and Transferring Files
SSH keys and config, scp and rsync, port forwarding, curl, ss, dig, ping and traceroute, and what each tells you.
On this page
- Words used in this lesson
- The big picture: what happens when you type ssh
- Keys: making a pair
- Trusting the server: knownhosts
- The SSH config file: stop typing long commands
- Reading the server's side of the story
- Copying files: scp
- Copying smartly: rsync
- Tunnels: port forwarding
- Jump hosts: ProxyJump
- ssh-agent: type your passphrase once
- The small network tools
- A debugging order that works
- Try it
- Interview questions
- Key takeaways
- What's next
In one line: SSH lets you log in to another computer and run commands on it, safely, using a key pair instead of a password. Once SSH works, you can copy files (scp, rsync), reach private services through a tunnel (port forwarding), and use a few small tools (curl, ss, dig, ping, traceroute, nc) to see why a network call fails.
Almost every server you will ever touch is far away. It is in a data centre, or in a cloud account, and you cannot plug in a screen. You reach it over the network. The way to do that on Linux is SSH. If you deploy code, read production logs, or fix a broken service at night, you will type ssh many times a week.
This is lesson 5 of nine. In lesson 4 we wrote safe shell scripts. Here we use the shell across the network. In the next lesson, packages, services, cron and systemd, we will manage what runs on that remote server. If the word "port" or "DNS" feels fuzzy, Networking lesson 3 (DNS) goes deeper on names, and the networking series covers the layers under all of this. Here we stay practical: which command, what the output means, what the usual mistake is.
What I ran / checked. Everything below ran on Ubuntu 24.04.5 (Linux kernel 6.18), as root inside a sandbox VM, with OpenSSH 9.6p1 (client and server), rsync 3.2.7, curl 8.5.0, iproute2 6.1.0 (ss), dig 9.18 and GNU bash 5.2. I started a real local sshd on 127.0.0.1 port 2222 with my own config and my own host key, key-only login, for a throwaway user called deploy. It listens on the loopback address only, so nothing outside the machine can reach it. I made all keys fresh for the lab; the key text you see is from the lab and is not used anywhere else. Sources I read (in my own words below): the ssh_config(5) and sshd_config(5) manual pages from the openssh/openssh-portable repository on GitHub, and the networking part of the jlevy/the-art-of-command-line README. Not run or not reachable: the sandbox has no real internet DNS (my dig example.com returned NXDOMAIN from the sandbox resolver and curl https://example.com was refused by the sandbox proxy), so I could not show real public DNS answers. I did not run ssh -R (remote forward), ssh -D (SOCKS proxy), ControlMaster, or ssh-copy-id; those parts are labelled "reviewed, not run". I did not verify any GitHub star counts. Timings and port numbers in the output change on every run.
Words used in this lesson
- SSH (Secure Shell) is a protocol and a program for logging in to a remote computer over an encrypted connection.
- Client and server. The computer you type on runs the SSH client (
ssh). The remote computer runs the SSH server, a background program calledsshd(the "d" means daemon). - Daemon is a program that runs in the background and waits for work, like
sshd. - Key pair is two files made together: a private key (secret, stays on your machine) and a public key (safe to share, goes on the server).
- Fingerprint is a short hash of a key. It is a name tag you can compare by eye.
- Host key is the key the server uses to prove who it is to you.
known_hostsis a file on your side that remembers the host key of each server you have visited.authorized_keysis a file on the server that lists the public keys allowed to log in as that user.- Port is a numbered door on a computer. One program listens behind each open door. SSH normally uses port 22.
- Port forwarding (tunnel) means sending traffic for one port through an SSH connection to another computer.
- Jump host (bastion) is a server you SSH through to reach other servers that are not open to the internet.
scpcopies files over SSH.rsynccopies only what changed, over SSH.- Socket is one end of a network connection.
ssshows sockets. - DNS turns a name like
example.cominto an IP address.digasks DNS questions. - HTTP status code is the number in the reply (200 OK, 404 not found).
curlmakes HTTP requests from the shell.
The big picture: what happens when you type ssh
You type ssh deploy@server. Three things happen, in this order.
- Your client checks the server. The server shows its host key. Your client looks in
known_hosts. If it has never seen this server, it asks you whether to trust it. If it has seen the server and the key is different, it stops and warns you loudly, because that can mean someone is pretending to be the server. - The server checks you. You prove you own the private key that matches a public key in
~/.ssh/authorized_keyson the server. The private key never leaves your machine. Your client signs a challenge with it. - You get a shell (or one command runs, and the connection closes).
Everything between your machine and the server is encrypted. That is the "S" in SSH.
Here is the first real login from my lab. I ran one command on the "remote" machine (which is really this same VM, reached through 127.0.0.1):
$ ssh -F config web 'whoami; hostname; echo $SSH_CONNECTION'deployvm127.0.0.1 41874 127.0.0.1 2222whoami printed deploy, not root. That proves the command ran as the remote user, inside the SSH session. SSH_CONNECTION is a variable the server sets. It lists the client address and port, then the server address and port. This is handy when you are not sure which machine you are on.
[!tip] Running one command instead of a full shell: ssh host 'command' runs that command, prints the output, and returns the command's exit code as its own. That makes SSH usable inside scripts (see lesson 4). I checked: ssh web 'exit 7'; echo $? printed 7. The special value 255 means SSH itself failed (could not connect or log in), not the remote command.
Keys: making a pair
A password can be guessed, typed on a bad keyboard, or stolen from a shared note. A key pair is better because the secret part never travels. Make one with ssh-keygen:
$ ssh-keygen -t ed25519 -N "" -C "demo@lab" -f ./id_demoGenerating public/private ed25519 key pair.Your identification has been saved in ./id_demoYour public key has been saved in ./id_demo.pubThe key fingerprint is:SHA256:K4lfPANmXMsDRv1FcSgmzTgvhaIdwCIFdjC4xmPPDac demo@lab$ ls -l id_demo id_demo.pub-rw------- 1 root root 399 Oct 12 03:42 id_demo-rw-r--r-- 1 root root 90 Oct 12 03:42 id_demo.pub$ cat id_demo.pubssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKc6phyzJOJd7nVyWW2viDHqCRDwATR18jjftcZOhWVs demo@labWhat each flag did:
-t ed25519chose the key type. Ed25519 is a modern, small, fast type and a good default. An older type, RSA, still works everywhere; use at least 3072 bits (-b 3072). I made one too:3072 SHA256:r0IfL3YF... (RSA).-N ""set an empty passphrase. I did this only because this is a throwaway lab key. For a real key, set a passphrase. A passphrase encrypts the private key file on disk, so a stolen laptop does not mean a stolen server.-Cis just a comment (a label, heredemo@lab).-fchose the file name.
Look at the two files. The private key is -rw------- (mode 600, only me). The public key is -rw-r--r--. The one-line public key is safe to paste into chats, tickets and GitHub. The private key must never leave your machine.
SSH refuses a private key that others can read. I copied my key, made it chmod 644, and tried to use it:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ WARNING: UNPROTECTED PRIVATE KEY FILE! @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@Permissions 0644 for 'loose' are too open.It is required that your private key files are NOT accessible by others.This private key will be ignored.Load key "loose": bad permissionsdeploy@127.0.0.1: Permission denied (publickey,keyboard-interactive).The fix is chmod 600 keyfile. Notice the last line says only "Permission denied". The useful hint is a few lines above it. Read the whole output, not only the last line. (This is the permissions skill from lesson 1.)
Putting the public key on the server
The server needs your public key in ~deploy/.ssh/authorized_keys. In real life you often use ssh-copy-id user@host, which does this for you (I did not run it; I copied the file by hand, which is all it does). The server is strict about permissions here too:
~/.sshshould be mode700and owned by that user.authorized_keysshould be mode600.
I tested what happens when the folder is open to everyone (chmod 777 ~deploy/.ssh). The client only said Permission denied (publickey...). The server log told the real story:
Authentication refused: bad ownership or modes for directory /home/deploy/.sshAfter chmod 700 ~deploy/.ssh the login worked again. This setting is called StrictModes in sshd_config: the server refuses keys in files that other users could change, because they could add their own key. The lesson: when SSH login fails and the client message is vague, read the server's log. I show how to read it in a moment.
A second trap I hit: the locked account
My first lab login failed even with a correct key and correct permissions. The server log said:
User deploy not allowed because account is lockedI had made the user with useradd -m deploy. On this system that sets the password field to !, which means "locked". With UsePAM no in my lab config, sshd honours that and refuses even key logins. I fixed it with usermod -p '*' deploy, which means "no password can ever match, but the account is not locked". On a typical distribution with UsePAM yes this particular problem does not show up, but it is a good reminder that the account itself can be the reason for a refusal, not only the key. Again the answer was in the server log, not in the client message.
Trusting the server: known_hosts
The first time you connect to a new server, the client has no record of it. In my lab I used StrictHostKeyChecking=accept-new, which accepts a new host and records it, but still refuses if a known host changes:
Warning: Permanently added '[127.0.0.1]:2222' (ED25519) to the list of known hosts.Then I looked at the file:
$ cut -c1-80 known_hosts[127.0.0.1]:2222 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKUmn8bP3pOvlvdulAu61/G6RI+The square brackets and :2222 are used because the port is not 22. You can compare the fingerprint you see to the real one on the server:
$ ssh-keygen -lf hostkey.pub # run on the server (or its admin does)256 SHA256:8f+xw0kI9HWFJ9CiSfbEYwm7ptKedbkWvtLKGUlPULs host (ED25519)$ ssh -F config2 -v web true 2>&1 | grep "Server host key"debug1: Server host key: ssh-ed25519 SHA256:8f+xw0kI9HWFJ9CiSfbEYwm7ptKedbkWvtLKGUlPULsThey match. That is how you check you are talking to the right machine on the first visit. In real life, many people just type "yes". The better habit is to get the fingerprint from a trusted place (your cloud console, or the person who set up the server) at least for important servers.
"WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED" is not an error to silence. It can be harmless (you rebuilt the server, so it has a new key). It can also mean a man-in-the-middle. Do not blindly delete the line or turn off checking. Confirm with whoever runs the server first. To remove one stale entry cleanly use ssh-keygen -R hostname. Never put StrictHostKeyChecking no in a config you use for real servers.
The SSH config file: stop typing long commands
Typing ssh -p 2222 -i ~/keys/id_demo deploy@127.0.0.1 every time is tiring and easy to get wrong. The client reads ~/.ssh/config, where you give each server a short name. Here is the file I used:
Host * ServerAliveInterval 30 UserKnownHostsFile /tmp/lab-linux-b/ssh1/known_hosts StrictHostKeyChecking accept-newHost web HostName 127.0.0.1 Port 2222 User deploy IdentityFile /tmp/lab-linux-b/ssh1/id_demo IdentitiesOnly yesHost inner HostName 127.0.0.1 Port 2222 User deploy IdentityFile /tmp/lab-linux-b/ssh1/id_demo IdentitiesOnly yes ProxyJump webNow ssh web is enough. Some notes from the manual page, in my own words:
- The client uses the first value it finds for each setting, reading from the top. So put the specific
Host webblocks before a generalHost *block if you want the specific value to win. (MyHost *only sets things the others do not set, so order did not matter here.) IdentitiesOnly yestells the client to offer only the key named inIdentityFile. Without it, the client may offer every key it has (and every key in your ssh-agent) one by one. A server allows only a few tries, so with many keys you can get locked out with "Too many authentication failures".ServerAliveInterval 30sends a small "are you there" message every 30 seconds, so an idle connection is not dropped by a firewall.
See what SSH will actually do: ssh -G
ssh -G name prints the final settings for that name without connecting. This is the best way to debug a config file:
$ ssh -F config -G web | egrep -i '^(hostname|port|user|identityfile|identitiesonly|serveraliveinterval) 'user deployhostname 127.0.0.1port 2222identitiesonly yesserveraliveinterval 30identityfile /tmp/lab-linux-b/ssh1/id_demo$ ssh -F config -G db | egrep -i '^(hostname|port|user|proxyjump|identityfile) 'user adminhostname 10.0.0.5port 22identityfile ~/.ssh/id_rsaidentityfile ~/.ssh/id_ecdsaidentityfile ~/.ssh/id_ecdsa_skidentityfile ~/.ssh/id_ed25519identityfile ~/.ssh/id_ed25519_skidentityfile ~/.ssh/id_xmssidentityfile ~/.ssh/id_dsaproxyjump webTwo things to notice. For db I did not set a port, so you see the default, 22. And because I did not set IdentityFile for db, the client will try its default list of key files. That is exactly the "many keys" situation IdentitiesOnly is meant to avoid. (The tool also printed Pseudo-terminal will not be allocated because stdin is not a terminal., which is harmless noise here.)
Debugging a login: ssh -v
Add -v (up to -vvv) to see the steps. A few lines are enough to follow the story:
$ ssh -F config2 -v web true 2>&1 | egrep -i "offering|authenticated|Authentications that can"debug1: Authentications that can continue: publickey,keyboard-interactivedebug1: Offering public key: /tmp/lab-linux-b/ssh1/id_demo ED25519 SHA256:K4lfPANm... explicitAuthenticated to 127.0.0.1 ([127.0.0.1]:2222) using "publickey".You can read it as: "the server accepts public keys; I offered this key; it worked."
Reading the server's side of the story
The server writes its own log. On a real server that is usually journalctl -u ssh (see lesson 6) or /var/log/auth.log. In my lab I sent it to a file with -E and set LogLevel VERBOSE. These are real lines from the run:
Accepted key ED25519 SHA256:K4lfPANm... found at /home/deploy/.ssh/authorized_keys:1Accepted publickey for deploy from 127.0.0.1 port 41930 ssh2: ED25519 SHA256:K4lfPANm...and from the failures:
User deploy not allowed because account is lockedAuthentication refused: bad ownership or modes for directory /home/deploy/.sshThe pattern: client says "Permission denied", server explains why. If you only have access to one side, you are guessing.
The minimal server config I used is worth reading, because it is what a safe baseline looks like:
Port 2222ListenAddress 127.0.0.1HostKey /tmp/lab-linux-b/ssh1/hostkeyPasswordAuthentication noPermitRootLogin noAllowUsers deployAuthorizedKeysFile .ssh/authorized_keysUsePAM noLogLevel VERBOSESubsystem sftp /usr/lib/openssh/sftp-serverPasswordAuthentication nomeans only keys work. Bots on the internet try passwords all day on port 22; this turns that whole attack off.PermitRootLogin nomeans nobody logs in directly as root; they log in as a normal user and usesudo(lesson 1).AllowUsers deploymeans only that user may log in.sshd -t -f sshd_configtests the config without starting the server. It printed nothing wrong (I echoedconfig-ok). Always run it before you restartsshdon a real server, because a broken config can lock you out.
Before you change sshd_config on a real server, keep a second SSH session open. If the new config is broken or locks you out, the existing session stays alive, and you can fix it. Test with sshd -t, reload, then open a new session to check it works before you close the old one.
Copying files: scp
scp copies files over SSH, with the same syntax as cp plus host: in front of the remote path.
$ scp -F config note.txt web:/tmp/note-remote.txt$ ssh -F config web 'cat /tmp/note-remote.txt'hello from laptop$ scp -F config web:/tmp/note-remote.txt ./back.txt && cat back.txthello from laptopUpload is scp local host:remote. Download is scp host:remote local. Use -r for folders and -P (capital P) for the port, not -p (that one means "preserve times").
My first scp failed with a confusing error. The output was:
subsystem request failed on channel 0scp: Connection closedThe reason: newer OpenSSH versions run scp through the SFTP subsystem by default, and my minimal sshd_config had no Subsystem sftp line. I added Subsystem sftp /usr/lib/openssh/sftp-server, restarted my lab sshd, and it worked. A normal Ubuntu server config already has this line, so you will mostly meet this problem on locked-down or hand-written configs. I did not verify from the sources exactly which OpenSSH release changed the scp default, so I will not name one here; what I can say is that it happened on 9.6p1.
Copying smartly: rsync
scp copies everything every time. rsync compares source and destination and sends only the differences. For deploys, backups and big folders this is much faster. The usual flags: -a (archive: keep permissions, times and folders), -v (verbose), -n (dry run, show what would happen), and -e (which remote shell to use).
First run: it sends everything.
$ rsync -av -e "ssh -F config" src2/ web:/tmp/rs2/sending incremental file listcreated directory /tmp/rs2./a.txtsub/sub/b.txtsent 222 bytes received 96 bytes 212.00 bytes/sectotal size is 4 speedup is 0.01Second run: nothing changed, so nothing is sent (no file names listed):
$ rsync -av -e "ssh -F config" src2/ web:/tmp/rs2/sending incremental file listsent 118 bytes received 13 bytes 262.00 bytes/sectotal size is 4 speedup is 0.03Now I add one new file c.txt and do a dry run with --itemize-changes (-i), which prints a code for each change:
$ rsync -avn --itemize-changes -e "ssh -F config" src2/ web:/tmp/rs2/sending incremental file list.d..t...... ./<f+++++++++ c.txtsent 156 bytes received 23 bytes 358.00 bytes/sectotal size is 6 speedup is 0.03 (DRY RUN)<f+++++++++ means "a file is being sent to the remote and it is new". The .d..t...... line says the directory's time changed.
The trailing slash trap
This one catches everybody. Compare these two commands, which differ by one /:
$ rsync -a -e "ssh -F config" src2 web:/tmp/rs3/ # no slash after src2$ ssh -F config web 'find /tmp/rs3 | sort'/tmp/rs3/tmp/rs3/src2/tmp/rs3/src2/a.txt/tmp/rs3/src2/c.txt/tmp/rs3/src2/sub/tmp/rs3/src2/sub/b.txtWithout the slash, rsync copies the folder itself (so you get rs3/src2/...). With src2/ it copies the contents (so you get rs2/a.txt as above). Rule: src/ means "what is inside"; src means "the folder itself". The slash on the destination does not matter in the same way.
--delete and why you test it first
By default rsync never deletes anything at the destination. With --delete, it makes the destination an exact mirror, which means removing extra files there. I made a destination that had an extra file zzz, then ran a dry run with --delete:
$ rsync -avn --delete -e "ssh -F config" src2/ web:/tmp/rs4/sending incremental file listdeleting zzzsent 126 bytes received 27 bytes 306.00 bytes/sectotal size is 4 speedup is 0.03 (DRY RUN)The line deleting zzz is exactly what would happen. A wrong path with --delete can wipe a real folder. Always run with -n first when you use --delete.
Tunnels: port forwarding
Sometimes a service is not open to the internet on purpose. Think of an admin page or a database that listens only on the server's 127.0.0.1. SSH can carry a port from your laptop through the encrypted connection to the server, so you can reach it as if it were local.
In the lab, I started a tiny "internal admin" web server that listens only on 127.0.0.1:8081 (so it stands for a private service). Then I asked SSH to forward my local port 9000 to it through the server:
$ ssh -F config2 -L 9000:127.0.0.1:8081 -N -f web$ ss -ltnp | egrep ':(8081|9000)\b'LISTEN 0 128 127.0.0.1:9000 0.0.0.0:* users:(("ssh",pid=12965,fd=4))LISTEN 0 5 127.0.0.1:8081 0.0.0.0:* users:(("python3",pid=12957,fd=3))$ curl -s http://127.0.0.1:9000/index.json{"service":"internal-admin","ok":true}Read -L 9000:127.0.0.1:8081 as: "Listen on my local port 9000. Send whatever arrives, through the SSH connection, to 127.0.0.1:8081 as seen from the server." The words after the colon are resolved on the server side. That is the part people miss: 127.0.0.1 there means the server's own loopback, not your laptop. -N means "do not run a command, just forward", and -f means "go to the background after connecting".
ss shows the proof: a local ssh process is now listening on 9000.
If the local port is already in use, the tunnel fails. On my second try I got:
channel_setup_fwd_listener_tcpip: cannot listen to port: 9000Could not request local forwarding.A previous background tunnel was still running. Find it with ss -ltnp 'sport = :9000' and stop that process, or pick another local port.
pkill -f can kill your own shell. To stop the tunnel I ran pkill -f 'ssh -F config.*-L 9000'. The pattern -f matches the full command line of every process, and my own shell command line contained that same text, so pkill killed the shell running it (the tool reported exit code 144). Use the process id instead: ssh -f leaves a normal process, so find it with pgrep -a ssh or ss -ltnp, then kill PID. Or use a control socket (-M -S path, then ssh -S path -O exit host, reviewed, not run).
Other forwards (reviewed, not run)
- Remote forward,
-R 8080:127.0.0.1:3000: the server listens on 8080 and sends traffic back to port 3000 on your machine. People use it to show a local dev site to someone else. By default the server listens only on its own loopback unlessGatewayPortsis set. I did not run this. - Dynamic forward,
-D 1080: SSH becomes a SOCKS proxy on your port 1080, so a browser can send all traffic through the server. I did not run this.
Both exist because SSH is a general encrypted pipe. The manual page for ssh_config documents LocalForward, RemoteForward and DynamicForward as config-file equivalents.
Jump hosts: ProxyJump
In many companies the servers that matter have no public address. You first SSH to one public "bastion" (jump) server, and from there to the others. Typing two commands is clumsy. ProxyJump (or -J) does it in one.
My inner block had ProxyJump web. Here the "inner" server is the same VM again, but the connection really did go client, then web, then inner:
$ ssh -F config2 inner 'echo via-jump $(whoami)'via-jump deployInside, SSH starts a second SSH connection to the jump host and asks it to forward the data. Your private key stays on your laptop; it is not copied to the jump host. That is much safer than the old way of putting your key on the bastion.
A nested connection does not get all your command-line -o options. My first attempt used ssh -F config -o UserKnownHostsFile=... -J web host. It failed with Host key verification failed. The jump step ran its own ssh without those -o options. When I moved UserKnownHostsFile and StrictHostKeyChecking into the config file under Host * (as shown earlier) it worked. The lesson: put settings in the config file, not in per-command flags, when you use jump hosts. I found this out by testing; I did not find it stated in the pages I read, so treat it as what I saw on 9.6p1.
ssh-agent: type your passphrase once
If your key has a passphrase (it should), typing it for every connection is annoying. ssh-agent is a small background program that holds your decrypted keys in memory. You add a key once with ssh-add, and every later ssh asks the agent.
$ eval $(ssh-agent -s) # starts it, sets two environment variables$ ssh-add id_demoIdentity added: id_demo (demo@lab)$ ssh-add -l256 SHA256:K4lfPANmXMsDRv1FcSgmzTgvhaIdwCIFdjC4xmPPDac demo@lab (ED25519)ssh-add -l lists the fingerprints the agent holds. This is also how you know which key ssh is offering. (My lab key had no passphrase, so no prompt appeared; with a passphrase, ssh-add asks once.)
Do not use "agent forwarding" (ForwardAgent yes or ssh -A) to servers you do not fully trust. It lets the remote machine ask your local agent to sign things, as you, while you are connected. Anyone with root on that server can use it. ProxyJump is the safer way to hop through a bastion. This is advice from the manual's warnings about the feature; I did not run it.
The small network tools
When something does not connect, these tools tell you where it breaks. Use them in the order of the journey: name, route, port, application.
curl: talk HTTP from the shell
curl makes a web request and prints the answer. A few flags will cover most of your needs:
$ curl -s http://127.0.0.1:9000/index.json{"service":"internal-admin","ok":true}$ curl -sS -o /dev/null -w 'code=%{http_code} size=%{size_download} connect=%{time_connect}s total=%{time_total}s\n' http://127.0.0.1:9000/index.jsoncode=200 size=39 connect=0.000200s total=0.001811s$ curl -sI http://127.0.0.1:9000/index.json | head -5HTTP/1.0 200 OKServer: SimpleHTTP/0.6 Python/3.13.16Date: Mon, 12 Oct 2026 03:42:54 GMTContent-type: application/jsonContent-Length: 39$ curl -s -o /dev/null -w 'code=%{http_code}\n' http://127.0.0.1:9000/nopecode=404-sis silent (no progress bar).-Sshows errors even when silent. Together,-sSis the usual pair in scripts.-o /dev/nullthrows away the body.-wprints a template after the request;%{http_code}and%{time_total}are the two most useful variables.-Isends aHEADrequest and prints only the headers.-vshows the full conversation. A few key lines:
* Trying 127.0.0.1:9000...* Connected to 127.0.0.1 (127.0.0.1) port 9000> GET /index.json HTTP/1.1< HTTP/1.0 200 OK< Content-type: application/json< Content-Length: 39Lines beginning > are what you sent. Lines beginning < are what you received. Lines beginning * are curl's own notes.
Notice that a 404 is a successful connection. curl exits with 0 for both 200 and 404, because the network part worked. If you want a failing exit code on HTTP errors (like 4xx and 5xx), add -f (--fail). People forget this and their script says "success" while the server said "not found".
curl error codes
curl exit codes tell you what kind of failure it was:
$ curl -sS http://127.0.0.1:9999/ # nothing listens on 9999curl: (7) Failed to connect to 127.0.0.1 port 9999 after 0 ms: Couldn't connect to server# exit code 7$ nc -l 127.0.0.1 9100 & # a server that accepts but never answers$ curl -sS -m 2 http://127.0.0.1:9100/curl: (28) Operation timed out after 2002 milliseconds with 0 bytes received# exit code 28Two different problems, two different codes:
- 7: connection refused. The computer answered "nobody is listening on that port". The server is up, the program is not (or it is on a different port).
- 28: timed out. The connection hung. Either a firewall dropped your packets, the host is not reachable, or the server accepted the connection but never replied (that was my case).
Always set -m (max time) or --connect-timeout in scripts. By default curl can wait a very long time.
I wanted to show a "timeout from a firewall" by calling an unrouteable address like 10.255.255.1, but the sandbox proxy answered with Destination IP is in a private/reserved range instead, so that example was not run. The silent-server example above is real and gives the same exit code 28.
ss: what is listening, and who is connected
ss shows sockets. It is the modern replacement for netstat. The flags you want: -l listening, -t TCP, -n numbers (do not look up names), -p which process (needs root to see other users' processes).
$ ss -ltnp 'sport = :2222'State Recv-Q Send-Q Local Address:Port Peer Address:Port ProcessLISTEN 0 128 127.0.0.1:2222 0.0.0.0:* users:(("sshd",pid=12772,fd=4))This is the answer to "is my service really listening, and on which address?" Reading the Local Address is important:
127.0.0.1:2222means "only reachable from this same machine".0.0.0.0:2222means "reachable on every network interface".[::]:2222is the same idea for IPv6.
A very common bug: the app listens on 127.0.0.1 inside a container or VM, so nothing from outside can reach it even though the port is "open". ss shows this in one line.
A summary of connections by state (this lab VM was busy, the numbers vary):
$ ss -tan | awk 'NR>1{c[$1]++} END{for(k in c) print k,c[k]}'ESTAB 36LISTEN 12TIME-WAIT 102ESTAB is an open connection. TIME-WAIT is a connection that closed normally and is waiting a short time before the number can be reused; a big number is normal on a busy server and is rarely a problem by itself.
nc: is this port open?
nc (netcat) opens a plain connection. With -z it only checks whether the port accepts a connection:
$ nc -zv -w2 127.0.0.1 2222Connection to 127.0.0.1 2222 port [tcp/*] succeeded!$ nc -zv -w2 127.0.0.1 2223nc: connect to 127.0.0.1 port 2223 (tcp) failed: Connection refusedIt is a quick "can I reach that port" test, with no HTTP needed.
dig: ask DNS
dig asks a DNS server a question and shows the whole reply. dig +short name prints only the answer. In my sandbox there is no real internet DNS, so this is what a failed lookup looks like. It is still useful, because you will see this format when a domain does not exist:
$ dig example.com +time=2 +tries=1 | egrep 'status|SERVER';; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 62900;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)status: NXDOMAIN means "this name does not exist" (in my sandbox, it means the resolver there could not find it). SERVER: 8.8.8.8#53 shows which resolver answered, taken from /etc/resolv.conf. The real shape of a good answer is a line like example.com. 300 IN A 93.184.216.34 (name, TTL in seconds, class, type, address); I did not see that line myself in the sandbox, so I am describing it from the usual format and not from my output. Networking lesson 3 (DNS) goes deep on record types and caching.
Two important DNS facts for a developer:
digtalks to DNS directly. But your programs usually ask the system throughgetent hosts name, which also reads/etc/hostsfirst. They can give different answers. I checked:getent hosts localhostprinted127.0.0.1 localhost(from/etc/hosts), butdig localhostreturnedNXDOMAINbecause the DNS server knows nothing about it. Ifdigand your app disagree,getentshows what your app sees.- DNS answers are cached for their TTL. After you change a record, old answers can live on for a while.
ping and traceroute
$ ping -c 3 127.0.0.1 | tail -2--- 127.0.0.1 ping statistics ---3 packets transmitted, 3 received, 0% packet loss, time 2041msrtt min/avg/max/mdev = 0.040/0.048/0.058/0.007 ms$ traceroute -n -m 3 127.0.0.1traceroute to 127.0.0.1 (127.0.0.1), 3 hops max, 60 byte packets 1 127.0.0.1 0.080 ms 0.006 ms 0.005 msping checks whether a host answers a tiny message (ICMP) and how long a round trip takes (here 0.05 milliseconds, because it is the same machine). traceroute lists the routers between you and the target. These only worked against the loopback address in my sandbox, so what you see above is only the format. On the real internet, a router or firewall may drop ping, so "no ping reply" does not prove a host is down. Use nc -z or curl on the actual port to test the service itself.
A debugging order that works
When "I cannot reach the server", ask the questions in this order. Each tool answers one.
| Question | Tool | What a bad answer means |
|---|---|---|
| Does the name resolve? | getent hosts name, dig +short name | No answer: DNS or /etc/hosts problem |
| Can I reach the port? | nc -zv host port | Refused: nothing listening. Timeout: firewall or routing |
| Is something listening there? (on the server) | ss -ltnp | Not in the list: the service is not running, or on another port |
| Which address is it bound to? | ss -ltn | 127.0.0.1 only: not reachable from outside |
| Does the app answer correctly? | curl -v | 5xx or 4xx: the network is fine, the app is not |
| Why was SSH refused? | ssh -v, then the server log | Key, permission, account or AllowUsers rules |
Try it
You need a Linux machine with ssh, ssh-keygen and curl. You do not need a second server; use your own laptop as both ends (or ask a friend for a server).
- Make a new ed25519 key with a passphrase. Which file is the one you can share? What are its permissions?
- Write a
~/.ssh/configblock calledlabwith aHostName,User,PortandIdentityFile. Runssh -G laband find your four values in the output. - Run
python3 -m http.server 8081 --bind 127.0.0.1. Usess -ltnpto prove it listens only on loopback. Usecurl -sIto see its headers. - Copy a folder
a/with two files twice, usingrsync -avto a second folder. What does the second run print? Then add a file and run with-n. - Run
curl -sS http://127.0.0.1:9999/. What is the exit code? What would be different if the port were open but the server silent? - Explain, in one sentence, what
-L 9000:127.0.0.1:8081forwards to which127.0.0.1.
Answers
- The
.pubfile is the shareable one, usually mode 644. The private key has no extension and must be mode 600. ssh -G labprints lines likehostname ...,port ...,user ...,identityfile ...in lower case.ssshould show127.0.0.1:8081, not0.0.0.0:8081. The headers includeHTTP/1.0 200 OKand aContent-typeline.- The second
rsync -avprints no file names, only the sent/received summary. With-nand a new file it lists only that file. - Exit code 7 (connection refused). With a silent server and
-m 2you get exit code 28 (timed out). - Local port 9000 is forwarded to port 8081 on the
127.0.0.1of the remote server (the loopback seen by the server), not of your laptop.
Interview questions
How does SSH key authentication work? (Must know)
I generate a key pair. The private key stays on my machine. The public key goes in ~/.ssh/authorized_keys on the server. When I connect, the server sends a challenge that only the owner of the matching private key can answer, my client signs it, and the server checks the signature with the public key. The private key is never sent over the network. I also protect the private key file with a passphrase and mode 600, and I use ssh-agent so I only type the passphrase once.
What is the difference between the public and private key, and what happens if the private key leaks? (Must know)
The public key can be shared; anyone can put it on a server. The private key proves identity, so it must stay secret. If it leaks, anyone who has it (and its passphrase, if one was set) can log in anywhere that trusts that public key. I would remove the matching line from every authorized_keys, make a new pair, and check server logs for logins I do not recognise. This is also why I never put a private key in a repository or a container image.
"SSH says Permission denied (publickey)". How do you debug it? (Must know)
I run ssh -v first to see which key is offered and whether the server accepts publickey at all. Then I check the usual causes: wrong user name, wrong key file or IdentitiesOnly hiding the right key, private key permissions too open, the public key not in authorized_keys, or ~/.ssh and authorized_keys having the wrong permissions or owner. If the client side tells me nothing, I read the server log. In my lab it told me the exact reasons: bad ownership or modes for directory and account is locked.
What does knownhosts protect you from, and what does the "host identification has changed" warning mean? (Must know)
It remembers the host key of every server I have visited, so on later visits the client can tell whether I am talking to the same server. A changed key can be harmless (the server was reinstalled) or an attack (someone is in the middle). I do not just delete the line. I confirm with whoever owns the server, then remove the old entry with ssh-keygen -R.
What is the difference between scp and rsync? When do you choose each? (Must know)
scp copies the whole file every time. rsync compares both sides and sends only the differences, can resume, can mirror with --delete, and has a dry-run mode -n. For a one-off small file I use scp. For deploys, backups or big folders I use rsync. I also remember the trailing slash rule: src/ copies the contents, src copies the folder itself.
Explain ssh -L 9000:127.0.0.1:8081 host. (Must know)
It opens local port 9000 on my machine. Connections to it travel through the encrypted SSH link to host, and from there to 127.0.0.1:8081 as seen from host. So I can open http://localhost:9000 and reach a service that only listens on the server's loopback and is not exposed to the internet. In my lab I confirmed it with ss (a local ssh listening on 9000) and curl.
How would you reach a database server that is only available from inside a private network? (Senior signal)
I would use a bastion (jump host) with ProxyJump in my SSH config, and a local forward to the database port, so my laptop talks to localhost:5432 and SSH carries it through. I would not open the database to the internet. I would avoid agent forwarding because root on the bastion could use my agent. With ProxyJump the private key stays on my laptop. A trap I found: settings given only as command-line -o options may not be applied to the jump step, so I keep them in the config file.
A service is "up" but another machine cannot connect. What do you check? (Trap)
First, ss -ltnp on the server: which address is it bound to? If it says 127.0.0.1:PORT, it only accepts local connections, so remote clients fail even though the port is "open". Then I test from the client with nc -zv host port. "Connection refused" means nothing is listening on that address and port. A timeout points to a firewall or security group. The trap is checking only that the process is running.
What do curl exit codes 7 and 28 mean? And what does curl return for a 404? (Trap)
7 is "could not connect" (refused: no listener). 28 is "timed out" (no reply, or a firewall dropping packets). For a 404 or 500, curl exits with 0 by default because the HTTP exchange itself worked. To make curl fail on HTTP errors I use -f. In scripts I also add -sS and always set -m so it cannot hang forever.
Why disable password login and root login in sshdconfig? (Senior signal)
Password logins can be guessed, and public SSH ports are scanned and attacked all the time. With PasswordAuthentication no only keys work. PermitRootLogin no means an attacker has to compromise a normal user first, and every admin action is tied to a named user through sudo. I test the config with sshd -t and keep a second session open when I change it, so I cannot lock myself out.
ping gets no reply. Is the server down? (Trap)
Not necessarily. Many networks and firewalls drop ICMP, the protocol ping uses. The better test is to connect to the real port: nc -zv host 443 or curl -v. If that works, the service is fine. If it fails, then compare "refused" (nothing listening) with "timeout" (blocked or unreachable).
dig shows the right IP but my program still connects to the wrong one. Why? (Senior signal)
dig asks a DNS server directly. A program usually goes through the system resolver, which may read /etc/hosts first, use a local cache, or use different DNS servers. I check getent hosts name to see what the system gives the program. In my lab getent hosts localhost found it in /etc/hosts while dig localhost said NXDOMAIN. Programs can also cache a DNS answer for a long time, for example a long-lived connection pool, so after a DNS change they may need a restart.
Key takeaways
- SSH checks the server first (
known_hosts), then you (key inauthorized_keys). The private key never leaves your machine; set a passphrase and mode 600. - When login fails with a vague "Permission denied", read the server's log. I got
bad ownership or modesandaccount is lockedthere. - Put server names in
~/.ssh/config, useIdentitiesOnly yes, and debug withssh -G(show settings) andssh -v(show steps). scpcopies;rsynccopies only differences. Remembersrc/versussrc, and use-nbefore--delete.-L local:host:portforwards through the server, andhostis resolved on the server. UseProxyJumpfor bastions, not agent forwarding.curl: 7 is refused, 28 is a timeout, and a 404 still exits 0 unless you use-f.ss -ltnpshows what listens and on which address. Loopback-only is a very common reason a "running" service cannot be reached.- Do not trust
pingalone. Test the real port.
What's next
In lesson 6 you will manage what runs on the server you can now reach: installing software with apt, running your app as a systemd service that restarts when it crashes, reading its logs, and scheduling jobs with cron and timers. Practise now: set up key login to a machine you own, write a Host block for it, and use ss on the server to see exactly which ports it is listening on.
Published by Jainik · Last updated · How these lessons are made
Keep learning
Related lessons
- What Happens When You Type a URL (The Full Version) · The Browser Contract
- Volumes, Networks and Running Containers Safely · Docker & Kubernetes
- Services, Ingress and Networking in Kubernetes · Docker & Kubernetes
Part of the study guide
Related series