Disks, Memory, Logs and Troubleshooting a Slow Server

A message arrives: "the site is slow". Or worse: "the site is down". You SSH in (lesson 5). Now what? Many people type random commands and hope. Calm engineers follow a short routine, and they know a few traps that make the numbers lie: a disk that is 100% full while du finds nothing, a "free memory" number that looks scary but is fine, a load average that looks high on a computer with many cores.

This is lesson 7 of nine. In lesson 6 you learned how services and cron jobs run. Here we find out why they misbehave. The last lesson, Linux interview questions, uses these ideas as questions. For process basics (ps, states, signals), see lesson 2, and for searching text see lesson 3. For the bigger world of dashboards and alerts, Observability lesson 0 shows what teams build so they need to SSH in less often.

Words used in this lesson

• CPU is the part that runs instructions. Core is one CPU worker; this VM has 4.
• Load average is the average number of tasks that are running or waiting to run (or, on Linux, waiting on disk), shown for 1, 5 and 15 minutes.
• RAM is fast memory. Swap is disk space used as slow extra memory.
• Page cache is memory the kernel uses to keep recently read file data, so reading again is fast. It can be given back when programs need it.
• RSS (resident set size) is how much real RAM a process is using right now.
• Virtual memory (VSZ) is the address space a process asked for; much of it may never be used.
• OOM killer is the kernel part that kills a process when memory runs out. OOM means out of memory.
• Inode is the record that stores a file's details (owner, size, where its data is). A file system has a fixed number of them.
• File system is the structure that stores files on a disk, or in RAM for tmpfs.
• I/O wait is time the CPU spends idle because it is waiting for the disk.
• Saturation means a resource is so busy that work waits in a queue.
• USE method is a checklist: for each resource look at Utilisation, Saturation and Errors.
• Log is a text record of events. Log rotation means renaming and compressing old logs so they do not fill the disk.
• Sparse file is a file that claims a large size but has not actually used the space.

A method before the commands

Fast fixers do not know more commands. They ask the same questions every time. A simple way to do that is the USE method, which I know from performance-engineering writing (I could not open the original page, so this is my own summary): for every resource, ask

1. Utilisation: how busy is it (a percentage)?
2. Saturation: is work waiting in a queue for it?
3. Errors: is it reporting failures?

The resources on a server are mostly four: CPU, memory, disk (space and speed), network. That makes a grid of about a dozen quick checks, and it stops you from staring at the CPU when the real problem is a full disk.

Then add the application itself: its error rate and its logs. A server can have healthy CPU, memory and disk, and still be slow because a database query is slow, which no OS tool will show you. The OS tools tell you whether to look at the machine or at the app.

Step 1: how loaded is it? uptime and load average

$ uptime 03:46:37 up 43 min,  0 user,  load average: 11.64, 6.89, 4.25$ nproc4$ cat /proc/loadavg11.64 6.89 4.25 12/278 24128

The three numbers are the load averages for the last 1, 5 and 15 minutes. At that moment I had two while :; do :; done loops running (I started them to make load), plus the other jobs on this shared VM.

How to read load:

• Compare it with the number of cores (nproc = 4). A load of 4 on 4 cores means "fully busy but nothing is waiting". A load of 11.64 on 4 cores means roughly three times more work than the CPUs can do, so tasks are queueing.
• Compare the three numbers. 11.64, 6.89, 4.25 is rising: the 1-minute number is higher than the 15-minute. 1.0, 5.0, 9.0 would mean it is falling: the busy time has passed.
• The file /proc/loadavg has the same three numbers, then running/total tasks (12/278) and the last process id.

Step 2: who is using the CPU? ps, top and vmstat

$ ps -eo pid,ppid,stat,%cpu,%mem,etime,comm,args --sort=-%cpu | head -4  PID  PPID STAT %CPU %MEM     ELAPSED COMMAND         COMMAND23979 23967 R    89.3  0.0       00:06 spin.sh         /bin/bash ./spin.sh23978 23967 R    83.4  0.0       00:06 spin.sh         /bin/bash ./spin.sh23638 13097 Rl   77.3  3.4       00:13 compile         /root/go/pkg/mod/golang.org/toolchain@.../compile @/tmp/args1669152048

-eo chooses which columns to show, and --sort=-%cpu sorts with the biggest first. The STAT column: R is running, S is sleeping, D is waiting on disk (uninterruptible), Z is a zombie (lesson 2). Two spin.sh processes, each near 90% of one core, are the runaway culprits. ELAPSED shows they have only run 6 seconds: a process that burns CPU for hours is a different story from one that started a minute ago.

top gives a live view. In batch mode (-b -n1) it prints one snapshot, which is good for pasting into a ticket:

$ top -b -n1 | head -8top - 03:46:39 up 43 min,  0 user,  load average: 11.83, 7.01, 4.30Tasks: 120 total,   5 running, 115 sleeping,   0 stopped,   0 zombie%Cpu(s): 93.5 us,  4.8 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  1.6 stMiB Mem :  16093.2 total,   4340.2 free,   2648.0 used,   9449.3 buff/cacheMiB Swap:      0.0 total,      0.0 free,      0.0 used.  13445.1 avail Mem

The %Cpu(s) line is where the story is:

• us is time in user programs. sy is time in the kernel. id is idle. wa is waiting for I/O. st is "steal", time a virtual machine wanted but the host gave to others.
• Here us is 93.5% and id is 0.0%: the CPU is the bottleneck, and it is user code.
• If wa were high and us low, the disk would be the suspect. If st is high on a cloud VM, a neighbour is using the physical machine ("noisy neighbour") and no tuning inside your VM fixes it.

vmstat: one line per second

$ vmstat 1 3procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu------- r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu10  0      0 4609896 346260 9281200    0    0  1254  5623 5654   21 23  8 58 10  1  012  0      0 4506488 346272 9284208    0    0     0     0 3023 4437 94  5  0  0  1  013  0      0 4459572 346292 9306028    0    0 19880     8 4438 5404 93  6  0  0  0  0

The first line is an average since boot, not the current state. Ignore it; read the lines after it. What matters:

• r is the number of tasks waiting to run. 12 or 13 on 4 cores: the run queue is long, so the CPU is saturated. A healthy value is at most about the number of cores.
• b is tasks blocked on I/O.
• si and so are memory swapped in and out. Anything but zero here, sustained, means the machine is short of RAM (I have no swap, so both are 0).
• us, sy, id, wa: the same split as top.

If r is high and id is zero, you have CPU saturation. If r is small but wa and b are high, look at the disk.

A runaway process: what to do

You found a process using 100% of a core. Before you kill it, ask: is it supposed to be doing that? A build or a video conversion is. A stuck loop is not. Gather facts first:

ps -o pid,ppid,etime,stat,args -p 23979     # how long, who started itls -l /proc/23979/cwd /proc/23979/exe       # where it runs from, what it is

Then be polite: send SIGTERM (kill PID) and wait; only if it ignores that, use SIGKILL (kill -9). I did this in the lab, and the loops ended; the load average began to fall. If the process is a service, fix the cause (a bug, a bad input), because a restart without a fix will repeat. For why SIGTERM comes first, see lesson 2.

Step 3: memory. free and the "free is not low" confusion

$ free -h               total        used        free      shared  buff/cache   availableMem:            15Gi       3.1Gi       3.6Gi        14Mi       9.3Gi        12GiSwap:             0B          0B          0B

Beginners look at the free column: 3.6 GB free out of 15 GB. "Only a quarter is free!" That is the wrong column. Linux uses spare RAM to keep file data in the page cache (buff/cache, here 9.3 GB), because reading from RAM is much faster than from disk. When a program needs memory, the kernel gives cache back. So the useful number is available: an estimate of how much memory could go to new programs without swapping. Here that is 12 GB, which is healthy. In /proc/meminfo the same figures appear:

$ egrep '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|SwapTotal|Dirty)' /proc/meminfoMemTotal:       16479424 kBMemFree:         3814164 kBMemAvailable:   13199888 kBBuffers:          347080 kBCached:          8720996 kBSwapTotal:             0 kBDirty:            600276 kB

MemAvailable is the line to check. Dirty is file data changed in memory and not yet written to disk. I read the kernel's proc.rst for the meaning of these fields; it defines MemAvailable as an estimate of memory available to start new applications without swapping.

Which process uses the memory?

$ python3 hog.py &        # allocates and touches 400 MB, then sleeps$ grep -E 'VmSize|VmRSS|RssAnon' /proc/$!/statusVmSize:   423940 kBVmRSS:    418360 kBRssAnon:  412492 kB$ ps -eo pid,rss,comm --sort=-rss | head -4  PID   RSS COMMAND   97 499976 claude24705 418360 python3 5112 251892 dkapi

RSS is the real RAM a process holds (418 MB for my 400 MB test, plus Python itself). VSZ/VmSize is larger when a program reserves address space it has not used yet, so do not judge memory by VSZ. Sort by RSS. free -m before and after showed used going from 2111 to 2379 MB while the process ran, and back down to 1826 MB after it exited.

One more number is in /proc/PID/oom_score (I saw 683 for this process, and oom_score_adj was 0). When memory runs out, the OOM killer chooses a victim; a higher score means "more likely to be killed", and it grows with the memory a process uses. oom_score_adj lets you protect or sacrifice a process on purpose. I read this in the kernel's proc.rst.

What running out of memory looks like

I did not make the kernel kill anything. What I did run: a memory limit with ulimit -v 200000 (about 200 MB) and then asked Python for 400 MB:

$ (ulimit -v 200000; python3 -c "b=bytearray(400*1024*1024)")MemoryError

That is the application-level shape of "not enough memory": the allocation fails and the program reports an error or crashes. A real kernel OOM kill is different and silent from the program's point of view: the process just vanishes, with exit status 137 (128 + signal 9, lesson 2), and a line appears in the kernel log. You look for it like this (reviewed, not run, because I had no OOM kill to find):

dmesg -T | grep -i -E 'out of memory|killed process'journalctl -k | grep -i 'killed process'

Step 4: disk space. df, du and the three ways "full" lies

"No space left on device" is one of the most common production errors. And the cause is not always "too many files". I made a small 16 MB RAM file system, filled it, and saw three different kinds of "full".

Kind 1: really out of space

$ mount -t tmpfs -o size=16m,nr_inodes=200 tmpfs mnt$ df -h mntFilesystem      Size  Used Avail Use% Mounted ontmpfs            16M     0   16M   0% /tmp/lab-linux-b/disk/mnt$ dd if=/dev/zero of=mnt/big.log bs=1M count=100dd: error writing '/tmp/lab-linux-b/disk/mnt/big.log': No space left on device17+0 records in16+0 records out$ echo hi > mnt/x.txtbash: echo: write error: No space left on device$ df -h mnt | tail -1tmpfs            16M   16M     0 100% /tmp/lab-linux-b/disk/mnt

df ("disk free") reports per file system: size, used, available and the mount point. Even a tiny echo hi > file fails now. Typical symptoms on a real server: the database cannot write, the app cannot create a log or a temp file, SSH sessions act strange, package installs fail. A very common culprit is a log that grew without limit.

Now find where the space went. du ("disk usage") adds up sizes. This is the pattern I use:

$ du -h --max-depth=1 mnt/app | sort -h2.0M	mnt/app/uploads4.0M	mnt/app/cache29M	mnt/app/logs36M	mnt/app$ find mnt -type f -size +1M -printf '%s %p\n' | sort -rn | head -3 | numfmt --to=iec --field=120M mnt/app/logs/app.log9.0M mnt/app/logs/old.log.14.0M mnt/app/cache/c1

(That run used a separate 64 MB tmpfs so there was something to measure.) Work top down: du -h --max-depth=1 / (add -x to stay on one file system), go into the biggest folder, repeat. Then find ... -size +100M for single big files. sort -h sorts human-readable sizes (2.0M, 29M) correctly; plain sort -n does not.

Kind 2: free inodes ran out (space is free, but you still cannot write)

On a real disk, each file uses one inode. A file system is created with a fixed number of them. If you create a very large number of tiny files (session files, cache entries, mail), you can use all the inodes while the disk is nearly empty. I made a tmpfs with only 200 inodes and a loop that creates empty files:

$ for i in $(seq 1 250); do : > mnt/logs/f$i 2>/dev/null || { echo "failed at file $i"; break; }; donefailed at file 199$ df -h mnt | tail -1tmpfs            16M     0   16M   0% /tmp/lab-linux-b/disk/mnt$ df -i mnt | tail -1tmpfs             200   200     0  100% /tmp/lab-linux-b/disk/mnt$ touch mnt/logs/newtouch: cannot touch 'mnt/logs/new': No space left on device

Look at that carefully. df -h says 0% used, 16M free, and yet touch fails with the same message "No space left on device". The second command, df -i, shows 100% of inodes used. (It stopped at 199 files because the root directory and logs folder used the first inodes.) When "no space left" appears but df -h shows free space, run df -i. The fix is to delete the pile of tiny files, then find out which program makes them.

Kind 3: you deleted a big file, but the space did not come back

This one confuses almost everyone. A process has a log file open and writes to it. You see the disk full and run rm app.log. df still says full. Why? On Linux, rm removes the name of the file. The data stays on disk until no process has it open. Let me show it with a process holding a 12 MB file open:

$ dd if=/dev/zero of=mnt/held.log bs=1M count=12$ (sleep 600 < mnt/held.log &)       # a process that keeps the file open$ df -h mnt | tail -1tmpfs            16M   12M  4.0M  75% /tmp/lab-linux-b/disk/mnt$ rm mnt/held.log$ df -h mnt | tail -1tmpfs            16M   12M  4.0M  75% /tmp/lab-linux-b/disk/mnt$ du -sh mnt0	mnt

After rm, df still says 12M used, but du says 0. du walks file names, so it cannot see a deleted file. df asks the file system how many blocks are in use, so it can. If df and du disagree by a lot, suspect a deleted-but-open file. Find it:

$ lsof +L1 | grep held$ ls -l /proc/$(cat held.pid)/fd | grep deletedlr-x------ 1 root root 64 Oct 12 03:46 0 -> /tmp/lab-linux-b/disk/mnt/held.log (deleted)

lsof +L1 lists open files with a link count below one (deleted but still open). (Careful: on this busy sandbox it also listed unrelated deleted files from other jobs, so I filter by name.) The /proc/PID/fd folder shows each file the process has open, and the (deleted) tag gives it away. After I stopped the process that held it, the space came back:

$ kill $(cat held.pid)$ df -h mnt | tail -1tmpfs            16M     0   16M   0% /tmp/lab-linux-b/disk/mnt

The Art of Command Line README lists the same trick (lsof | grep deleted).

The fix in production is either to restart the process that holds the file (the clean way), or to truncate instead of deleting. Truncating keeps the same file but makes it empty:

$ (sleep 300 >> mnt/app/logs/app.log &)      # a process that keeps appending$ : > mnt/app/logs/app.log                    # empty it, do not remove it$ df -h mnt | tail -1tmpfs            64M   16M   49M  24% /tmp/lab-linux-b/disk/mnt

Used space fell from 36 MB to 16 MB at once, because the file was emptied in place. : > file and truncate -s 0 file do this. (One caution: a program that opened the file without append mode may keep writing at its old position and create a sparse file, so for real services prefer log rotation with the right options.)

A bonus: sparse files

$ truncate -s 500M mnt/app/sparse.img$ ls -lh mnt/app/sparse.img | awk '{print $5, $9}'500M mnt/app/sparse.img$ du -h mnt/app/sparse.img0	mnt/app/sparse.img

ls says 500 MB, du says 0, and df did not change. A sparse file has a large nominal size but no blocks yet. VM disk images and database files are often sparse. So ls -l size is not the same as disk use. When in doubt about space, trust du and df.

Where space usually goes on a server

• Logs under /var/log or the application's own folder, or the systemd journal.
• Docker images, stopped containers and volumes (docker system df shows sizes; see Docker lesson 2). I did not run Docker commands in this lesson.
• Package caches (/var/cache/apt/archives was only 8 KB here).
• Core dumps, temp files, old backups, uploads.
• Uploads or data that your app writes and nobody ever deletes.

Step 5: disk speed. iostat

If space is fine but things are slow, the disk may be too busy. iostat -dx 1 (from the sysstat package) prints per-disk statistics. I ran iostat -dx 1 2 and kept the second report. It is very wide, so here are the interesting columns for the busy device vda:

Device   r/s     rkB/s    r_await  rareq-sz  w/s   wkB/s  w_await  aqu-sz  %utilvda      279.00  14296.00   0.20     51.24   0.00   0.00    0.00    0.06    6.40vdb        0.00      0.00   0.00      0.00   0.00   0.00    0.00    0.00    0.00

• r/s and w/s are reads and writes per second. rkB/s is the read throughput.
• r_await and w_await are the average time (in milliseconds) a request takes, including waiting in the queue. This is the "latency" number. A few milliseconds is normal on SSD; hundreds is trouble.
• aqu-sz is the average queue length. Over about 1 per device means requests are waiting (saturation).
• %util is the share of time the device was busy. 6.4% here, so the disk was fine. Note that on modern SSDs or NVMe drives that handle many requests at once, 100% util does not always mean "slow"; watch await and aqu-sz too.

The pattern for a disk-bound server: top shows low us and high wa; vmstat shows b greater than zero; iostat shows high await and %util. If iostat shows nothing but things are still slow, go back to the application.

Step 6: the network, quickly

You learned the tools in lesson 5. For a "slow" report, the fast checks are:

ss -s                       # total sockets, how many in each statess -tan | awk 'NR>1{c[$1]++} END{for(k in c) print k,c[k]}'ss -ltnp                    # what listens, and on which addresscurl -sS -o /dev/null -w 'dns=%{time_namelookup} connect=%{time_connect} first-byte=%{time_starttransfer} total=%{time_total}\n' URL

The curl -w line is a great slow-request detector: it splits the time into DNS lookup, TCP connect, waiting for the first byte (the server thinking) and total. If connect is quick but starttransfer is slow, the server is slow to answer; if time_namelookup is large, DNS is slow. On my loopback test everything was around 0.0001 seconds (dns=0.000020 connect=0.000000 start=0.000000 total=0.000107), so it only shows the shape. A large number of CLOSE-WAIT sockets usually means an application forgot to close connections. A huge number of TIME-WAIT is normal on a busy server (I saw 102 on this VM).

Step 7: logs. Where they are and how to search

Logs say what the system was doing. The main places:

On this VM ls /var/log showed files such as alternatives.log, apt, bootstrap.log, dpkg.log (601 KB), and others: package-manager logs from my apt runs. There was no syslog, because there is no regular logging service in the sandbox.

With systemd: journalctl (reviewed, not run)

journalctl -u myapp --since "10 min ago"      # one service, recentjournalctl -p err -b                           # errors and worse, this bootjournalctl -u myapp -f                         # followjournalctl --disk-usage                        # how much space the journal usessudo journalctl --vacuum-size=500M             # trim the journal to 500 MB

My attempt in the sandbox printed No journal files were found. and -- No entries --, because there is no journal. These commands come from the manuals and my experience; I could not run them here.

Without systemd: files and tail -F

tail -f follows a file. tail -F (capital F) follows the file by name and keeps working after the file is rotated (renamed and replaced), which is almost always what you want:

$ timeout 2 tail -n0 -F live.log &$ echo "line2" >> live.logline2

Answering questions from an access log

I generated a synthetic web access log, 2,000 lines in the common "combined"-like layout with a response time at the end:

10.0.0.2 - - [12/Oct/2026:10:00:00 +0000] "GET /api/users HTTP/1.1" 200 8979 0.13010.0.0.3 - - [12/Oct/2026:10:07:01 +0000] "GET / HTTP/1.1" 200 814 0.182

The fields (counting by spaces): 1 is the client IP, 7 is the path, 9 is the status code, and the last one ($NF) is the time in seconds. With awk, sort and uniq (see lesson 3) you answer real questions in seconds.

How many of each status code?

$ awk '{print $9}' access.log | sort | uniq -c | sort -rn   1548 200    145 500    117 304     72 502     64 404     54 504

Which paths return server errors (5xx)?

$ awk '$9>=500{print $7}' access.log | sort | uniq -c | sort -rn     84 /api/report     64 /     39 /api/users     38 /api/orders     32 /static/app.js     14 /login

Which path is slowest on average?

$ awk '{s[$7]+=$NF; n[$7]++} END{for(p in s) printf "%-16s %5d req  avg %.3fs\n", p, n[p], s[p]/n[p]}' access.log | sort -k5 -rn | head -3/api/report        113 req  avg 3.440s/                  608 req  avg 0.101s/login             104 req  avg 0.099s

Now the data points at one endpoint: /api/report is the slowest (3.4 seconds against 0.1) and has the most 5xx. I built the data that way on purpose, so in a real case you would still need to confirm the cause, but the method is the same: count by status, group by path, and compare times. The percentage with 504 also tells you the story: 504 means a gateway timed out waiting for the app.

Errors per hour:

$ awk '$9>=500{split($4,a,":"); print a[2]}' access.log | sort | uniq -c    102 10     77 11     92 12

If the count suddenly jumped at a specific hour, line it up with deploys and cron jobs.

Log rotation (reviewed, not run)

Logs that grow forever fill the disk. logrotate renames old logs (app.log becomes app.log.1, then app.log.2.gz), compresses them, deletes the oldest, and tells the program to reopen its file. It is driven by files in /etc/logrotate.d/. It was not installed on this VM, so I did not run it. Two things you must know:

• A program that keeps its file open must be told to reopen it after the rename (a signal, or the copytruncate option, which copies and then empties the original). Otherwise it keeps writing into the renamed file, and you are back to the "deleted but open" problem above.
• Apps in containers should usually write logs to standard output and let the platform handle them, not write files inside the container.

A first-two-minutes checklist

I turned these commands into one read-only script, so you can paste it on a server. It changes nothing:

#!/usr/bin/env bash# triage.sh: a read-only first look at a slow Linux boxecho "--- uptime / load (cores: $(nproc))"; uptimeecho "--- memory"; free -mecho "--- disk space"; df -h -x tmpfs -x devtmpfs 2>/dev/null | head -8echo "--- inodes"; df -i -x tmpfs -x devtmpfs 2>/dev/null | head -5echo "--- top CPU"; ps -eo pid,%cpu,%mem,stat,comm --sort=-%cpu | head -4echo "--- top memory"; ps -eo pid,%cpu,%mem,rss,comm --sort=-rss | head -4echo "--- cpu/io wait (vmstat)"; vmstat 1 2 | tail -1

shellcheck found nothing to complain about. Real output from this VM:

--- uptime / load (cores: 4) 03:47:44 up 44 min,  0 user,  load average: 5.75, 6.23, 4.22--- memory               total        used        free      shared  buff/cache   availableMem:           16093        1409        6234          14        8793       14683Swap:              0           0           0--- disk spaceFilesystem      Size  Used Avail Use% Mounted on/dev/vda        252G   38G  1.9G  96% //dev/vdc        273M  244M   24M  92% /opt/claude-code/dev/vdd         47M   31M   16M  67% /opt/env-runner--- inodesFilesystem       Inodes  IUsed    IFree IUse% Mounted on/dev/vda       16777216 789465 15987751    5% /...--- top CPU  PID %CPU %MEM STAT COMMAND25165 91.5  0.1 Rl   go   97 10.2  3.2 Rl   claude--- cpu/io wait (vmstat) 4  0      0 6321704 364504 8684592    0    0     0     0 7459 10021 17 12 71  0  1  0

Read this the way you would on a real incident. Load is 5.75 on 4 cores: a bit busy. Memory: 14.6 GB available, no swap, fine. Disk: / is 96% used with only 1.9 GB available, which is the thing to watch (the Size is 252G and Used is 38G, so most of that file system is reserved or used by other things; the Avail column is the number that matters for writes). vmstat: 71% idle, 0 wait. The inodes are fine at 5%. So on this machine, the most likely trouble spot would be disk space, not CPU or memory.

Here is the order I follow:

1. Is it the whole machine or one app? uptime, then top.
2. CPU: load against cores, vmstat r, then the top processes.
3. Memory: free -h (look at available), swapping, OOM lines in dmesg.
4. Disk space: df -h, then df -i, then du top-down, then lsof +L1 for deleted-but-open files.
5. Disk speed: iostat -dx 1, and wa in top.
6. Network: ss -s, curl -w timings.
7. Logs: the app's errors around the time it got slow.
8. What changed? A deploy, a cron job, a certificate, a traffic spike.
9. Fix, then verify, and write down what you found.

Try it

You need a Linux machine where you can use sudo. Use a VM, not a production box.

1. Run uptime and nproc. Is the load per core above 1?
2. Create a 20 MB RAM file system: mkdir m && sudo mount -t tmpfs -o size=20m tmpfs m. Fill it with dd if=/dev/zero of=m/big bs=1M count=100. What error do you get, and what does df -h m show?
3. Mount another with -o size=20m,nr_inodes=50 and create files in a loop until it fails. Compare df -h with df -i.
4. Keep a file open with (sleep 300 < m/big &), delete the file, and compare df with du. How do you find the holder? How do you free the space without restarting the process?
5. Run free -h. Which column tells you how much memory new programs can use? Explain in one sentence why free is small and it is still fine.
6. In any log with a status code, use awk and sort | uniq -c to count the codes.
7. Unmount when finished: sudo umount m.

Answers

1. A load per core near or above 1 means the CPUs are busy; well above 1 means queueing.
2. dd: error writing ... No space left on device, and df -h shows 100% used (20M of 20M).
3. df -h can show lots of free space, while df -i shows 100% inodes used. The failure message is the same "No space left on device".
4. After rm, df still shows the used space but du does not. Find the holder with lsof +L1 or ls -l /proc/PID/fd | grep deleted. Free the space by truncating the open file before deleting it (: > file), or by restarting the holder.
5. available. The kernel uses spare RAM for the page cache and gives it back when needed.
6. For example awk '{print $9}' access.log | sort | uniq -c | sort -rn for a common log format, where field 9 is the status.
7. sudo umount m (make sure no process is using it, or it says "target is busy").

Interview questions

Key takeaways

• Use a routine: CPU, memory, disk, network, then logs. For each, ask about utilisation, saturation and errors.
• Compare load average with the core count, read the trend, and remember Linux load includes tasks waiting on disk.
• In free, look at available, not free. Cache is not a problem.
• "Disk full" has three faces: out of space (df -h), out of inodes (df -i), and deleted-but-open files (df bigger than du, find with lsof +L1).
• Truncate an open log (: > file) instead of deleting it. Rotate logs, and use tail -F.
• vmstat shows the run queue (r), blocked tasks (b), swap (si/so) and the CPU split (wa, st). Ignore its first line.
• curl -w splits a slow request into DNS, connect, first byte and total time.
• awk | sort | uniq -c turns a log into answers. Averages hide the slow tail, so use percentiles in real monitoring.
• Not run here: journalctl (no systemd), a real OOM kill, logrotate. The sandbox was shared, so CPU and load numbers had outside noise.

What's next

The last lesson, Linux and Shell Interview Questions With Model Answers, collects the questions that come up again and again from lessons 0 to 7 and adds the traps: what is a hard link, why set -e can surprise you, what a zombie is, and more. Practise now: run the triage script on a machine you own and write down, in three lines, what you think is the weakest resource.