Processes, Signals and Job Control

When you start node server.js, the program file on disk becomes a running thing in memory. That running thing is a process. When something goes wrong on a server, you almost always end up asking: which process is it, who started it, is it alive, is it using the CPU, and how do I stop it safely? This lesson answers all five with real commands.

You have seen users and permissions in lesson 1. Every process runs as a user, and that is why the permission rules apply to programs. After this lesson, Pipes, Redirection and Text Tools shows how to connect processes together.

Words used in this lesson

• Process is a running instance of a program, with its own memory, a PID and an owner.
• PID is the process ID, a number. PPID is the PID of its parent.
• Parent and child: a process creates another by forking. The creator is the parent. The new one is the child. A shell is the parent of every command you run.
• PID 1 is the first process started by the kernel. It is the ancestor of all others. On a normal server it is systemd. In a container it is usually your app.
• Signal is a small message the kernel delivers to a process, like SIGTERM. The process can handle some signals in its own code.
• Exit code (or exit status) is a number 0 to 255 that a process returns when it ends. 0 means success.
• Foreground job owns your terminal. A background job runs while you keep typing.
• Job control is the shell's feature for running, pausing and switching jobs: &, Ctrl-Z, jobs, fg, bg.
• Daemon is a long-running background process, usually a service.
• Zombie is a finished process whose parent has not yet collected its exit code. Orphan is a running process whose parent has died.
• Process group is a set of processes that can be signalled together, such as one pipeline.
• Niceness is a number from -20 to 19 that hints how willing a process is to give up the CPU. Higher means "nicer" (lower priority).
• Load average is a count of how many processes want to run (or wait for disk), averaged over 1, 5 and 15 minutes.

What a process is

A program is a file on disk, such as /usr/bin/sleep. A process is that program running. You can run the same program ten times and get ten processes with ten different PIDs.

Each process has, among other things:

• a PID and a parent PID (PPID),
• the user and group it runs as (from lesson 1),
• a working directory and an environment (the variables from lesson 0),
• a set of open files (including your terminal as file number 0, 1 and 2),
• a state (running, sleeping, stopped, zombie).

The kernel shows all this as files under /proc/<PID>/. Let me start a background process and look. (The & at the end of a command starts it in the background, and $! holds the PID of the last background job.)

sleep 30 &SP=$!echo "bg pid=$SP"ps -o pid,ppid,stat,cmd --ppid $$

bg pid=24938  PID  PPID STAT CMD24938 24919 S    sleep 3024939 24919 R    ps -o pid,ppid,stat,cmd --ppid 24919

The sleep has PPID 24919, which is my shell ($$ is the shell's own PID). Even the ps command I just ran is a child of the shell, and it shows itself too. Now look into /proc:

cat /proc/$SP/cmdline | tr '\0' ' '; echogrep -E '^(Name|State|Pid|PPid|Uid|VmRSS)' /proc/$SP/statusls -l /proc/$SP/cwd /proc/$SP/exe | cut -c1-120

sleep 30 Name:	sleepState:	S (sleeping)Pid:	24938PPid:	24919Uid:	0	0	0	0VmRSS:	    1680 kBlrwxrwxrwx 1 root root 0 Oct 12 03:47 /proc/24938/cwd -> /tmp/lab-linux-a/l2lrwxrwxrwx 1 root root 0 Oct 12 03:47 /proc/24938/exe -> /usr/bin/sleep

The cmdline file separates arguments with a null byte, so I converted them to spaces with tr. cwd is a symlink to the process's working directory, and exe points to the program file. These two are the first things I read when I do not know what a mystery process is. VmRSS is the memory the process currently holds in RAM (1680 kB).

Listing processes: ps

ps prints a snapshot. Without options it only shows processes in your terminal, which is rarely what you want. The two common forms:

ps -ef | head -4        # POSIX style: every process, full formatps aux | head -3        # BSD style: every process, with CPU and memory

UID        PID  PPID  C STIME TTY          TIME CMDroot         1     0  0 03:03 ?        00:00:03 /process_api --firecracker-init --addr 0.0.0.0:2024 root         2     0  0 03:03 ?        00:00:00 [kthreadd]root         3     2  0 03:03 ?        00:00:00 [pool_workqueue_release]USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMANDroot         1  0.1  0.0  24840  5592 ?        SLl  03:03   0:03 /process_api --firecracker-init --addr 0.0.0.root         2  0.0  0.0      0     0 ?        S    03:03   0:00 [kthreadd]

(The two styles look odd side by side because ps is old and merges two traditions. Both work. ps -ef shows the parent PID, ps aux shows CPU and memory.) Things to notice:

• PID 1 here is /process_api, started at boot. On a normal Ubuntu server it would be /sbin/init (a link to systemd).
• Names in square brackets, like [kthreadd], are kernel threads. They are not programs on disk.
• TTY is the terminal. ? means no terminal: a daemon.
• VSZ is virtual memory (a promise, often big). RSS is the memory really in RAM. Use RSS to judge memory use.

You can choose your own columns and sort. This is my favourite form:

ps -eo pid,ppid,stat,%cpu,%mem,rss,comm --sort=-rss | head -5

  PID  PPID STAT %CPU %MEM   RSS COMMAND   97    87 Sl   10.1  2.9 489316 claude 5112     1 Ssl   3.2  1.5 251892 dkapi 1015     1 Sl    3.1  0.9 149332 dockerd 8953     1 Sl    1.2  0.5  98364 kube-controller

-e means every process, -o picks columns, --sort=-rss sorts by memory, biggest first. This is "who uses my RAM" in one line.

The STAT column

The STAT letters tell you what the process is doing:

A pile of processes in D is a classic sign of a slow disk or a hung network mount. We use this again in lesson 7.

Trees: who started whom

pstree -p $$

bash(24919)-+-head(24937)            `-pstree(24936)

pstree draws the parent-child tree. The -p shows PIDs. On a real server pstree -p (with no PID) shows everything under PID 1, and you can see, for example, nginx with its worker children. That is how you spot a leftover child that would not die.

Finding by name: pgrep and pkill

pgrep prints PIDs that match a name, and pkill sends a signal to them:

sleep 1234 &pgrep -af 'sleep 1234'

29727 sleep 1234

-a shows the full command line and -f matches against the full command line instead of just the program name. Prefer pgrep to the older ps aux | grep name, which also finds the grep itself.

bash -c 'sleep 1234 & pkill -f "sleep 1234"; echo survived'echo "bash -c status=$?"

Terminatedbash -c status=143

The bash -c '...' process has sleep 1234 inside its own command line, so pkill -f matched it and killed it before echo survived could run. (It exited with 143. We explain that number below.) Be more precise: pkill -x name matches the exact program name, or use a PID from a PID file. In my own session, this accident killed my own tool shell once, because my pasted command text contained the pattern. Always run pgrep -af pattern first to see what would match, and never pkill by a broad name on a shared machine.

What is on the machine: top, load and uptime

top shows a live, updating table. Press q to quit, M to sort by memory, P by CPU, k to kill, 1 to show each CPU. In scripts and logs, use batch mode, which prints once:

top -b -n 1 | head -12 | cut -c1-100

top - 03:48:26 up 45 min,  0 user,  load average: 7.65, 6.64, 4.45Tasks: 122 total,   3 running, 118 sleeping,   0 stopped,   1 zombie%Cpu(s): 88.1 us,  9.5 sy,  0.0 ni,  2.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st MiB Mem :  16093.2 total,   4860.3 free,   1987.6 used,   9589.8 buff/cache     MiB Swap:      0.0 total,      0.0 free,      0.0 used.  14105.5 avail Mem 

Read it top to bottom:

1. Uptime and load average: three numbers for the last 1, 5 and 15 minutes.
2. Tasks: counts by state. Here there is 1 zombie.
3. %Cpu(s): us is user code, sy is kernel, id is idle, wa is waiting for I/O. A high wa points to a slow disk.
4. Memory: look at avail Mem, not free. Linux uses spare RAM as a cache (buff/cache), so "free" is always small on a healthy machine.

Load average is the number of processes that are running, ready to run, or stuck in D state, averaged over time. Compare it with the number of CPUs:

cat /proc/loadavgnproc

7.65 6.64 4.45 10/279 297814

My load is 7.65 on 4 CPUs. As a rule, a load persistently above the CPU count means more work than CPUs, so tasks wait. (The sandbox was busy compiling other things at that moment, which shows up as the compile processes at the top of top.) A load of 7.65 on a 16-core machine would be fine. Always divide by the core count.

Foreground, background and job control

When you run a command normally, it runs in the foreground: it owns the terminal and you wait. To get your prompt back, there are three ways:

• Add & at the end: long-task & starts it in the background.
• Press Ctrl-Z while it runs: this sends SIGTSTP, which pauses it. Then type bg to let it continue in the background, or fg to bring it back.
• Press Ctrl-C: this sends SIGINT, which asks it to stop.

The shell keeps a table of its jobs. jobs lists them. fg %1 brings job 1 to the front. bg %1 resumes a stopped job in the background. kill %1 signals job 1 (the % means "job number", not PID). Here is a real run (with set -m to turn job control on in a script):

set -msleep 60 &sleep 61 &jobsjobs -lkill %1sleep 0.2jobskill %2; wait 2>/dev/nullecho "done: $(jobs | wc -l) jobs left"

[1]-  Running                 sleep 60 &[2]+  Running                 sleep 61 &[1]- 25244 Running                 sleep 60 &[2]+ 25245 Running                 sleep 61 &[1]-  Terminated              sleep 60[2]+  Running                 sleep 61 &done: 0 jobs left

+ marks the "current" job (what fg with no argument uses) and - the previous one. -l adds PIDs. After kill %1, the shell reports Terminated.

Why do you need this on a server? Because a build, a dump or a data copy can run for 20 minutes, and you want to run tail or top meanwhile. Though for really long jobs over SSH, there are better tools than &: see the next section.

wait pauses until background jobs finish, and wait $PID returns that job's exit status. That is how scripts run things in parallel:

task_a & pid_a=$!task_b & pid_b=$!wait $pid_a; echo "a exited with $?"wait $pid_b; echo "b exited with $?"

When you log out, your jobs may die: nohup and disown

When a terminal closes (you close the SSH window), the kernel sends SIGHUP ("hang up") to the shell, and the shell passes it to its jobs. Most programs die on SIGHUP. To survive it, start the program with nohup, or run disown on a job that is already running. I tested what SIGHUP does to a normal sleep and a nohup sleep:

sleep 100 & a=$!nohup sleep 101 >/dev/null 2>&1 & b=$!sleep 0.2kill -HUP $a $bsleep 0.3kill -0 $a 2>/dev/null && echo "plain sleep: alive" || echo "plain sleep: died on HUP"kill -0 $b 2>/dev/null && echo "nohup sleep: alive" || echo "nohup sleep: died on HUP"

hup.sh: line 5: 25871 Hangup                  sleep 100plain sleep: died on HUPnohup sleep: alive

nohup simply sets "ignore SIGHUP" before it starts your program. You can see this in the process's signal masks: SigIgn: 0000000000000001 means signal number 1 (HUP) is ignored. (Bit 0 of the mask is signal 1.)

nohup sleep 100 >/dev/null 2>&1 & n=$!grep -E 'SigIgn|SigCgt' /proc/$n/status

SigIgn:	0000000000000001SigCgt:	0000000000000000

Without redirection, nohup writes output to a file nohup.out, which is why I redirected to /dev/null. disown removes a running job from the shell's list, so the shell will not send it a hang-up. For serious work, use tmux or screen (a session that survives disconnects), or better, let systemd run the service (lesson 6). A job started with & is fragile. A service managed by systemd is not.

Signals

A signal is a one-way message from the kernel (or another process) to a process. Each has a number and a name. List them:

kill -l | head -4

 1) SIGHUP	 2) SIGINT	 3) SIGQUIT	 4) SIGILL	 5) SIGTRAP 6) SIGABRT	 7) SIGBUS	 8) SIGFPE	 9) SIGKILL	10) SIGUSR111) SIGSEGV	12) SIGUSR2	13) SIGPIPE	14) SIGALRM	15) SIGTERM16) SIGSTKFLT	17) SIGCHLD	18) SIGCONT	19) SIGSTOP	20) SIGTSTP

The ones you need to know:

The word kill is misleading. The command kill just sends a signal. Its default is SIGTERM. The numbers above match this machine and are the same on most Linux CPUs, but a few (like SIGUSR1) differ on other systems, so use names in scripts: kill -TERM, not kill -15.

The polite way and the forceful way

SIGTERM (15) means "please finish what you are doing and exit". The program may catch it, save its state, close connections and exit. SIGKILL (9) means "you are removed now". The kernel stops the process without telling it. No cleanup code runs. Temp files stay, lock files stay, half-written data may stay.

So the rule is: always try SIGTERM first, wait a few seconds, and use SIGKILL only if it ignores you. I made a worker that cleans up on SIGTERM:

# trap.shcleanup() { echo "got TERM, cleaning up tmp files"; rm -f work.tmp; exit 0; }trap cleanup TERMtrap 'echo "ignoring INT"' INTtouch work.tmpecho "worker pid $$ started"while true; do sleep 0.1; done

I ran it in the background, sent SIGINT, then SIGTERM:

bash trap.sh & W=$!sleep 0.5; kill -INT $W; sleep 0.3; kill -TERM $Wwait $W; echo "worker exit status=$?"ls work.tmp

worker pid 25046 startedignoring INTgot TERM, cleaning up tmp filesworker exit status=0ls: cannot access 'work.tmp': No such file or directory

The trap command says "when this signal arrives, run this code instead of the default". The worker ignored INT (printed a message and went on) and cleaned up on TERM, so work.tmp was removed. With kill -9, this cleanup code would never run. Let me prove that SIGKILL cannot be caught. Bash even lets you write the trap, but it never fires:

trap 'echo "KILL handler ran"' KILLtrap 'echo "TERM handler ran"' TERMwhile true; do sleep 0.1; done

bash trapk2.sh & T=$!; sleep 0.4; kill -9 $T; wait $T 2>/dev/nullecho "status=$? (no handler output above)"

status=137 (no handler output above)

No "KILL handler ran" line. The process was just gone. SIGSTOP is the same: it cannot be caught, blocked or ignored.

A program that ignores SIGTERM stays alive until you use SIGKILL:

# stubborn.sh:  trap '' TERM ; echo started ; loop foreverbash stubborn.sh & S=$!; sleep 0.4kill $S; sleep 0.4kill -0 $S && echo "still alive after TERM"kill -KILL $S; wait $S 2>/dev/null; echo "after KILL status=$?"

startedstill alive after TERMafter KILL status=137

Pause and resume

SIGSTOP pauses a process without ending it. SIGCONT resumes it. Watch the STAT column go S, T, S:

sleep 100 & p=$!; sleep 0.2ps -o pid,stat,comm -p $p | tail -1kill -STOP $p; sleep 0.2; ps -o pid,stat,comm -p $p | tail -1kill -CONT $p; sleep 0.2; ps -o pid,stat,comm -p $p | tail -1

25030 S    sleep25030 T    sleep25030 S    sleep

This is a handy trick to freeze a runaway job while you look at it, or to throttle a heavy job during business hours.

A signal to a whole group

A pipeline or a script with children makes several processes. They share a process group. A negative PID in kill means "the whole group". I started a subshell with two sleeps:

set -m( sleep 100 & sleep 101 & wait ) &g=$!ps -eo pid,ppid,pgid,stat,comm | awk -v g=$g 'NR==1 || $3==g'kill -TERM -- -$gsleep 1ps -eo pid,ppid,pgid,stat,comm | awk -v g=$g 'NR==1 || $3==g'

  PID  PPID  PGID STAT COMMAND28019 28018 28019 S    bash28021 28019 28019 S    sleep28022 28019 28019 S    sleep  PID  PPID  PGID STAT COMMAND28022     1 28019 Z    sleep

Three processes shared group 28019. One kill -TERM -- -28019 ended them all. The -- stops kill from reading -28019 as an option. One leftover line remains: a Z (zombie) with PPID 1. We explain this just below.

Exit codes

When a process ends, it returns a number. Shells keep the last one in $?. Conventions you can rely on:

I measured each one:

sh -c 'exit 3'; echo "exit 3 -> $?"(exit 256); echo "exit 256 -> $?"nonexistent_cmd 2>/dev/null; echo "not found -> $?"touch nx; ./nx 2>/dev/null; echo "no exec perm -> $?"grep -q zzz /etc/hostname; echo "grep nomatch -> $?"sleep 100 & p=$!; kill $p; wait $p; echo "TERM -> $?"sleep 100 & p=$!; kill -9 $p; wait $p; echo "KILL -> $?"sleep 100 & p=$!; kill -INT $p; wait $p; echo "INT -> $?"

exit 3 -> 3exit 256 -> 0not found -> 127no exec perm -> 126grep nomatch -> 1TERM -> 143KILL -> 137INT -> 130

(The last three came from the same run as status=143, 137 and 130 in my test script.) Two important points:

• An exit code is one byte. exit 256 wraps to 0, which shows why you should keep codes between 0 and 255.
• A code of 137 in a CI log or a Kubernetes/Docker event almost always means "killed with SIGKILL". The usual cause is the out-of-memory killer. 143 means someone sent SIGTERM, such as a normal stop or a deploy.

Node.js shows that a handler turns the 143 into a clean 0:

// server.jsconst http = require("http");const srv = http.createServer((q, r) => r.end("ok\n")).listen(0, () => console.log("listening, pid", process.pid));process.on("SIGTERM", () => {  console.log("SIGTERM: closing server, finishing work");  srv.close(() => { console.log("closed, exit 0"); process.exit(0); });});

node server.js & N=$!; sleep 0.7; kill -TERM $N; wait $N; echo "node exit status=$?"node -e 'setInterval(()=>{},1000)' & N=$!; sleep 0.5; kill -TERM $N; wait $N; echo "node default exit status=$?"

listening, pid 30001SIGTERM: closing server, finishing workclosed, exit 0node exit status=0node default exit status=143

With the handler, the server stops accepting new connections, lets running requests finish, and exits with 0. Without a handler, Node does the default, and exits with 143. This is how graceful shutdown works in every language, and why a deploy tool sends SIGTERM first and waits before SIGKILL.

Zombies and orphans

Two terms get confused.

A zombie is a process that has finished but whose exit status nobody has read. The parent must call wait() to collect it. Until then, the kernel keeps a tiny entry in the process table. It uses no CPU and almost no memory, but it takes a PID. I made one with Python (the child exits at once, the parent sleeps and does not call wait):

import os, time, subprocesspid = os.fork()if pid == 0:    os._exit(0)           # child exits at oncetime.sleep(0.3)print(subprocess.run(["ps","-o","pid,ppid,stat,comm","-p",str(pid)],capture_output=True,text=True).stdout.strip())os.waitpid(pid, 0)        # now the parent collects itprint("after waitpid:")print(subprocess.run(["ps","-o","pid,ppid,stat,comm","-p",str(pid)],capture_output=True,text=True).stdout.strip())

  PID  PPID STAT COMMAND25255 25254 Z    python3after waitpid:  PID  PPID STAT COMMAND

STAT is Z while the parent ignored it. After waitpid, the entry is gone. You cannot kill a zombie, because it is already dead. Sending signals does nothing. The fix is to make the parent call wait, or to kill/restart the parent. When the parent dies, the zombie is adopted by PID 1, which reaps it. A few zombies are harmless. Thousands mean a buggy parent that leaks PIDs.

An orphan is the opposite: the child still runs, but its parent died. The kernel gives it a new parent, normally PID 1. I made one:

( sleep 5 & echo $! > orphan.pid )   # the subshell ends right awaysleep 0.2p=$(cat orphan.pid)ps -o pid,ppid,comm -p $p

  PID  PPID COMMAND25265     1 sleep

PPID is 1. That is also how a proper daemon works: it detaches and PID 1 becomes its parent. And it is why that "one leftover zombie with PPID 1" in the process-group demo happened: its parent bash died first, PID 1 adopted it, and in this sandbox PID 1 (process_api) was slow to reap it. On a systemd machine this clean-up is nearly instant.

Why PID 1 is special in containers

PID 1 is treated specially by the kernel: a signal that PID 1 has no handler for is ignored. On a normal server systemd has handlers. In a Docker container, your app may be PID 1 and may have none, so docker stop sends SIGTERM, nothing happens, and after the timeout Docker sends SIGKILL. I reproduced it in a fresh PID namespace (unshare --pid --fork --mount-proc), where my script becomes PID 1:

# pid1.shecho "inside: my pid is $$"kill -TERM $$echo "still alive after SIGTERM to myself (PID 1 has no default handler for TERM)"sleep 0.1kill -KILL $$echo "not reached"

timeout 10 unshare --pid --fork --mount-proc bash pid1.sh

inside: my pid is 1still alive after SIGTERM to myself (PID 1 has no default handler for TERM)not reached

Both signals were ignored, even SIGKILL, when sent from inside the namespace to PID 1. The script ran to the end, and it printed the line I labelled "not reached", which proves that neither signal stopped it. The practical fix in containers: give the app a SIGTERM handler (as the Node example did), or use a tiny init such as docker run --init (which runs a small init as PID 1 that forwards signals and reaps zombies).

Priority: nice and renice

When the CPU is busy, the scheduler shares it. Niceness hints at the share. The default is 0. A higher number makes a process "nicer" (it yields more). Only root can use negative values.

nice -n 10 sh -c 'ps -o pid,ni,comm -p $$'sleep 100 & p=$!renice -n 5 -p $pps -o pid,ni,comm -p $p | tail -1

  PID  NI COMMAND26098  10 sh26108 (process ID) old priority 0, new priority 526108   5 sleep

Use nice -n 19 command for backups, compression and big builds that should not slow the site down. It only matters when the CPU is contended. It does not limit memory or disk.

Limits: timeout and ulimit

timeout runs a command and stops it after a time limit. It exits with 124 if it had to stop it:

timeout 1 sleep 5; echo "timeout status=$?"timeout -s KILL 1 sleep 5; echo "kill status=$?"timeout 2 true; echo "ok=$?"

timeout status=124kill status=137ok=0

By default timeout sends SIGTERM. With -s KILL it sends SIGKILL (so the status is 137). I use timeout in scripts and CI so that a hung command cannot block forever.

ulimit shows or sets per-process limits, such as the maximum number of open files. A server that fails with "Too many open files" has hit one of these:

ulimit -nulimit -u

2000064299

The first is the limit of open files per process (20000 here, but the common default on many systems is 1024). The second is the limit of processes per user. Limits set with ulimit apply to the current shell and its children. For services, set LimitNOFILE= in the systemd unit.

Putting it together: a stuck process, step by step

A short recipe that matches what I would type on a real server:

1. Find it: pgrep -af myapp, or ps -eo pid,ppid,stat,%cpu,%mem,etime,cmd --sort=-%cpu | head.
2. Look at it: ls -l /proc/PID/cwd /proc/PID/exe, and ps -o pid,ppid,user,stat,etime,cmd -p PID.
3. Ask nicely: kill PID. Wait 10 to 30 seconds. Check with kill -0 PID or ps -p PID.
4. If it ignores you and its state is not D: kill -KILL PID.
5. Check children: pstree -p PID before, and pgrep -P PID after. Kill leftover children or the process group (kill -TERM -- -PGID).
6. Find why: look at the log and the exit code (137 or 143?) before you restart. Killing a process only hides the symptom.

Try it

Do these in a scratch folder. Use only processes you started yourself. Do not run pkill with broad names.

1. Start sleep 300 &. Find its PID with echo $!, then run ps -o pid,ppid,stat,cmd -p PID. Whose child is it? What is its STAT?
2. Send it SIGSTOP, check STAT again, then SIGCONT. What letters do you see?
3. Kill it with a plain kill, then run wait PID; echo $?. What number do you get, and what does it mean?
4. Start another sleep 300 & and kill it with kill -9. What is the exit code now?
5. Write a script that traps TERM, prints a message, and exits 0. Run it in the background, then send SIGTERM. Check the exit code. Then change kill to kill -9: does the message print?
6. Run cat /proc/loadavg and nproc. Is your machine overloaded by the "load above core count" rule?
7. Run timeout 1 sleep 5; echo $? and then timeout 1 true; echo $?. Why are they different?
8. Run ps -eo pid,ppid,stat,comm | awk '$3 ~ /Z/'. Do you have any zombies? If one exists, which process do you have to deal with?

Answers

1. It is a child of your shell (PPID is echo $$). STAT is S (sleeping).
2. After SIGSTOP the letter is T (stopped). After SIGCONT it goes back to S.
3. 143. That is 128 + 15, "killed by SIGTERM".
4. 137, which is 128 + 9, "killed by SIGKILL".
5. With kill (TERM) the message prints and the exit code is 0. With kill -9 the message does not print and the exit code is 137, because SIGKILL cannot be caught.
6. If the first number of loadavg stays well above the nproc value, there are more runnable tasks than CPUs. A short spike is normal. A sustained value above the core count is the warning sign.
7. timeout 1 sleep 5 had to stop the command at 1 second, so it exits 124. timeout 1 true finished in time, so it returns the command's own status, 0.
8. A zombie has Z in STAT. You cannot kill the zombie itself. Look at its PPID: that parent must call wait, or be restarted. When the parent exits, PID 1 adopts and clears the zombie.

Interview questions

Key takeaways

• A process is a running program with a PID, a parent, a user and a state. /proc/PID shows its details, and ps -eo ... --sort= plus pstree and top are the main views.
• SIGTERM asks, SIGKILL forces and cannot be caught. Try TERM first, wait, then KILL. SIGSTOP and SIGCONT pause and resume.
• Exit code 0 is success. 126 is not executable, 127 is not found. 128+N means killed by signal N: 137 is KILL, 143 is TERM.
• & starts a background job. A closing terminal sends SIGHUP. Use nohup, disown, tmux or systemd for things that must survive.
• Zombies are dead processes waiting for their parent to collect them. You fix the parent. Orphans are live processes adopted by PID 1.
• PID 1 ignores signals it has no handler for. In containers, handle SIGTERM or use an init.
• Load average only makes sense next to the CPU count. nice and renice shift CPU priority. timeout and ulimit set limits.
• pkill -f matches full command lines, even your own. Check with pgrep -af first.

What's next

Next: Pipes, Redirection and Text Tools, Lesson 3. You will learn stdin, stdout and stderr, how | connects two processes, and how to search and reshape text with grep, sed, awk, find and xargs, with real one-line recipes. Practise now: start three sleep jobs, look at them with ps and pstree, stop one, kill one with SIGTERM and one with SIGKILL, and write down all three exit codes.