Shell Scripting: Variables, Conditions, Loops and Functions

Sooner or later you type the same five commands for the third time. That is the moment to write a script. Scripts run your deploys, your backups, your CI steps and the first line of many Docker containers. They are also where a small mistake can delete the wrong folder, because the shell is a very forgiving language. It will happily run rm -rf / if a variable is empty.

This lesson teaches the parts of bash (the most common shell) that you need to write short, safe scripts. It follows pipes and text tools in lesson 3, because scripts are mostly those pipelines with names and decisions around them. You will also need exit codes from lesson 2. The next lesson, SSH and networking, uses scripts to run commands on many servers.

Words used in this lesson

• Script is a text file of shell commands that you can run again and again.
• Shebang is the first line, #!/usr/bin/env bash. It tells the system which program should read the file.
• Variable is a name that holds text. You set it with name=value (no spaces) and read it with $name.
• Expansion is when the shell replaces something with its value before running the command, like $name or *.txt.
• Word splitting is when the shell cuts an unquoted value into pieces at spaces. This is the cause of many bugs.
• Quoting is using "..." or '...' to stop the shell from splitting or expanding.
• Exit status is the number a command returns. 0 is success. It is stored in $?.
• Test is a command that checks something and returns 0 or 1. [ ... ] and [[ ... ]] are tests.
• Function is a named block of commands you can call like a command.
• Subshell is a child copy of the shell. Changes to variables inside it do not come back.
• Positional parameters are the arguments given to a script: $1, $2, ... and $@ for all of them, $# for how many.
• Trap is a rule: "when this signal or event happens, run this command."
• Lint means checking code for likely mistakes without running it. ShellCheck is a linter for shell scripts.

Your first script, and the shebang

A script is just a file. Make it, mark it executable (lesson 1), and run it:

cat > hello.sh <<'EOF'#!/usr/bin/env bashecho "hello from $0"EOFchmod +x hello.sh./hello.sh

The first line is the shebang. When you run ./hello.sh, the kernel reads those first two characters, #!, and starts the program named after them with your file as input. I use #!/usr/bin/env bash because env finds bash wherever it is installed. You may also see #!/bin/bash. Both work on most Linux machines. Running a script with bash hello.sh also works, and the shebang is then ignored.

Why does it matter which shell? On Ubuntu and Debian, /bin/sh is not bash. It is dash, a smaller shell that only knows the basic POSIX features. Look:

ls -l /bin/shprintf '#!/bin/sh\nif [[ -n "$1" ]]; then echo yes; fi\narr=(a b)\n' > posix.shsh posix.sh x

lrwxrwxrwx 1 root root 4 Mar 31  2024 /bin/sh -> dashposix.sh: 2: [[: not foundposix.sh: 3: Syntax error: "(" unexpected

The double brackets [[ ]] and arrays ( ) are bashisms (bash-only features). If you write #!/bin/sh you must stay with plain POSIX syntax. If you want the nice features, write #!/usr/bin/env bash. A common bug is writing bash code under an sh shebang and finding out when it runs on another machine. ShellCheck catches it, as you will see later.

Variables

You assign with = and no spaces around it. You read with $:

cat > vars.sh <<'EOF'#!/usr/bin/env bashname="Ada"count=3echo "Hello, $name. Count is $count."echo "Next is $((count + 1))"echo "Braces: ${name}ism and ${name:-nobody} and ${missing:-default}"today=$(date +%Y)echo "Year has ${#today} chars"file="report.final.txt"echo "${file%.txt} | ${file##*.} | ${file#report.}"echo "upper: ${name^^}  lower: ${name,,}"EOFbash vars.sh

Hello, Ada. Count is 3.Next is 4Braces: Adaism and Ada and defaultYear has 4 charsreport.final | txt | final.txtupper: ADA  lower: ada

Let me explain each line, because this is dense:

• $((count + 1)) is arithmetic. It does whole-number maths only.
• ${name}ism uses braces so the shell knows where the name ends. $nameism would look for a variable called nameism.
• ${missing:-default} means "use default if missing is empty or not set". This is how you give a default value.
• $(date +%Y) is command substitution: run the command and put its output here. Prefer $(...) to the old backticks, because it nests and reads better.
• ${#today} is the length of the text.
• ${file%.txt} removes the shortest match of .txt from the end. ${file##*.} removes the longest match of *. from the start, which leaves the extension. ${file#report.} removes report. from the start. These string tricks are done by bash itself, with no extra process, so they are fast.
• ${name^^} and ${name,,} change case (bash 4 and newer).

Variable names are case sensitive. By habit, environment variables (like PATH, HOME) are UPPER_CASE, and your own script variables are lower_case, so you do not overwrite something important by accident.

Local, global and exported

By default a variable set in a script is visible everywhere in that script, even inside functions. Use local inside a function to keep it private. A variable is only passed to child programs if you export it:

cat > scope.sh <<'EOF'x=globalf() { local x=inner; y=leaked; echo "in f: x=$x"; }fecho "after: x=$x y=$y"export FOO=barbash -c 'echo "child sees FOO=$FOO BAZ=${BAZ:-unset}"'BAZ=1; bash -c 'echo "child sees BAZ=${BAZ:-unset}"'EOFbash scope.sh

in f: x=innerafter: x=global y=leakedchild sees FOO=bar BAZ=unsetchild sees BAZ=unset

Three facts in one run. local x did not change the outer x. y had no local, so it leaked out of the function. And only the exported FOO reached the child process. BAZ was set but not exported, so the child saw nothing. This is the same environment idea from lesson 6: cron jobs and services often fail because a variable that exists in your terminal does not exist in theirs.

Arrays

Bash has arrays when you need a list of things that may contain spaces:

arr=("one two" three four)echo "${#arr[@]} items"for i in "${arr[@]}"; do echo "- $i"; donedeclare -A age=([ada]=36 [bob]=41)   # associative array (key -> value)echo "ada=${age[ada]}"

3 items- one two- three- fourada=36

"${arr[@]}" expands to each element as a separate, properly kept word. "one two" stayed as one item. Associative arrays (declare -A) work like a dictionary. They need bash 4+, so they do not exist in the old bash 3.2 that macOS ships.

Quoting: the most important topic

Almost every shell bug is a quoting bug. The rule is short: always put double quotes around variable expansions ("$var", "$(cmd)", "$@"), unless you have a specific reason not to.

Why? When a variable is unquoted, the shell does two things to its value: it splits it into words at spaces, and then it expands wildcards (*, ?) in each word. Here is a file with a space in its name:

mkdir q && cd q && touch "my file.txt" other.txtf="my file.txt"ls -l $f          # unquotedls -l "$f"        # quoted

ls: cannot access 'my': No such file or directoryls: cannot access 'file.txt': No such file or directory-rw-r--r-- 1 root root 0 Oct 17 12:00 my file.txt

Unquoted, ls received two arguments, my and file.txt. Neither exists. Quoted, it received one. Now count the arguments precisely with a tiny helper script:

cat > cnt.sh <<'EOF'echo "got $# args"; for a in "$@"; do echo "  [$a]"; doneEOFv="a b  c"bash cnt.sh $vbash cnt.sh "$v"

got 3 args  [a]  [b]  [c]got 1 args  [a b  c]

Unquoted gives three arguments and loses the double space. Quoted keeps exactly what was there. The same thing happens with a wildcard:

x="*.sh"echo $x        # unquoted: the shell expands the starecho "$x"      # quoted: stays as text

cnt.sh scope.sh vars.sh*.sh

Double quotes, single quotes, no quotes

v="a b  c"echo 'single: $v'echo "double: $v"

single: $vdouble: a b  c

dir=""echo "target: /${dir}"echo "safe: /${dir:?dir is empty}"

target: /bash: line 1: dir: dir is empty

Exit codes: the language of "did it work?"

Every command returns a number when it ends. 0 means success, anything else means failure. The shell keeps the last one in $?. This is the same exit status from lesson 2.

true;  echo "true=$?"false; echo "false=$?"ls /nonexist 2>/dev/null; echo "ls=$?"bash -c 'exit 3'; echo "exit3=$?"nosuchcmd 2>&1; echo "rc=$?"

true=0false=1ls=2exit3=3/bin/bash: line 77: nosuchcmd: command not foundrc=127

A few codes you will meet: 1 (general error), 2 (misuse, for example bad options), 126 (found but not executable), 127 (command not found), 130 (stopped with Ctrl-C), 137 (killed by SIGKILL, which is 128 + 9), 143 (stopped by SIGTERM, 128 + 15).

Because the exit status drives everything, if in the shell does not take a true/false value. It takes a command, runs it, and checks if the status is 0:

if grep -q "n=7" cond.sh; then echo "found"; fi

grep -q prints nothing and only returns 0 (found) or 1 (not found). This is the shell's normal pattern.

Two operators join commands by their status:

true && echo A || echo B     # Afalse && echo A || echo B    # Btrue && false || echo "gotcha"

ABgotcha

cmd1 && cmd2 runs cmd2 only if cmd1 succeeded. cmd1 || cmd2 runs cmd2 only if cmd1 failed. The last line is the trap: A && B || C is not the same as if/else. If B fails, C runs too, even though A worked. For real branching, use if.

Conditions: if, [ ], [[ ]] and case

The [ is not special syntax. It is a command (a builtin, and also a program at /usr/bin/[) that needs its closing ] as the last argument, and that is why you need spaces inside the brackets. The double bracket [[ ]] is a bash keyword and it is safer and more powerful. In bash scripts, use [[ ]] by default.

cat > cond.sh <<'EOF'#!/usr/bin/env bashn=7; s="hello"; f=cond.shif [ "$n" -gt 5 ]; then echo "n>5"; fiif [[ $s == h* ]]; then echo "starts with h (glob in [[ ]])"; fiif [[ $s =~ ^h(.)l ]]; then echo "regex matched, group1=${BASH_REMATCH[1]}"; fi[ -f "$f" ] && echo "$f is a file"[ -d /tmp ] && echo "/tmp is a dir"[ -z "" ] && echo "empty string is -z"[ -n "$s" ] && echo "non-empty is -n"[ -x "$f" ] || echo "$f is not executable"if [ "10" \> "9" ]; then echo "string compare: 10 > 9"; else echo "string compare: 10 < 9 (string order!)"; fiif [ 10 -gt 9 ]; then echo "number compare: 10 > 9"; ficase "$s" in  hi|hello) echo "greeting";;  bye) echo "farewell";;  *) echo "unknown";;esacEOFbash cond.sh

n>5starts with h (glob in [[ ]])regex matched, group1=econd.sh is a file/tmp is a dirempty string is -znon-empty is -ncond.sh is not executablestring compare: 10 < 9 (string order!)number compare: 10 > 9greeting

Useful tests:

Pay attention to the line that says 10 < 9. The operator \> compared them as text, so "10" comes before "9" (because 1 sorts before 9). For numbers, use -gt, -lt or (( a > b )). Mixing the two is a classic mistake.

Why [[ ]] is safer than [ ]

Inside single brackets, the shell expands the variables first, then [ sees the result as words. If a variable is empty or has spaces, the command gets broken:

bash -c 'v="";    [ $v = "x" ] && echo yes'bash -c 'v="a b"; [ $v = "a" ] && echo yes'bash -c 'v="";    [[ $v = "x" ]] && echo yes; echo "[[ ok, rc=$?"'bash -c '[ 5 -gt abc ]; echo rc=$?'

bash: line 1: [: =: unary operator expectedbash: line 1: [: too many arguments[[ ok, rc=1bash: line 1: [: abc: integer expression expectedrc=2

The first two failed because [ $v = "x" ] became [ = "x" ] (empty variable vanished) and [ a b = "a" ] (split into two words). With [[ ]] the variable is not split, so it just works. Quote anyway ("$v"), and that makes [ ] safe too. Note the last case: a comparison with something that is not a number gives an error and status 2, not "false". Do not rely on that.

case

When you compare one value with many patterns, case is cleaner than a ladder of if. Each branch ends with ;;, and *) is the catch-all. You will see it a lot in option parsing (below).

Loops

cat > loops.sh <<'EOF'for i in 1 2 3; do echo -n "$i "; done; echofor i in {1..5..2}; do echo -n "$i "; done; echofor ((i=0;i<3;i++)); do echo -n "c$i "; done; echofor f in *.sh; do echo "file: $f"; donei=0; while [ $i -lt 3 ]; do echo "while $i"; i=$((i+1)); doneprintf 'a b\nc d\n\ne f\n' > lines.txtwhile IFS= read -r line; do echo "line=[$line]"; done < lines.txtfor i in 1 2 3 4 5; do [ $i -eq 2 ] && continue; [ $i -eq 4 ] && break; echo "kept $i"; doneEOFbash loops.sh

1 2 3 1 3 5 c0 c1 c2 file: cnt.shfile: cond.shfile: loops.shfile: scope.shfile: vars.shwhile 0while 1while 2line=[a b]line=[c d]line=[]line=[e f]kept 1kept 3

• for x in list loops over words. {1..5..2} is a range (start, end, step). for ((...)) is the C style.
• for f in *.sh loops over files. This is the right way to loop over files.
• while repeats while a command succeeds. until is the opposite.
• while IFS= read -r line; do ... done < file reads a file line by line. IFS= stops the shell from trimming spaces at the ends of the line. -r stops it from treating backslashes as escapes. Always write IFS= read -r. Notice the empty line was kept as line=[].
• break leaves the loop, continue jumps to the next round.

Do not loop over ls

This one is so common that ShellCheck has an error for it. With a file named a b.txt:

touch "a b.txt" c.txtfor f in $(ls); do echo "[$f]"; donefor f in *; do echo "ok [$f]"; done

[a][b.txt][c.txt]ok [a b.txt]ok [c.txt]

$(ls) returns text. The shell splits that text at spaces, so a b.txt becomes two broken items. A glob (*) hands over real file names, and each stays whole. To find files in many folders, use find ... -print0 with xargs -0 or while read -d '', as in lesson 3.

The pipe-into-while trap (subshell)

Each side of a pipe runs in its own subshell (a child copy of the shell). Variables you change in the child are lost when it ends:

c=0printf 'x\ny\nz\n' | while read -r l; do c=$((c+1)); doneecho "pipe version: c=$c"c=0while read -r l; do c=$((c+1)); done < <(printf 'x\ny\nz\n')echo "process-subst version: c=$c"

pipe version: c=0process-subst version: c=3

The counter stayed at 0 with the pipe, because the loop ran in a child. With process substitution (< <(cmd), from lesson 3) the loop stays in the main shell, and the counter is right. (There is also the shopt -s lastpipe option, which I did not test.)

Functions

A function is a named group of commands. Arguments arrive as $1, $2, ... inside the function, which is separate from the script's arguments. A function can return a status (0 to 255) with return, and it can print a result, which you capture with $(...).

cat > fn.sh <<'EOF'greet() {  local who="${1:-world}"  echo "hello, $who"}is_even() { (( $1 % 2 == 0 )); }add() { echo $(( $1 + $2 )); }die() { echo "error: $*" >&2; exit 1; }greet; greet Adamsg=$(greet Bob); echo "captured: $msg"is_even 4 && echo "4 even"; is_even 5 || echo "5 odd (rc=$?)"sum=$(add 2 3); echo "sum=$sum"die "something broke"echo "not reached"EOFbash fn.sh; echo "script rc=$?"

hello, worldhello, Adacaptured: hello, Bob4 even5 odd (rc=1)sum=5error: something brokescript rc=1

Four ideas here:

1. local who=... keeps the variable inside the function.
2. Results come back in two different ways. A function "returns" success/failure through its status (is_even), and "returns" data by printing it (add). Do not try return 5 for data. It only keeps the number 0 to 255 (I tested return 300 and got 44, which is 300 modulo 256).
3. The die helper prints to stderr (>&2, lesson 3), so error text does not mix with normal output, and then exits with 1. Almost every real script has a die or fail function.
4. exit inside a function exits the whole script. return leaves only the function.

Arguments: $1, $@, shift and getopts

cat > args.sh <<'EOF'#!/usr/bin/env bashecho "script: $0"echo "count:  $#"echo "first:  ${1:-<none>}  second: ${2:-<none>}"echo "all via \"\$@\":"; for a in "$@"; do echo "  [$a]"; doneecho "\"\$*\" joins: [$*]"shiftecho "after shift: $# left, first now ${1:-<none>}"EOFbash args.sh one "two words" three

script: args.shcount:  3first:  one  second: two wordsall via "$@":  [one]  [two words]  [three]"$*" joins: [one two words three]after shift: 2 left, first now two words

"$@" keeps "two words" as one item. That is why you almost always want "$@" when you pass arguments on to another command. shift removes the first argument and moves the rest down.

Options with getopts

For -v and -n NAME style flags, use the builtin getopts. It is small, standard and good enough for most scripts:

cat > opts.sh <<'EOF'#!/usr/bin/env bashset -euo pipefailusage() { echo "usage: $0 [-v] [-n NAME] FILE..." >&2; exit 2; }verbose=0; name="world"while getopts ":vn:h" opt; do  case $opt in    v) verbose=1;;    n) name=$OPTARG;;    h) usage;;    :) echo "option -$OPTARG needs a value" >&2; usage;;    \?) echo "unknown option -$OPTARG" >&2; usage;;  esacdoneshift $((OPTIND - 1))[ $# -ge 1 ] || { echo "need at least one FILE" >&2; usage; }echo "verbose=$verbose name=$name files=$*"EOFbash opts.sh -v -n Ada a.txt "b c.txt"bash opts.sh -n;   echo rc=$?bash opts.sh -x a; echo rc=$?bash opts.sh -v;   echo rc=$?

verbose=1 name=Ada files=a.txt b c.txtoption -n needs a valueusage: opts.sh [-v] [-n NAME] FILE...rc=2unknown option -xusage: opts.sh [-v] [-n NAME] FILE...rc=2need at least one FILEusage: opts.sh [-v] [-n NAME] FILE...rc=2

The option string ":vn:h" means: the leading : turns on quiet error mode (we print our own errors), v is a flag, n: takes a value (the colon after it), h is a flag. The value lands in $OPTARG. After the loop, shift $((OPTIND - 1)) removes the options so $@ holds only the file names. Usage errors use exit code 2 and print to stderr, as most Unix tools do. getopts does not understand long options like --verbose. For those, use a while/case loop on $1 with shift.

A quick trick for a required argument: target=${1:?usage: script.sh TARGET}. It stops with that message if $1 is missing.

set -euo pipefail: the safety belt

By default bash is optimistic. When a command fails, it carries on. Here is what that looks like with a cd that fails:

cat > unsafe.sh <<'EOF'#!/usr/bin/env bashcd /tmp/lab-linux-c/l4/nodirecho "now in: $(pwd)"echo "would delete files in $PWD"EOFbash unsafe.sh; echo rc=$?

unsafe.sh: line 2: cd: /tmp/lab-linux-c/l4/nodir: No such file or directorynow in: /tmp/lab-linux-c/l4would delete files in /tmp/lab-linux-c/l4rc=0

The cd failed, and the script went on in the wrong folder and ended with status 0 ("success"). If the next line had been rm -rf *, you would delete the wrong files. Three options fix most of this. Put them at the top:

set -euo pipefail

-e: stop when a command fails

set -ecd /tmp/lab-linux-c/l4/nodirecho "never printed"

e.sh: line 3: cd: /tmp/lab-linux-c/l4/nodir: No such file or directoryrc=1

The script stopped at the first failure and returned the failure status.

-u: unset variables are errors

set -udir="/tmp/x"echo "target: $dri/data"     # typo: dri

u.sh: line 4: dri: unbound variablerc=1

Without -u, $dri would be silently empty, and you would work on /data. With -u, a typo stops the script. Use ${var:-} when an empty value is acceptable.

-o pipefail: a pipeline fails if any part fails

Normally a pipeline's status is the status of the last command only:

bash -c 'false | cat; echo "no pipefail rc=$?"'bash -c 'set -o pipefail; false | cat; echo "pipefail rc=$?"'bash -c 'set -eo pipefail; ls /nonexist 2>/dev/null | wc -l; echo "not reached"'; echo rc=$?

no pipefail rc=0pipefail rc=10rc=2

false | cat looked fine because cat succeeded. With pipefail the failing false is reported. In the third line, ls failed (status 2), wc still printed 0, and with -e the script stopped with status 2 and never reached the echo. You can also look at ${PIPESTATUS[@]} right after a pipeline to see each part's status. I ran false | true | cat and got 1 0 0.

set -e has surprises (this is a favourite interview topic)

set -e is useful, but it is not a full error handler. It has exceptions, and some of them are odd. These all ran in one script with set -e on:

set -efalse || echo "1: || protects false"test -f /nonexist && echo hiecho "2: survived failed && list"f() { false; echo "3: inside f after false (in if context)"; }if f; then echo "f ok"; fin=0(( n++ )) || echo "4: (( n++ )) returned nonzero when n was 0"let "m=0"echo "5: before final"

1: || protects false2: survived failed && list3: inside f after false (in if context)f ok4: (( n++ )) returned nonzero when n was 0

Notice what is missing: the script died at let "m=0" and never printed "5". The command let "m=0" returns status 1 when the result of the expression is zero (the same reason (( n++ )) returned non-zero at the start). In plain terms: the maths commands use "zero means false", which is the opposite of the exit-status rule, and set -e then reads it as a failure.

What the run shows:

1. A failing command on the left of || or && is not fatal. That is how you say "this may fail".
2. test -f /nonexist && echo hi failed, but a failure in a && list (not the last command) is ignored.
3. Inside a function that is called in an if condition, set -e is switched off. Line 3 printed even though false ran before it. This surprises people, because they expect the function to stop at false.
4. (( n++ )) and let can return 1 for a perfectly normal calculation.

One more, with local:

set -ef() { local v=$(false); echo "local masked failure, kept going"; }fg() { local v; v=$(false); echo "not reached"; }g

local masked failure, kept goingrc=1

In f, the status of local v=$(false) is the status of local (always 0), not of false. The error is hidden. In g, declaring first and assigning on a separate line shows the failure, and the script stops. ShellCheck warns about this one too (SC2155). So: declare and assign local variables on separate lines when the value comes from a command.

Don't forget IFS

You may see IFS=$'\n\t' after set -euo pipefail in "strict mode" guides. It changes how the shell splits words. I did not need it in these examples because I quote everything, and it can confuse other tools, so I will not add it by default. Quoting is the real fix.

trap: clean up even when things go wrong

Scripts often make temporary files or start background jobs. If the script fails or is stopped, you want those cleaned up. trap 'command' EVENT runs a command when an event happens. EXIT is a special event that fires whenever the script ends, normally or not.

cat > trap.sh <<'EOF'#!/usr/bin/env bashset -euo pipefailtmp=$(mktemp -d)cleanup() { echo "cleanup: removing $tmp"; rm -rf "$tmp"; }trap cleanup EXITecho "working in $tmp"echo data > "$tmp/work.txt"if [ "${1:-}" = fail ]; then echo "failing now"; false; fiecho "finished normally"EOFbash trap.sh;      echo rc=$?bash trap.sh fail; echo rc=$?

working in /tmp/tmp.tjXZGwxKyIfinished normallycleanup: removing /tmp/tmp.tjXZGwxKyIrc=0working in /tmp/tmp.wRJPIF2qmDfailing nowcleanup: removing /tmp/tmp.wRJPIF2qmDrc=1

In both runs the temp folder was removed. In the second run the script failed, and trap still cleaned up, and the exit status (1) was kept. The folder names change each time. mktemp -d creates a new, private temp directory with a random name. That is safer than /tmp/myfile, which another user could pre-create.

You can also trap signals (lesson 2) to shut down nicely. Here is a script that handles SIGTERM (what kill sends by default):

cat > trap2.sh <<'EOF'trap 'echo "caught SIGINT, stopping"; exit 130' INTtrap 'echo "caught SIGTERM"; exit 143' TERMtrap 'echo "exit trap, status $?"' EXITecho "pid $$"sleep 30 &waitEOFbash trap2.sh & p=$!; sleep 0.5; kill -TERM $p; wait $p; echo "rc=$?"

pid 1879caught SIGTERMexit trap, status 143[1]+  Exit 143                bash trap2.shrc=143

The TERM trap ran, then exit 143 triggered the EXIT trap, which printed the status. 143 is the conventional 128 + 15. (The background sleep 30 was left behind; in a real script, kill your children in the trap. I stopped it by hand.) You cannot trap SIGKILL (kill -9), so cleanup code never runs then, which is why -9 is a last resort.

There is also an ERR trap that runs when a command fails (with set -e). It is handy for printing the line number:

bash -c 'trap '\''echo "ERR at line $LINENO (rc=$?)"'\'' ERR; set -e; echo before; false; echo after'

beforeERR at line 1 (rc=1)

Here documents and printing

A here document feeds a block of text to a command. If the end word is quoted ('EOT'), nothing inside is expanded:

name=Adacat <<EOTHi $name, today is fixedpath: \$HOME stays literal when escapedEOTcat <<'EOT'Quoted delimiter: $name is NOT expanded, `date` neitherEOT

Hi Ada, today is fixedpath: $HOME stays literal when escapedQuoted delimiter: $name is NOT expanded, `date` neither

For output, prefer printf over echo when the text is not fixed. echo treats a leading -n or -e as options:

echo -n "-n"; echoecho "-e"printf "%s\n" "-n" "-e"printf "%5.1f|%-6s|%03d\n" 3.14159 ab 7

-n-e  3.1|ab    |007

The two echo lines printed nothing visible, because -n and -e were taken as options. printf "%s\n" "$text" prints any text exactly. The last line shows the format codes: %5.1f is a float in 5 characters with 1 decimal, %-6s is left-aligned text, %03d is a number padded with zeros.

ShellCheck: the best free helper

ShellCheck reads your script without running it and points out likely bugs. Install it with sudo apt install shellcheck (Debian and Ubuntu), brew install shellcheck (macOS), or download a release from its GitHub page. It was already on this machine (version 0.9.0). Here is a deliberately bad script:

#!/bin/bashbackup_dir=$1for f in $(ls $backup_dir); do  cp $f /backup/donecd $backup_dirrm -rf $backup_dir/*if [ $count == 3 ]; then echo hi; fiecho `date`cat file | grep foo | wc -l[ -n $x ] && echo set

shellcheck bad.sh printed this (I trimmed the "Did you mean" suggestions to save space; the real output includes them):

In bad.sh line 3:for f in $(ls $backup_dir); do         ^---------------^ SC2045 (error): Iterating over ls output is fragile. Use globs.              ^---------^ SC2086 (info): Double quote to prevent globbing and word splitting.In bad.sh line 4:  cp $f /backup/     ^-- SC2086 (info): Double quote to prevent globbing and word splitting.In bad.sh line 6:cd $backup_dir^------------^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.In bad.sh line 7:rm -rf $backup_dir/*       ^-----------^ SC2115 (warning): Use "${var:?}" to ensure this never expands to /* .In bad.sh line 8:if [ $count == 3 ]; then echo hi; fi     ^----^ SC2154 (warning): count is referenced but not assigned.In bad.sh line 9:echo `date`     ^----^ SC2046 (warning): Quote this to prevent word splitting.     ^----^ SC2006 (style): Use $(...) notation instead of legacy backticks `...`.In bad.sh line 10:cat file | grep foo | wc -l    ^--^ SC2002 (style): Useless cat. Consider 'cmd < file | ..' or 'cmd file | ..' instead.           ^------^ SC2126 (style): Consider using 'grep -c' instead of 'grep|wc -l'.In bad.sh line 11:[ -n $x ] && echo set     ^-- SC2070 (error): -n doesn't work with unquoted arguments. Quote or use [[ ]].

Every lesson in this page appears: loop over ls (SC2045), missing quotes (SC2086), cd without a check (SC2164), the empty-variable rm -rf (SC2115), a variable that was never set (SC2154), old backticks (SC2006), pointless cat (SC2002). The exit status of shellcheck was 1, so you can use it in CI to fail a build. The code like SC2086 links to a page on the ShellCheck wiki that explains the rule (the wiki itself was blocked here, so I did not read it, but the message text is enough).

Now the fixed version, which I called good.sh:

#!/usr/bin/env bashset -euo pipefailbackup_dir=${1:?usage: good.sh DIR}dest=${2:-/tmp/backup-demo}mkdir -p "$dest"for f in "$backup_dir"/*; do  [ -e "$f" ] || continue  cp -- "$f" "$dest/"donecd "$backup_dir" || exit 1echo "copied $(find "$dest" -type f | wc -l) files"

$ shellcheck good.sh; echo "shellcheck rc=$?"shellcheck rc=0$ bash good.sh srccopied 2 files$ bash good.shgood.sh: line 3: 1: usage: good.sh DIR

(My test folder src had a file called a b.txt and one called c.txt. The spaced name was copied correctly.) Small details that make this safe: [ -e "$f" ] || continue handles the case where the glob matches nothing and stays as the literal *. cp -- "$f" stops a file name that starts with - from being read as an option. cd ... || exit 1 is explicit.

ShellCheck also checks your dialect. Run on the #!/bin/sh file from the start of this lesson, it said:

posix.sh line 2: SC3010 (warning): In POSIX sh, [[ ]] is undefined.posix.sh line 3: SC3030 (warning): In POSIX sh, arrays are undefined.

If a rule is wrong for your case, silence it on one line, with a reason:

# shellcheck disable=SC2086   # we want word splitting: $x holds several flagsls $x /dev/null

I ran it with that comment above the line and ShellCheck gave no output. Use it rarely. Also, shellcheck -f gcc file prints one line per finding in a compiler-like format, for editors, and -S warning hides the "style" and "info" levels.

Two other built-in helpers: bash -n script.sh only checks the syntax (it printed nothing and returned 0 for the bad file above, because that file was valid bash, just risky; so it is far weaker than ShellCheck), and bash -x script.sh prints each command as it runs:

$ bash -x dbg.sh+ x=5+ '[' 5 -gt 3 ']'+ echo bigbig

The + lines are the trace, after expansion. That is the quickest way to see how your quoting really worked. In a longer script, set PS4='+ line $LINENO: ' to show line numbers.

Run, source, and the child-process rule

There is one more idea that confuses beginners. When you run a script (./x.sh or bash x.sh), it runs in a child process. It can read your exported variables, but anything it sets is lost when it ends. When you source it (. ./x.sh or source x.sh), the commands run in your current shell:

printf 'FOO=fromfile\n' > envf.shFOO=origbash envf.sh;  echo "after run: $FOO". ./envf.sh;   echo "after source: $FOO"

after run: origafter source: fromfile

This is how .env files and virtualenv activation work: they must be sourced, otherwise they would set variables in a child that dies at once. The same rule explains why a script that does cd somewhere does not change your terminal's folder.

Background jobs and wait

For small parallel work, start jobs with & and collect them with wait:

for i in 1 2 3; do (sleep 0.2; echo "job $i done") & donewaitecho "all done"

job 1 donejob 2 donejob 3 doneall done

(The three "job" lines can come in any order; I sorted my output to show it clearly. wait with no arguments waits for every background child.) For real concurrency control, use xargs -P N (lesson 3) or a tool like GNU parallel (not installed here).

A complete example

Here is a small, real script that uses most of this lesson. It counts HTTP status codes in one or more access logs, with options, checks, a temp file and cleanup. I made a 5-line fake log, a.log, with documentation-range IP addresses.

#!/usr/bin/env bash# logsum.sh - count HTTP status codes in one or more access logs.set -euo pipefailusage() {  cat >&2 <<'USAGE'usage: logsum.sh [-t N] [-q] LOGFILE...  -t N   show only the top N codes (default: all)  -q     quiet: no header lineUSAGE  exit 2}top=0quiet=0while getopts ":t:qh" opt; do  case $opt in    t) top=$OPTARG ;;    q) quiet=1 ;;    *) usage ;;  esacdoneshift $((OPTIND - 1))[[ $# -ge 1 ]] || usage[[ $top =~ ^[0-9]+$ ]] || { echo "error: -t needs a number" >&2; exit 2; }tmp=$(mktemp)trap 'rm -f "$tmp"' EXITfor log in "$@"; do  if [[ ! -r $log ]]; then    echo "error: cannot read $log" >&2    exit 1  fi  awk '{print $9}' "$log" >> "$tmp"done(( quiet )) || echo "status counts from $# file(s):"if (( top > 0 )); then  sort "$tmp" | uniq -c | sort -rn | head -n "$top"else  sort "$tmp" | uniq -c | sort -rnfi

ShellCheck gave no output (clean, exit 0). Runs:

$ ./logsum.sh a.logstatus counts from 1 file(s):      3 200      1 500      1 404$ ./logsum.sh -q -t 1 a.log a.log      6 200$ ./logsum.shusage: logsum.sh [-t N] [-q] LOGFILE...  -t N   show only the top N codes (default: all)  -q     quiet: no header line$ ./logsum.sh -t x a.log; echo rc=$?error: -t needs a numberrc=2$ ./logsum.sh nope.log; echo rc=$?error: cannot read nope.logrc=1

After all those runs there were no leftover /tmp/tmp.* files, so the trap ... EXIT worked, even for the run that failed on a missing file. The structure is one you can copy: strict mode, a usage function, getopts, validate input, make a temp file, trap its removal, do the work, print results.

A checklist for every script

1. Shebang #!/usr/bin/env bash for bash features (or #!/bin/sh and POSIX only).
2. set -euo pipefail near the top.
3. Quote every expansion: "$var", "$(cmd)", "$@".
4. Use ${var:?} before dangerous commands and || exit after cd.
5. Use [[ ]] for tests, (( )) for numbers, case for many choices.
6. Loop over files with globs (for f in *.txt), never over ls output.
7. Use local in functions. Declare and assign separately when the value comes from a command.
8. Send errors to stderr (>&2), and exit with a non-zero code on failure.
9. Make temp files with mktemp and remove them with trap ... EXIT.
10. Run shellcheck before you commit, and in CI.

Try it

1. Write greet.sh that takes a name as $1, prints Hello, NAME, and uses Hello, stranger if no name is given. Run it with Ada and with "Ada Lovelace".
2. Make a file called my notes.txt. Write one loop that prints every .txt file name in square brackets, once the wrong way ($(ls)) and once the right way (glob). What is different?
3. Write a script with set -euo pipefail that does cd /no/such/dir and then echo done. What do you see, and what is the exit status?
4. Write a script that creates a temp directory, writes a file into it, then exits with an error on purpose. Prove with ls that the directory is gone afterwards.
5. Run shellcheck on this line and explain the warning: rm -rf $HOME_BACKUP/*.
6. Why does count=0; seq 3 | while read -r n; do count=$((count+n)); done; echo $count print 0? Change it to print 6.
7. Under set -e, run (( a-- )) with a=5, then a=1, then a=0. Which one kills the script, and why?

Answers

1. name=${1:-stranger}; echo "Hello, $name". With "Ada Lovelace" as one quoted argument, $1 holds both words. Without quotes on the command line, $1 would only be Ada.
2. for f in $(ls *.txt) splits my notes.txt into my and notes.txt. for f in *.txt keeps the name whole.
3. With -e, bash prints the cd error and exits at once with status 1. done is never printed. Without -e it would print done and exit 0.
4. Use tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT; echo x > "$tmp/f"; exit 1. Afterwards ls "$tmp" (print the name first) says "No such file or directory".
5. SC2115 (and SC2086). If the variable is empty, the command becomes rm -rf /*. Use rm -rf "${HOME_BACKUP:?}"/*.
6. The while loop is on the right side of a pipe, so it runs in a subshell and its count is thrown away. Use while read -r n; do ...; done < <(seq 3) and you get 6.
7. Only the second case with value zero kills it. (( expr )) returns status 0 when the value of the expression is non-zero, and status 1 when the value is zero. (( a-- )) gives the old value of a. I ran it under set -e with a=5 and a=1: both survived (status 0). With a=0 the value is 0, the status is 1, and the script died. That is the same reason (( n++ )) failed when n was 0 earlier.

Interview questions

Key takeaways

• A script is a file with a shebang. Use #!/usr/bin/env bash for bash, and know that /bin/sh is dash on Debian and Ubuntu.
• Quote every expansion ("$var", "$@"). Unquoted values are split into words and wildcards expand.
• Exit status 0 is success. if runs a command and checks its status. A && B || C is not if/else.
• Use [[ ]] for tests, -gt/-lt for numbers (not >/<, which compare text), and case for many choices.
• Loop over files with a glob, never over ls. Read lines with while IFS= read -r line. A loop at the end of a pipe runs in a subshell.
• Functions: local for variables, return for status, print for data, >&2 and exit 1 for errors.
• set -euo pipefail fixes the worst defaults, but set -e has real exceptions (functions in if, local x=$(...), (( )), let).
• trap cleanup EXIT removes temp files on every kind of exit except SIGKILL.
• Run ShellCheck on every script and in CI. Use bash -x to see what actually ran.
• Not tested here: bash on macOS (3.2, no associative arrays), long-option parsers, shopt -s lastpipe, GNU parallel.

What's next

Next is SSH, Networking Commands and Transferring Files. Scripts become more powerful when they run on other machines, so you will learn SSH keys, scp, rsync, tunnels and curl. Practise now: take one command sequence you repeat at work, turn it into a script with set -euo pipefail and a usage function, and run shellcheck on it.