Pipes, Redirection and Text Tools: grep, sed, awk, find and xargs

Imagine a 2 GB web server log and a boss who asks: "Which IP addresses made the most failed requests last night?" You do not open the file in an editor. You build a pipeline of five small commands, and you have the answer in two seconds. This lesson teaches that skill.

We use only a tiny fake log (10 lines) so you can see every result. The commands are the same on 10 lines or 10 million. This follows processes in lesson 2, because each stage of a pipeline is a separate process. The next lesson, Shell Scripting, puts these pieces into reusable scripts.

Words used in this lesson

• stdin, stdout, stderr are the three standard streams. They are file numbers 0, 1 and 2 (called file descriptors).
• Redirection sends a stream to or from a file instead of the terminal: >, >>, <, 2>.
• Pipe (|) connects the stdout of one command to the stdin of the next.
• Pipeline is a chain of commands joined by pipes.
• Filter is a command that reads text from stdin and writes changed text to stdout (like grep).
• Regular expression (regex) is a pattern language for matching text. a.c, ^start, [0-9]+.
• Field is a piece of a line split by a separator, such as a space or a comma.
• Here document (<<EOF) feeds a block of lines in your script to a command's stdin.
• Process substitution (<(cmd)) lets a command's output appear as if it were a file.
• Null-separated names use the byte \0 between items, so names with spaces or newlines are safe.

The three streams

When a program starts, three channels are already open:

They are separate on purpose. If your program prints results and also complains about a missing file, you can save the results and still see the complaints. Let me show it. ls is asked for two names. One exists, one does not:

ls access.log nonexist

ls: cannot access 'nonexist': No such file or directoryaccess.log

Two lines, both on the screen. The error line is stderr. The access.log line is stdout. (The error came first only because of buffering. Do not rely on order between the two.) Now redirect:

ls access.log nonexist > out.txtecho "out.txt:"; cat out.txt

ls: cannot access 'nonexist': No such file or directoryout.txt:access.log

> sends only stdout to the file. The error still went to the screen. To catch errors, use 2>:

ls access.log nonexist > out.txt 2> err.txtecho "err:"; cat err.txt

err:ls: cannot access 'nonexist': No such file or directory

Now each stream went to its own file. To send both to the same place, use 2>&1, which reads "send stream 2 to wherever stream 1 goes right now":

ls access.log nonexist > both.txt 2>&1cat both.txt

ls: cannot access 'nonexist': No such file or directoryaccess.log

Bash also has the short form &> that does the same thing (ls ... &> all.txt). I tested it and got the same result.

ls access.log nonexist 2>&1 > only-out.txtcat only-out.txt

ls: cannot access 'nonexist': No such file or directoryaccess.log

The first line is the error that still reached the screen, and the second line is the content of only-out.txt (just stdout). A very common bug in scripts and cron lines.

Overwrite, append, and the bit bucket

• > overwrites (it empties the file first, even if the command then fails).
• >> appends.
• 2>/dev/null throws errors away. /dev/null is the "black hole" file from lesson 0.

echo one > a.txtecho two >> a.txtcat a.txtecho three > a.txtcat a.txtls nonexist 2>/dev/null; echo "status=$?"

onetwothreestatus=2

(The first cat printed one and two. The second printed three, because > replaced the file.) Hiding the error message does not change the exit code: status=2. Use 2>/dev/null when you know the noise is harmless. Do not use it everywhere, or you will hide the real reason a script fails.

Input: <, here documents and here strings

< feeds a file to stdin. These two do the same job, but not the same output:

wc -l < access.logwc -l access.log

1010 access.log

With <, wc does not know a file name, so it prints only the number. That is handy in scripts (n=$(wc -l < file)).

A here document gives a command a block of text from your script. If the closing word is quoted, nothing in the block is expanded. If it is not quoted, variables and $(...) are expanded:

cat <<'EOT'name: $USERhome: $(echo hi)EOTcat <<EOTuser is $(id -un)EOT

name: $USERhome: $(echo hi)user is root

A here string (<<<) gives one string to stdin:

tr a-z A-Z <<< "hello there"

HELLO THERE

What is /dev/stdout?

The device names /dev/stdin, /dev/stdout and /dev/stderr are links to /proc/self/fd/0, 1 and 2:

lrwxrwxrwx 1 root root 15 Oct 12 03:03 /dev/stderr -> /proc/self/fd/2lrwxrwxrwx 1 root root 15 Oct 12 03:03 /dev/stdin -> /proc/self/fd/0lrwxrwxrwx 1 root root 15 Oct 12 03:03 /dev/stdout -> /proc/self/fd/1

That is "everything is a file" again: even your streams have a path, which is how you can write echo oops > /dev/stderr or give a tool a file name that really means "the screen".

Pipes: connect commands

A pipe | takes the stdout of the left command and feeds it into the stdin of the right command. Both run at the same time. The kernel moves the data between them in a small buffer, so nothing is written to disk.

head -3 access.log | cut -d' ' -f1

203.0.113.5203.0.113.5198.51.100.7

head -3 produces three lines. cut -d' ' -f1 cuts each line at spaces (-d' ') and keeps field one (-f1), which is the IP address.

The best way to build a pipeline is one step at a time. Run the first command and look. Add | and the next command. Look again. Never write the whole thing blind.

A pipe carries only stdout

The error messages (stderr) do not go through the pipe. They still go to the terminal:

ls nonexist | wc -lls nonexist |& wc -l

ls: cannot access 'nonexist': No such file or directory01

In the first line, wc counted 0 lines because nothing came through the pipe, and the error was printed to the screen. With |& (short for 2>&1 |), the error message went into the pipe, so wc counted one line.

tee: save and pass on

tee writes its stdin to a file and to stdout. It is like a T-junction in a pipe. Use it to keep a copy of the middle of a pipeline:

ls | head -3 | tee first3.txt | wc -lcat first3.txt

3a.txtaccess.logall.txt

(tee -a appends. And | sudo tee file is how you write a root-owned file from a non-root pipeline, as we saw in lesson 1.)

Pipelines and exit codes

A pipeline's exit code is, by default, the code of the last command. A failure earlier in the chain is hidden:

false | true; echo "no pipefail: $?"set -o pipefailfalse | true; echo "pipefail: $?"set +o pipefailfalse | true | false; echo "${PIPESTATUS[@]}"

no pipefail: 0pipefail: 11 0 1

• By default false | true returns 0, even though false failed.
• set -o pipefail makes the pipeline fail if any stage fails (the code of the rightmost failing one).
• ${PIPESTATUS[@]} (bash only) lists the exit code of every stage. This one shows 1 0 1. Note that it must be read immediately after the pipeline.

We use pipefail in every serious script (lesson 4).

When the reader quits early: SIGPIPE

What happens when the right side closes before the left side is done? Example: yes prints y forever, and head -2 takes two lines and quits.

yes | head -2echo "yes status in pipeline: ${PIPESTATUS[0]}"

yyyes status in pipeline: 141

yes is not stuck forever. When head exits, the next write by yes hits a broken pipe, and the kernel sends it SIGPIPE (signal 13). 141 is 128 + 13. That is a normal way for pipelines to stop, but under set -o pipefail you can see this 141 as a "failure". It is a known cause of a script that fails "randomly" when the output is cut short with head.

grep: find lines that match

grep PATTERN [files] prints the lines that contain the pattern. It is the tool you use most. Our log:

203.0.113.5 - - [12/Oct/2026:10:00:01 +0000] "GET /index.html HTTP/1.1" 200 5120...198.51.100.7 - - [12/Oct/2026:10:00:06 +0000] "POST /api/login HTTP/1.1" 401 64

Each line is: IP, two dashes, time, the request in quotes, the status code, and the size in bytes.

grep 'POST' access.loggrep -c ' 404 ' access.loggrep -n '500' access.log | cut -c1-60

198.51.100.7 - - [12/Oct/2026:10:00:06 +0000] "POST /api/login HTTP/1.1" 401 64198.51.100.7 - - [12/Oct/2026:10:00:15 +0000] "POST /api/login HTTP/1.1" 200 30026:203.0.113.5 - - [12/Oct/2026:10:00:11 +0000] "GET /api/use9:203.0.113.5 - - [12/Oct/2026:10:00:20 +0000] "GET /api/ord

Options you will use all the time:

Some real runs:

grep -v 'GET' access.log | cut -c1-70grep -o '[0-9]\{1,3\}\(\.[0-9]\{1,3\}\)\{3\}' access.log | sort | uniq -c | sort -rngrep -E '" (4|5)[0-9]{2} ' access.log | awk '{print $9, $7}'

198.51.100.7 - - [12/Oct/2026:10:00:06 +0000] "POST /api/login HTTP/1.198.51.100.7 - - [12/Oct/2026:10:00:15 +0000] "POST /api/login HTTP/1.      4 203.0.113.5      4 198.51.100.7      2 192.0.2.44401 /api/login404 /api/users/42500 /api/users404 /missing.png500 /api/orders

The second command pulls out only the IP addresses (-o) and counts them. The third finds lines where the status code, after the closing quote, starts with 4 or 5, which means client or server errors.

Regex survival kit

The most common beginner trap is the dot. In a regex . means "any character", so 192.168.1.1 also matches 192x168y1z1. Compare grep with grep -F:

printf 'a.b\naxb\n' | grep 'a.b'printf 'a.b\naxb\n' | grep -F 'a.b'

a.baxba.b

The first matched both lines. The second (-F, fixed string) matched only the real a.b. When you search for a literal piece of text such as an IP, an error message or a filename, use -F. It is also faster.

-w helps with partial words:

echo "cat concatenate category" | grep -ow 'cat'

cat

Only the whole word cat was matched once, not the cat inside the other two words.

Grep exit codes

grep returns 0 if it found something, 1 if it found nothing, and 2 on a real error (like a missing file). This is how if grep -q ...; then works in scripts:

grep -q nothing access.log; echo "nomatch=$?"grep -q POST access.log;    echo "match=$?"grep POST nonexist 2>/dev/null; echo "error=$?"

nomatch=1match=0error=2

For searching code, grep -rn 'TODO' --include='*.js' --exclude-dir=node_modules . is the daily form. (Tools like ripgrep are faster, but grep is on every server.)

sort, uniq, cut, tr, wc: the small helpers

These do tiny jobs that, together, answer big questions.

• sort sorts lines. -n sorts as numbers, -r reverses, -u removes duplicates, -k N sorts by field N, -t, sets the separator.
• uniq collapses adjacent duplicate lines. -c adds counts. Because it only sees neighbours, you must sort first.
• cut -d, -f2 picks fields. tr translates or deletes single characters. wc -l counts lines.

Sorting as text vs numbers is a trap:

printf '10\n9\n100\n' | sortprintf '10\n9\n100\n' | sort -n

101009910100

As text, 10 comes before 9 (the character 1 is smaller than 9). Use -n for numbers.

And uniq needs sorted input:

printf 'b\na\nb\na\nc\n' | uniqprintf 'b\na\nb\na\nc\n' | sort | uniq -c

babac      2 a      2 b      1 c

uniq alone removed nothing, because equal lines were not neighbours. After sort, uniq -c counted them.

The most useful idiom in this lesson

... | sort | uniq -c | sort -rn | head

It means "count how often each line appears, show the most frequent first". Apply it to our log. Which requested paths are most common (field 7)? Which status codes occur?

awk '{print $7}' access.log | sort | uniq -c | sort -rn | head -4awk '{print $9}' access.log | sort | uniq -ccut -d' ' -f1 access.log | sort -u | wc -l

      2 /api/users      2 /api/orders      2 /api/login      1 /static/app.js      5 200      1 401      2 404      2 5003

Five requests succeeded (200), two were 404, two were 500, one was 401, and there were 3 different client IPs.

tr handles characters:

echo 'Hello World' | tr 'A-Z' 'a-z'echo 'a,b,,c' | tr -s ','printf 'a\r\nb\r\n' | tr -d '\r' | od -c | head -2

hello worlda,b,c0000000   a  \n   b  \n0000004

-s squeezes repeated characters into one, and -d deletes. Removing \r is the standard cure for files saved on Windows (CRLF line endings) that break scripts with weird ^M errors.

sed: edit a stream

sed ("stream editor") reads lines, applies edit commands, and writes them out. 90% of its use is the substitute command: s/old/new/.

echo "hello world hello" | sed 's/hello/bye/'echo "hello world hello" | sed 's/hello/bye/g'echo "hello world hello" | sed 's/hello/bye/2'

bye world hellobye world byehello world bye

Without a flag sed replaces only the first match on each line. g means all matches, and a number means only that one. When the text contains slashes, pick another delimiter. # works well:

echo "/usr/local/bin" | sed 's/\/usr/\/opt/'echo "/usr/local/bin" | sed 's#/usr#/opt#'

/opt/local/bin/opt/local/bin

Printing and deleting lines by address:

sed -n '2,3p' access.log | cut -c1-50sed '/GET/d' access.log | cut -c1-50sed -n '/orders/p' access.log | wc -l

203.0.113.5 - - [12/Oct/2026:10:00:02 +0000] "GET 198.51.100.7 - - [12/Oct/2026:10:00:05 +0000] "GET198.51.100.7 - - [12/Oct/2026:10:00:06 +0000] "POS198.51.100.7 - - [12/Oct/2026:10:00:15 +0000] "POS2

• -n turns off automatic printing, and p prints. So -n '2,3p' prints lines 2 to 3 only.
• /GET/d deletes lines matching GET. What is left are the two POST lines.

(The first output above shows two lines from the range command, then two lines from the delete command, then the count.) A few more forms:

echo "2026-10-12" | sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/'printf '# c\nkey=1\n\nkey2=2\n' | sed '/^#/d;/^$/d'echo "price 5" | sed 's/[0-9]\+/[&]/'

12/10/2026key=1key2=2price [5]

With -E, parentheses make groups, and \1, \2, \3 refer back to them, so the date was rewritten as day/month/year. /^#/d;/^$/d removes comment lines and empty lines. & in the replacement means "the whole matched text".

In-place editing: sed -i

-i changes the file directly. It is powerful and dangerous. Give it a suffix to keep a backup:

cp people.csv p2.csvsed -i 's/eng/engineering/' p2.csvhead -3 p2.csvsed -i.bak 's/ops/operations/' p2.csvls p2.csv*

name,team,salaryada,engineering,120bob,ops,95p2.csvp2.csv.bak

awk: work with fields

awk reads each line, splits it into fields ($1, $2, ..., $NF for the last, $0 for the whole line), and runs code on lines you select. The basic shape is awk 'PATTERN { ACTION }'. If you leave out the pattern, the action runs on every line. If you leave out the action, matching lines are printed.

awk '{print $1, $9}' access.log | head -2awk -F, 'NR>1 {print $1 " earns " $3}' people.csv | head -2

203.0.113.5 200203.0.113.5 200ada earns 120bob earns 95

• Fields are split on whitespace by default. -F, sets the separator to a comma.
• NR is the current line number. NR>1 skips the header line of the CSV.
• NF is the number of fields in the line, and $NF is the last one.

Filtering and counting:

awk '$9 >= 500' access.log | cut -c1-80awk '$9 == 404 {c++} END {print c " not found"}' access.log

203.0.113.5 - - [12/Oct/2026:10:00:11 +0000] "GET /api/users HTTP/1.1" 500 130203.0.113.5 - - [12/Oct/2026:10:00:20 +0000] "GET /api/orders HTTP/1.1" 500 1302 not found

A BEGIN block runs once before the first line, and an END block runs once after the last line. Variables start at zero (empty), so you can sum without setup:

awk -F, 'NR>1 {sum+=$3} END {print "total", sum, "avg", sum/(NR-1)}' people.csvawk -F, 'NR>1 {t[$2]+=$3; n[$2]++} END {for (k in t) print k, t[k], n[k]}' people.csv | sortawk '{b[$1]+=$10} END {for (i in b) print i, b[i]}' access.log | sort -k2 -nr

total 555 avg 111eng 360 3ops 195 2203.0.113.5 25860198.51.100.7 5272192.0.2.44 104

The arrays are associative: t[$2] uses the team name as a key. The second command sums the salary by team, and the third sums the bytes sent to each IP address. This "group by" with a few words is the reason people love awk.

Formatted output with printf:

awk -F, 'NR>1 && $3>100 {printf "%-5s %5d\n", $1, $3}' people.csv

ada     120cy      110eve     130

A small error-rate calculation: how many of the 10 requests failed (status 400 or above)?

awk '{t++} $9>=400 {e++} END {printf "%d of %d requests failed (%.0f%%)\n", e, t, 100*e/t}' access.log

5 of 10 requests failed (50%)

Two lesser-known facts: awk '!seen[$0]++' prints each line only the first time it appears, which removes duplicates while keeping order (unlike sort -u):

printf 'b\na\nb\nc\na\n' | awk '!seen[$0]++'

bac

And the awk on this machine is mawk. GNU awk (gawk) adds extras such as gensub, asort and --csv handling. If you copy a recipe from the web and mawk says "function not defined", that is the reason. Check with awk -W version (mawk) or awk --version (gawk). The examples in this lesson use only POSIX features and ran in mawk.

For CSV with quoted commas (a field like "Smith, Ada"), none of cut, awk -F, or sed is safe. Use a real CSV tool (csvkit, mlr, or a Python/Node script).

find: search the file tree

grep searches file contents. find searches file names and details. I made a small project with these files: src/a.js, src/b.js, src/c.ts, node_modules/lib/index.js, docs/README.md (10 days old), my docs/to do.txt (a name with spaces), run.sh (executable), big.bin (3000 bytes) and empty.log.

find . -name '*.js' | sortfind . -name '*.js' -not -path './node_modules/*' | sortfind . -path ./node_modules -prune -o -name '*.js' -print | sort

./node_modules/lib/index.js./src/a.js./src/b.js./src/a.js./src/b.js./src/a.js./src/b.js

(That is three runs: the first found three files, and the next two skipped node_modules.) Always put the pattern in quotes, so the shell does not expand *.js itself before find sees it. -prune is the efficient way to not even enter a folder.

Useful tests (all can be combined, and by default they are joined with AND):

Examples with their real results:

find . -type f -size +1kfind . -emptyfind . -type f -perm -u+xfind . -type f -mtime +7find . -type f -mmin +90 | sortfind . -iname 'readme*'

./big.bin./empty.log./run.sh./docs/README.md./docs/README.md./src/b.js./docs/README.md

Reading the output in order: big.bin is over 1 KB; empty.log is empty; run.sh is executable; README.md is more than 7 days old; files older than 90 minutes are README.md and src/b.js (I back-dated b.js by two hours); and -iname found the README in spite of its capital letters.

Doing something with the results

Two ways: -exec runs a command per file, and xargs builds command lines.

find . -name '*.js' -exec wc -l {} \; | sort -k2find . -name '*.js' -exec wc -l {} + | tail -1find . -name '*.log' -delete

1 ./node_modules/lib/index.js1 ./src/a.js1 ./src/b.js 3 total

• -exec cmd {} \; runs the command once per file. {} is replaced by the name.
• -exec cmd {} + packs many files into one command (like xargs), which is much faster for many files. The last line shows one wc call and its 3 total.
• -delete deletes matches. Always run the same find with -print first.

xargs: turn lines into arguments

Many commands (like rm, ls, cp) take file names as arguments, not from stdin. A pipe sends text to stdin. xargs bridges the gap: it reads items from stdin and puts them on the command line of another command.

printf '1\n2\n3\n4\n5\n' | xargsprintf '1\n2\n3\n4\n5\n' | xargs -n 2printf 'a\nb\n' | xargs -I{} echo "file-{}.txt"printf 'x\ny\n' | xargs echo rm

1 2 3 4 51 23 45file-a.txtfile-b.txtrm x y

• Plain xargs puts all items on one line (it runs echo by default).
• -n 2 uses at most 2 items per run.
• -I{} runs once per item and replaces {}.
• Putting echo in front (xargs echo rm) is a dry run: you see the command it would run. I always do this first.

Names with spaces: the biggest trap

Our project has a file my docs/to do.txt. A naive pipe breaks it, because xargs splits on spaces:

find . -name '*.txt' | xargs ls -l

ls: cannot access './my': No such file or directoryls: cannot access 'docs/to': No such file or directoryls: cannot access 'do.txt': No such file or directory

One file name became three wrong ones. Imagine | xargs rm: it would try to delete files named ./my, docs/to and do.txt. If any of those happened to exist, they would be gone. The fix is null-separated names. find -print0 ends each name with a zero byte, and xargs -0 splits on it. File names cannot contain a zero byte, so this is always safe:

find . -name '*.txt' -print0 | xargs -0 ls -l | cut -c1-60

-rw-r--r-- 1 root root 7 Oct 12 03:51 ./my docs/to do.txt

You can see the \0 with od -c:

0000000   .   /   m   y       d   o   c   s   /   t   o       d   o   .0000020   t   x   t  \0

The same safe pattern in a loop uses read -d '':

find . -name '*.txt' -print0 | while IFS= read -r -d '' f; do echo "got: [$f]"; done

got: [./my docs/to do.txt]

Parallel runs and empty input

xargs -P N runs N commands at the same time. I ran three one-second jobs two ways:

printf '1\n2\n3\n' | xargs -P 3 -I{} sh -c 'sleep 1; echo {}'     # real 0m1.010sprintf '1\n2\n3\n' | xargs -P 1 -I{} sh -c 'sleep 1; echo {}'     # real 0m3.019s

With -P 3, the three jobs overlapped and took 1 second. With -P 1, they ran one after the other and took 3 seconds. Timings are from my run, yours will vary a little, but the ratio is the point. Note that output from parallel jobs can arrive in any order.

By default, GNU xargs runs the command once even when the input is empty, which can be dangerous:

printf '' | xargs echo "ran"printf '' | xargs -r echo "ran"

ran

The first one ran echo ran with no items. The second (-r, or --no-run-if-empty) did nothing. Imagine find ... | xargs rm -rf, but with an empty input where the command becomes just rm -rf... it fails harmlessly here, yet other commands (like xargs ls) will list the current directory, which is not what you meant. Use -r on GNU systems.

xargs also splits very long lists to respect the kernel's limit for command length. On this machine:

Maximum length of command we could actually use: 2078776Size of command buffer we are actually using: 131072

That is why rm $(find ...) can fail with "Argument list too long" on huge folders while find ... -print0 | xargs -0 rm works: xargs breaks the job into batches.

Process substitution and friends

<(command) gives a command's output as a temporary file name. Many tools want two files, and this lets you compare two outputs without making temporary files:

diff <(printf 'a\nb\n') <(printf 'a\nc\n')comm -12 <(printf 'a\nb\nc\n') <(printf 'b\nc\nd\n')paste -d, <(printf '1\n2\n') <(printf 'x\ny\n')

2c2< b---> cbc1,x2,y

diff shows what changed, comm -12 prints lines common to two sorted inputs, and paste joins files side by side. Process substitution is a bash feature (also in zsh), but not in plain sh.

Another tool I reach for is jq for JSON, which is installed here:

echo '{"a":[1,2,{"b":"x"}]}' | jq -r '.a[2].b'

x

Do not parse JSON with grep and sed. Use jq.

Recipes that use everything

A few real one-liners on our fake log:

# 1. Top client IPsawk '{print $1}' access.log | sort | uniq -c | sort -rn | head -3# 2. Which paths returned 5xx, and how many times?awk '$9 ~ /^5/ {print $7}' access.log | sort | uniq -c# 3. Bytes sent per status codeawk '{b[$9]+=$10} END{for(s in b) print s, b[s]}' access.log | sort

      4 203.0.113.5      4 198.51.100.7      2 192.0.2.44      1 /api/orders      1 /api/users200 30808401 64404 104500 260

More small ones, also run for real:

# Rename *.txt to *.md (safe with spaces, using "--" and quotes)for f in *.txt; do mv -- "$f" "${f%.txt}.md"; done# Show only the last part of a pathecho "/a/b/c.txt" | awk -F/ '{print $NF}'     # c.txtbasename /a/b/c.txt; dirname /a/b/c.txt         # c.txt and /a/b# Count lines of real JS code, skipping node_modulesfind proj -name '*.js' -not -path '*/node_modules/*' -print0 | xargs -0 cat | wc -l    # 2

${f%.txt} removes the suffix .txt from the variable's value. This is a bash feature, so no extra process is needed. It is the style the pure-bash-bible teaches for many string jobs.

When not to use a one-liner

A pipeline is excellent for quick questions on text. Switch to a script or a "real" language when:

• the logic has more than three or four steps, or needs state;
• the data is JSON, XML or CSV with quotes (use jq, xmllint, csvkit, or Python/Node);
• you need to keep the code and test it. Put it in a file with a name and comments.

Try it

Make a folder and recreate the log (copy the 10 lines from this lesson into access.log).

1. Print how many requests each different status code got, most frequent first.
2. Print the IP addresses that caused at least one 500.
3. Run ls access.log nonexist > out.txt and then cat out.txt. Where did the error go? Redo it so that both messages end up in out.txt.
4. Replace every GET with FETCH in a copy of the log, using sed. Keep a backup with -i.bak. Show the difference with diff access.log access.log.bak | head.
5. Using awk, print the total of the last column (bytes).
6. Make a file named a b.txt and a file named c.txt. Delete only .txt files using find, -print0 and xargs -0. First do a dry run with echo.
7. Run false | true; echo $?, then enable set -o pipefail and run it again. Explain the two results.
8. Explain why sort data.txt > data.txt loses your data, and show a safe way.

Answers

1. awk '{print $9}' access.log | sort | uniq -c | sort -rn. The result is 5 200, then 2 404 and 2 500 (in some order), then 1 401.
2. awk '$9==500 {print $1}' access.log | sort -u gives 203.0.113.5. (Or grep ' 500 ' access.log | cut -d' ' -f1 | sort -u.)
3. The error went to the terminal, because > redirects only stdout. Use ls access.log nonexist > out.txt 2>&1 (or &>).
4. cp access.log copy.log && sed -i.bak 's/GET/FETCH/' copy.log, then diff copy.log copy.log.bak | head. (Careful with the file names: the backup is named after the file you edit.)
5. awk '{s+=$10} END {print s}' access.log prints 31236. (5120+20480+812+64+52+130+300+52+130+4096.)
6. Dry run: find . -name '*.txt' -print0 | xargs -0 echo rm. Real: find . -name '*.txt' -print0 | xargs -0 rm. A plain xargs would split a b.txt into a and b.txt.
7. The first prints 0 because a pipeline returns the status of its last command. With pipefail, the second prints 1, because false failed earlier in the pipeline.
8. The shell opens and empties data.txt for the > before sort starts, so sort reads an empty file. Safe ways: sort data.txt > data.sorted && mv data.sorted data.txt, or sort -o data.txt data.txt.

Interview questions

Key takeaways

• Every process has stdin (0), stdout (1) and stderr (2). > and >> write stdout, 2> writes stderr, 2>&1 merges them (put it after the >), < reads a file, /dev/null discards.
• command > file empties the file first. Never read and write the same file in one command.
• A pipe | carries only stdout, runs both sides at once, and returns the last command's exit code unless pipefail is on. Use tee to save the middle of a pipeline.
• grep searches lines (-i -v -c -n -o -w -r -l -E -F -q). It exits 0 on a match, 1 on none, 2 on error. Use -F for literal text.
• sort | uniq -c | sort -rn | head is the "top N" idiom. uniq needs sorted input. Use sort -n for numbers.
• sed 's/old/new/g' edits streams. -i edits files in place (and differs between GNU and macOS). awk handles fields, filters, sums and group-by, and its comparison can fall back to strings.
• find searches by name, type, size, time and permissions. Use -print0 | xargs -0 (or -exec ... +) for safe batch work, xargs -r to avoid empty runs, and a dry run with echo first.
• Build pipelines one stage at a time, and filter early.

What's next

Next: Shell Scripting: Variables, Conditions, Loops and Functions, Lesson 4. You will turn these one-liners into safe, reusable scripts with quoting rules, set -euo pipefail, arguments, if, for, while, functions, trap, and ShellCheck. Practise now: pick a log file or CSV on your machine and answer three questions about it using only the tools from this lesson.