Text Processing · about 50 min

Answering questions with pipelines

Why this matters

You now have the full classic toolkit: grep, cut, sort, uniq, tr, wc, sed and awk, connected with pipes. This lesson is about using them together on a realistic problem: a web server log and a few questions a system administrator actually asks. The point is less any single command than the method: how to go from a question in English to a pipeline that answers it, one verified step at a time.

The data

Open the shell for this lesson. /work/access.log is in the common log format used by Apache, nginx and most other web servers:

$ head -n 2 access.log
10.0.0.5 - - [30/Oct/2026:08:00:01 +0000] "GET / HTTP/1.1" 200 5120
192.168.1.20 - - [30/Oct/2026:08:00:07 +0000] "GET /about HTTP/1.1" 200 2048
$ wc -l < access.log
19

One request per line. Split on spaces, the fields are:

 $1           $2 $3 $4                     $5      $6    $7  $8         $9  $10
 10.0.0.5     -  -  [30/Oct/2026:08:00:01  +0000]  "GET  /   HTTP/1.1"  200 5120
 client IP          time                           method path protocol status bytes

(The - fields are an identity and a user name that are almost never filled in.) Before anything else, check your field numbers on real lines:

$ awk '{ print $1, $7, $9 }' access.log | head -n 3
10.0.0.5 / 200
192.168.1.20 /about 200
10.0.0.5 /products 200

Method: from a question to a pipeline

Write the question precisely. "Top visitors" becomes "the 3 IPs with the most lines, with their counts, largest first".
Extract only the data the question is about (awk '{print $1}', cut, grep -o).
Filter to the relevant lines (grep, an awk pattern).
Transform if needed (tr, sed).
Aggregate (sort | uniq -c, sort -u | wc -l, awk sums).
Order and trim (sort -nr, head).
Run after each stage and look at the output. Use head while building so you do not flood the screen.

The extraction step first keeps every later step simple: tools further down the pipe never see the irrelevant columns.

Question 1: who sends the most requests?

$ awk '{ print $1 }' access.log | head -n 3
10.0.0.5
192.168.1.20
10.0.0.5
$ awk '{ print $1 }' access.log | sort | uniq -c
      7 10.0.0.5
      2 10.0.0.8
      4 172.16.0.9
      5 192.168.1.20
      1 203.0.113.7
$ awk '{ print $1 }' access.log | sort | uniq -c | sort -nr | head -n 3
      7 10.0.0.5
      5 192.168.1.20
      4 172.16.0.9

Look at the order in the middle output: 172.16.0.9 comes before 192.168.1.20, which happens to match numeric order, but only by luck. sort compared the addresses as text, character by character, so 10.0.0.100 would sort before 10.0.0.5 (because 1 < 5). For IPs in true numeric order, sort each dotted part as a number: sort -t. -k1,1n -k2,2n -k3,3n -k4,4n. The lesson: text order is rarely the order a human expects for numbers, versions or addresses.

Question 2: how many different clients?

$ awk '{ print $1 }' access.log | sort -u | wc -l
5

Question 3: what is broken?

Failed requests have status 4xx (client errors, like 404 Not Found) or 5xx (server errors).

$ awk '$9 >= 400' access.log | wc -l
7
$ awk '$9 >= 500 { print $7 }' access.log | sort | uniq -c
      2 /checkout
$ awk '$9 == 404 { print $7 }' access.log | sort | uniq -c | sort -nr
      3 /favicon.ico
      1 /wp-login.php
      1 /old-page

Two 500 errors on /checkout deserve a look at the application log. The 404s are harmless: browsers request /favicon.ico automatically, and /wp-login.php is a bot probing for WordPress.

Question 4: how much data per status code?

Sums are awk's territory:

$ awk '{ bytes[$9] += $10 } END { for (s in bytes) print s, bytes[s] }' access.log | sort -n
200 49664
302 0
404 2560
500 2048

Question 5: requests per minute

The timestamp is in field 4, like [30/Oct/2026:08:03:40. Characters 14 to 18 of that field are 08:03. awk's substr(string, start, length) cuts it out:

$ awk '{ print substr($4, 14, 5) }' access.log | uniq -c
      3 08:00
      2 08:01
      3 08:02
      3 08:03
      3 08:04
      3 08:05
      2 08:06

Here uniq -c without sort is correct, because the log is already in time order, and that is the order we want to keep. Know when you need the sort, rather than adding it by reflex.

Ties and determinism

When two items have the same count, which comes first? With sort -nr, GNU sort breaks ties by comparing the whole lines (reversed, because of -r), which is deterministic but not something you chose. When a result must be reproducible, for a report, a test, or the exercise below, say explicitly how ties are ordered by giving sort more keys:

sort | uniq -c | sort -k1,1nr -k2,2

"Key 1 numeric descending, then key 2 ascending as text." -k1,1nr applies n and r to that key only, so the second key is still ascending.

Choosing the tool

You need to...	Reach for
keep lines containing text or a regex	`grep` (`-v`, `-c`, `-o`, `-E`)
pick a column with one clean delimiter	`cut -d -f`
pick columns separated by runs of spaces, or compare numbers	`awk`
change characters, case, squeeze	`tr`
replace text, delete or print line ranges	`sed`
order	`sort` (`-n -r -k -t -u`)
count duplicates	`sort \| uniq -c`
count lines	`wc -l`
sum, average, group	`awk` with variables and arrays

When a pipeline grows past what fits comfortably on one line, or needs real logic, that is the signal to write a script, which is where this course goes next.

Common mistakes

Wrong field number. Always print the fields on a few lines first. Quoted strings with spaces inside (like the request) occupy several fields when splitting on spaces.
Forgetting the first sort before uniq -c. Counts come out split.
Leaving uniq -c padding in output files. Pipe through awk '{print $1, $2}' if the format matters.
Undefined tie order. Add a second sort key when results must be reproducible.
Building the whole pipeline in one go. When the final output is wrong, you cannot tell which stage broke. Build and check stage by stage.

Exercises

Two tasks answering questions about access.log, a quiz on the double sort, and a word frequency script with exact tie rules.

Recap

Turn the question into a precise statement, then extract, filter, transform, aggregate, order and trim, checking after every stage.
sort | uniq -c | sort -nr | head gives the top N; sort -u | wc -l counts distinct values.
awk patterns filter on field values ($9 == 404); awk arrays aggregate (bytes[$9] += $10).
Make tie order explicit with extra sort keys (-k1,1nr -k2,2).
Text order is not numeric order for numbers, versions or IP addresses.

Going further: generate a bigger log with a loop once you know loops (module 10), and time your pipeline with time. Then try sort -t. -k1,1n -k2,2n -k3,3n -k4,4n on the IP list.

Exercises

Exercise 1 · task in a shell

Who hits the server most?

Open the shell for this lesson. /work/access.log is a web server log in the common log format (one request per line, the client IP address is the first field).

Save the 3 IP addresses with the most requests into /work/top-ips.txt, most requests first, one per line in the form count ip, for example 7 10.0.0.5. Leading or extra spaces (as printed by uniq -c) are fine.

Hint 1

Build it in stages: first print only the IPs (awk '{ print $1 }' or cut -d' ' -f1), look, then count.

Hint 2

awk '{ print $1 }' access.log | sort | uniq -c | sort -nr | head -n 3 > top-ips.txt

Reference solution

awk '{ print $1 }' access.log | sort | uniq -c | sort -nr | head -n 3 > top-ips.txt

Exercise 2 · task in a shell

Visitors and broken links

Two more questions about /work/access.log:

How many distinct client IP addresses are in the log? Save the bare number in /work/distinct.txt.
Which URLs (the path after GET, e.g. /old-page) returned the status 404? Save each such URL once, sorted alphabetically, one per line, into /work/404s.txt.

Hint for 2: the status code is field 9 and the URL is field 7 when splitting on spaces.

Hint 1

Distinct values: extract the field, sort -u, wc -l. For the URLs, an awk pattern can test the status: $9 == 404.

Hint 2

awk '$9 == 404 { print $7 }' access.log | sort -u > 404s.txt

Reference solution

awk '{ print $1 }' access.log | sort -u | wc -l > distinct.txt
awk '$9 == 404 { print $7 }' access.log | sort -u > 404s.txt

Exercise 3 · quiz

Why sort twice?

In awk '{print $1}' access.log | sort | uniq -c | sort -nr, what is each sort for?

Both sort alphabetically; the second is only there to be safe.
The first groups identical lines together so uniq -c can count them; the second orders the counted lines by their count.
The first removes duplicates; the second counts them.
The first is needed because awk prints in random order; the second sorts by IP address.

Answer

The first groups identical lines together so uniq -c can count them; the second orders the counted lines by their count.

uniq -c only counts runs of adjacent identical lines, so the first sort makes all copies of each IP adjacent. Its output is in alphabetical order of IP, with counts in front. The second sort, -nr, reorders those lines numerically by the count at the start, biggest first. Each tool does one job, and the job of "order by count" only makes sense after counting.

Exercise 4 · write code

Top three words

Write a script that reads text from standard input and prints the 3 most frequent words, one per line, in the form count word with a single space and no leading spaces.

Rules:

Compare words case-insensitively and print them in lowercase.
A word is a run of letters a-z; everything else (spaces, digits, punctuation, newlines) separates words.
Most frequent first; words with the same count are ordered alphabetically.
If there are fewer than 3 distinct words, print all of them.

Example: The cat and the dog. The END, and a cat! prints

3 the
2 and
2 cat

File: script.sh

Starting point

tr -s ' ' '\n' | sort | uniq -c | sort -nr | head -n 3

Example

input

The cat and the dog. The END, and a cat!

expected output

3 the
2 and
2 cat

Example

input

-- b b
A a; c

expected output

2 a
2 b
1 c

Hint 1

Normalize first: lowercase with tr 'A-Z' 'a-z', then tr -cs 'a-z' '\n' turns every run of non-letters into one newline. A leading non-letter leaves an empty first line; drop it with grep -v '^$'.

Hint 2

For the ordering, give sort two keys: count numerically descending, then the word: sort -k1,1nr -k2,2.

Hint 3

Finally head -n 3 and awk '{ print $1, $2 }' to remove uniq's padding.

Reference solution

tr 'A-Z' 'a-z' | tr -cs 'a-z' '\n' | grep -v '^$' | sort | uniq -c | sort -k1,1nr -k2,2 | head -n 3 | awk '{ print $1, $2 }'