Practical Projects · about 75 min

Project: log analyzer

Why this matters

"What happened on the web server yesterday?" is one of the most common questions an operator asks, and the answer is buried in a log with thousands or millions of lines. Every company has a dashboard for this eventually, but the shell answers it in seconds on any machine, with no setup. In this project you build a real command-line tool, step by step: explore the data by hand, turn the questions into a script, then add options and polish. Along the way you use most of the track: pipelines, awk, sort and uniq, getopts, validation, exit statuses, and strict mode.

The data

Open the shell for this lesson and look:

$ head -n 3 logs/access.log
203.0.113.7 - - [31/Oct/2026:00:00:00 +0000] "GET /index.html HTTP/1.1" 200 200
198.51.100.23 - - [31/Oct/2026:00:00:57 +0000] "GET /cart HTTP/1.1" 200 253
198.51.100.77 - - [31/Oct/2026:00:01:54 +0000] "GET /cart HTTP/1.1" 200 306

This is the Common Log Format used by Apache, nginx and many others. Split on spaces, the fields are:

 $1           $2 $3 $4                     $5      $6     $7          $8        $9   $10
 203.0.113.7  -  -  [31/Oct/2026:00:00:00  +0000]  "GET   /index.html HTTP/1.1" 200  200
 client IP    ident user  date and time    zone    method path        protocol  status bytes

The quoted request contains spaces, so it spans three fields. As long as paths contain no spaces (they are URL-encoded, so a space would be %20), field 9 is always the status and field 10 the response size. A - in the bytes field means no body was sent, for instance with status 304 (Not Modified).

Status codes come in classes: 2xx success, 3xx redirects and "not modified", 4xx client errors (404 not found, 403 forbidden), 5xx server errors. "Errors" in this report means 400 and above.

Step 1: answer the questions by hand

Before writing a script, find each answer interactively. This tells you what the script must do, and gives you expected values to test it against.

$ wc -l < logs/access.log
1500
$ awk '$9 == 404' logs/access.log | wc -l
115
$ cut -d' ' -f1 logs/access.log | sort | uniq -c | sort -rn | head -n 3
    428 203.0.113.7
    258 198.51.100.23
    171 203.0.113.99

Why awk '$9 == 404' and not grep 404? Because grep matches the text anywhere in the line: a byte count of 4048 or a path like /products/404 would also match. Selecting by field is precise. This is the first important lesson of the project: know your fields.

Step 2: design the tool

Decide the interface before the code:

usage: analyze.sh [-n N] LOGFILE

Output goes to standard output in a fixed, line-oriented format, so the report itself can be piped into grep or compared with diff.
Errors go to standard error. Status 2 for usage mistakes, 1 for runtime problems (unreadable file), 0 for success: the conventions from the "writing scripts people can use" lesson.
Robustness: real logs contain junk (truncated lines after a crash, blank lines). The rule "a request has at least 10 fields" filters them out, and it must apply to every number.

Step 3: one pass with awk

awk reads the file once and keeps running totals in variables and associative arrays:

awk '
NF >= 10 {
  n++                         # requests
  ip[$1]++                    # requests per IP; the array keys are the distinct IPs
  if ($9 >= 400) err++
  if ($10 != "-") bytes += $10
}
END {
  printf "requests: %d\nunique ips: %d\nerrors: %d\nbytes: %d\n", n, length(ip), err, bytes
}' "$log"

The pattern NF >= 10 selects lines; the action in braces runs for each.
Variables start empty, which counts as 0 in arithmetic. %d prints an empty variable as 0, which matters when, say, there are no errors at all.
length(ip) is the number of keys in the array (a gawk feature; in strictly POSIX awk you would count them in a loop).
Everything else in the script is bash: argument checks happen before awk runs, so awk never sees a missing file.

Note the single quotes around the awk program: $1, $9 are awk's fields, not shell parameters. The shell must leave them alone. Values go into awk with -v name="$value", never by pasting shell variables into the program text.

Step 4: rankings with sort and uniq

The top-N list is the classic sort | uniq -c | sort idiom:

 awk prints IPs ─▶ sort (groups equal lines) ─▶ uniq -c (counts each group)
               ─▶ sort -k1,1nr -k2,2 (count descending, then IP) ─▶ keep N, reformat

uniq only merges adjacent equal lines, which is why the first sort is needed. The second sort uses two keys: -k1,1nr means "field 1 only, numeric, reversed", and -k2,2 breaks ties by the IP as a string. Specifying a tie-breaker makes the output deterministic, which a test (and any person comparing two reports) needs.

A strict-mode trap: head and SIGPIPE

The obvious way to keep N lines is | head -n "$top". But head exits as soon as it has N lines. The process writing into the pipe then gets the signal SIGPIPE on its next write and dies with status 141. With set -o pipefail, that makes the whole pipeline fail and set -e stops your script, but only on inputs big enough that the writer was still writing when head quit. That is a bug which appears in production and never in testing. Letting awk read everything and print only the first N (awk 'NR <= top') avoids it.

Step 5: options and validation

top=3
while getopts ':n:' opt; do
  case $opt in
    n) top=$OPTARG ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
(( $# == 1 )) || usage
[[ $top =~ ^[0-9]+$ ]] && (( 10#$top >= 1 )) || usage

Validate every input before using it. Without the regex check, -n abc would reach the arithmetic, where bash treats abc as a variable name (empty, so 0), and -n '$(rm -rf ~)' inside (( )) would even be executed as a command substitution. Checking with a regex first is both correct and safe.

Testing the tool

A test log with known answers is worth more than any amount of reading code. The check for this lesson uses one: nine valid requests, one garbage line, one blank line, a - byte count, ties in the ranking. Build your own the same way: write the log, work out the expected report by hand, and compare with diff <(bash analyze.sh test.log) expected.txt.

Common mistakes

grep 404 to count 404s. Matches anywhere in the line; select field 9.
Forgetting the malformed-line rule in one of several pipelines. The numbers disagree with each other.
No tie-breaker in sort. Order of equal counts depends on input order; reports differ from run to run.
Expanding shell variables inside the awk program. awk "NR <= $top" works until a value contains something awk interprets; use -v.
head under pipefail. Intermittent exit status 141 on big inputs.
Validation after use. Check $#, readability and option values before doing any work.

Exercises

Part 1 is exploration in the shell; parts 2 and 3 build /work/analyze.sh in place, and each check runs it against a small test log with known answers plus the error cases. The quiz is about the design.

Recap

Know the log format: field 9 status, field 10 bytes, - for none.
Answer questions by hand first; the answers become test expectations.
One awk pass computes counters and distinct counts with arrays and END.
sort | uniq -c | sort -k1,1nr -k2,2 ranks deterministically.
Validate arguments and options up front; errors to standard error with 1 or 2.
Under pipefail, avoid head cutting a pipeline short; filter with awk instead.

Going further: add -s STATUS to report only requests with a given status, and a "busiest hour" line (the hour is characters 14-15 of field 4: substr($4, 14, 2) in awk). Then run the tool on a real log from your own machine, for example /var/log/nginx/access.log if you have one.

Exercises

Exercise 1 · task in a shell

Part 1: explore the log by hand

/work/logs/access.log is a web server access log for one day (Combined Log Format, without the referrer and user-agent fields). Open the shell and look at it first: head logs/access.log.

Answer three questions with pipelines and write the answers into /work/answers.txt, one per line, in this order:

the total number of requests (lines),
the number of requests that got status 404,
the IP address that made the most requests.

Each line must contain only the answer (a number, a number, an IP address).

Hint 1

wc -l < logs/access.log prints just the number (with a file name argument, wc also prints the name).

Hint 2

Fields are separated by spaces; the status is field 9. awk '$9 == 404' logs/access.log | wc -l.

Hint 3

Busiest IP: cut -d' ' -f1 logs/access.log | sort | uniq -c | sort -rn | head -n 1, then keep only the address with awk '{print $2}'. Group the three commands in { ...; } > answers.txt.

Reference solution

{
  wc -l < logs/access.log
  awk '$9 == 404' logs/access.log | wc -l
  cut -d' ' -f1 logs/access.log | sort | uniq -c | sort -k1,1nr -k2,2 | head -n 1 | awk '{print $2}'
} > answers.txt

Exercise 2 · task in a shell

Part 2: the core report

Create the script /work/analyze.sh, called as bash analyze.sh LOGFILE. It must print exactly these four lines first (more lines may follow later, in part 3):

requests: N
unique ips: N
errors: N
bytes: N

Only lines with at least 10 whitespace-separated fields are requests; ignore any others (blank or garbage lines).
errors counts requests with status 400 or higher.
bytes is the sum of field 10; a - there means 0.

Error handling:

no argument (or more than one): print usage: analyze.sh [-n N] LOGFILE to standard error, exit 2;
a file that does not exist or cannot be read: print analyze.sh: cannot read FILE (with the name given) to standard error, exit 1.

The check runs your script on its own small test log, so test with your own data as well as logs/access.log.

Hint 1

One awk program can compute all four numbers in a single pass: count lines with NF >= 10, collect IPs in an awk array ip[$1]++, and print in an END block.

Hint 2

length(ip) gives the number of keys of an awk array in gawk. Uninitialized awk variables print as empty with %s but as 0 with %d.

Hint 3

Check arguments before running awk: (( $# == 1 )) || usage and [[ -f $log && -r $log ]].

Reference solution

cat > analyze.sh <<'EOF'
#!/bin/bash
set -euo pipefail

usage() {
  echo "usage: analyze.sh [-n N] LOGFILE" >&2
  exit 2
}

(( $# == 1 )) || usage
log=$1
if [[ ! -f $log || ! -r $log ]]; then
  echo "analyze.sh: cannot read $log" >&2
  exit 1
fi

awk '
NF >= 10 {
  n++
  ip[$1]++
  if ($9 >= 400) err++
  if ($10 != "-") bytes += $10
}
END {
  printf "requests: %d\nunique ips: %d\nerrors: %d\nbytes: %d\n", n, length(ip), err, bytes
}' "$log"
EOF

Exercise 3 · task in a shell

Part 3: top IPs, status codes and -n

Extend /work/analyze.sh. After the four core lines it must print the busiest IPs and a status breakdown. For the check's test log the complete output is:

requests: 9
unique ips: 5
errors: 4
bytes: 2400
top 3 ips:
  4 10.0.0.1
  2 10.0.0.2
  1 10.0.0.3
status codes:
  200 3
  302 1
  304 1
  403 1
  404 2
  500 1

top N ips: lists the N busiest IPs as two spaces, the count, a space, the IP. Sort by count (highest first); break ties by IP in plain string order (10.0.0.3 before 10.0.0.4). If there are fewer than N IPs, list them all (the header still says N).
status codes: lists every status that occurs, numerically ascending, as two spaces, the code, a space, the count.
A new option -n N sets how many IPs to list (default 3). N must be a whole number of at least 1; otherwise print the usage line to standard error and exit 2. bash analyze.sh -n 1 file works; so does bash analyze.sh -n 10 file.

As before, ignore lines with fewer than 10 fields everywhere.

Hint 1

Top IPs: awk 'NF >= 10 {print $1}' "$log" | sort | uniq -c | sort -k1,1nr -k2,2 gives COUNT IP lines in the right order. Then keep N of them and reformat.

Hint 2

Under set -o pipefail, head can make the commands before it die of SIGPIPE on big inputs and fail the pipeline. awk -v top="$top" 'NR <= top {...}' reads everything and avoids that.

Hint 3

Options: while getopts ':n:' opt; do case $opt in n) top=$OPTARG ;; *) usage ;; esac; done; shift $((OPTIND - 1)), then validate $top with [[ $top =~ ^[0-9]+$ ]] and (( top >= 1 )).

Reference solution

cat > analyze.sh <<'EOF'
#!/bin/bash
# analyze.sh: summarize a web server access log.
set -euo pipefail

usage() {
  echo "usage: analyze.sh [-n N] LOGFILE" >&2
  exit 2
}

top=3
while getopts ':n:' opt; do
  case $opt in
    n) top=$OPTARG ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
(( $# == 1 )) || usage
[[ $top =~ ^[0-9]+$ ]] && (( 10#$top >= 1 )) || usage
top=$((10#$top))

log=$1
if [[ ! -f $log || ! -r $log ]]; then
  echo "analyze.sh: cannot read $log" >&2
  exit 1
fi

awk '
NF >= 10 {
  n++
  ip[$1]++
  if ($9 >= 400) err++
  if ($10 != "-") bytes += $10
}
END {
  printf "requests: %d\nunique ips: %d\nerrors: %d\nbytes: %d\n", n, length(ip), err, bytes
}' "$log"

echo "top $top ips:"
awk 'NF >= 10 { print $1 }' "$log" | sort | uniq -c | sort -k1,1nr -k2,2 \
  | awk -v top="$top" 'NR <= top { printf "  %d %s\n", $1, $2 }'

echo "status codes:"
awk 'NF >= 10 { print $9 }' "$log" | sort -n | uniq -c \
  | awk '{ printf "  %s %d\n", $2, $1 }'
EOF

Exercise 4 · quiz

Why one awk pass?

A first version of the core report ran four separate pipelines over the log (one wc -l, one cut | sort -u | wc -l, one awk for errors, one for bytes). Which statements about switching to a single awk program are true? Select all that apply.

It reads the file once instead of four times, which matters for multi-gigabyte logs.
It applies the 'at least 10 fields' rule in one place, so the numbers cannot disagree.
It is required, because pipelines cannot read the same file twice.
It starts fewer processes.

Answer

It reads the file once instead of four times, which matters for multi-gigabyte logs.
It applies the 'at least 10 fields' rule in one place, so the numbers cannot disagree.
It starts fewer processes.

Several passes over a file are perfectly legal, just slower: each pass reads the whole file again and starts its own processes. The subtler win is consistency: when the filtering rule ("ignore malformed lines") lives in one awk pattern, every number in the report is computed from the same set of lines. With four pipelines it is easy to filter in three of them and forget the fourth.