awk: fields, patterns and actions

Why this matters

cut breaks on space-aligned columns, grep cannot compare numbers, and none of the tools so far can add things up. awk can do all of that in one line. It is a small programming language designed for exactly one job: reading text records split into fields, and doing something with the ones you care about. "Sum the sizes of all files owned by bob", "average response time per URL", "print the lines where column 4 is over 10" are all one-liners. (The name comes from its authors, Aho, Weinberger and Kernighan, in 1977.)

The model: pattern { action }

An awk program is a list of rules of the form

pattern { action }

awk reads its input one record (by default, one line) at a time, splits it into fields, and for each rule whose pattern is true for this record, runs the action. Like sed, it is an automatic loop over lines; you only write the body.

• If the pattern is missing, the action runs for every line.
• If the action is missing, the default action is { print }, which prints the whole line.

$ awk '{ print $1 }' /etc/passwd | head -n 2
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
$ awk -F: '{ print $1 }' /etc/passwd | head -n 2
root
daemon
$ awk -F: '$3 >= 1000' /etc/passwd
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
learner:x:1000:1000::/home/learner:/bin/bash

The last one is a pattern without an action: print lines whose third field (the user ID) is at least 1000. On Debian that means "real people" rather than system accounts, plus the special nobody account ($3 >= 1000 && $3 < 65534 would exclude it).

Always put the program in single quotes: $1 means something to bash too, and you want awk to see it.

Fields and built-in variables

awk splits each record at the field separator. By default that is "any run of spaces and tabs", with leading and trailing whitespace ignored, which is exactly right for aligned output like ls -l or ps, where cut fails. -F sets another separator: -F, for CSV, -F: for /etc/passwd, -F'\t' for tabs.

$ echo 'alice 34 Lisbon PT' | awk '{ print NF, $NF, $1 }'
4 PT alice
$ ls -l /etc | awk 'NR > 1 { print $5, $NF }' | head -n 3
1108 alternatives
32 apparmor.d
138 apt

print a, b prints its arguments separated by OFS (a space by default) and ends with a newline. print a b (no comma) concatenates them with nothing in between.

Patterns

A pattern can be:

• a comparison: $4 > 10, $2 == "north", NF != 5, NR > 1
• a regex match: /ERROR/ (the whole line), $3 ~ /^gad/ (one field), $3 !~ /x/
• a combination with &&, ||, !: NR > 1 && $4 > 10
• the special patterns BEGIN (before any input is read) and END (after all input)
• a range /start/,/stop/ like in sed

Open the shell for this lesson; sales.csv is in /work.

$ awk -F, '$2 == "east"' sales.csv
2026-10-02,east,widget,20,2.50
2026-10-04,east,doohickey,1,49.00
2026-10-06,east,sprocket,30,0.75
2026-10-08,east,doohickey,3,49.00
$ awk -F, '$3 ~ /^g/ && $4 >= 8 { print $1, $3, $4 }' sales.csv
2026-10-03 gadget 8
2026-10-05 gadget 11

Strings and numbers

awk has no separate number type to declare. A value is treated as a number when used with arithmetic or compared with a number, and as a string otherwise. Field values that look like numbers compare numerically: $4 > 10 compares 12 and 10 as numbers. But the header field units does not look like a number, so "units" > 10 becomes a string comparison (and u sorts after 1, so it is true). That is why data scripts start with NR > 1.

Variables, arithmetic and END

Variables need no declaration. An unset variable is the empty string, which counts as 0 in arithmetic. This makes accumulating natural:

$ awk -F, 'NR > 1 { units += $4 } END { print units }' sales.csv
136
$ awk -F, 'NR > 1 { n++; sum += $4 } END { print "average:", sum / n }' sales.csv
average: 9.06667

The main rule runs for every line; END runs once at the end, when the totals are complete. BEGIN is for setup before the first line, such as printing a header or setting OFS:

$ awk -F, 'BEGIN { OFS = "\t" } NR > 1 { print $3, $4 }' sales.csv | head -n 2
widget    12
gadget    5

Arithmetic works as you expect: + - * / % and ^ for powers, with floating-point results. x++, x += 3 and friends exist too.

printf: controlled formatting

print uses a default number format (9.06667 above). printf gives control, using the same format language as C and the shell's printf command:

$ awk -F, 'NR > 1 { t += $4 * $5 } END { printf "%.2f\n", t }' sales.csv
832.19
$ awk -F, 'NR > 1 { printf "%-10s %5d\n", $3, $4 }' sales.csv | head -n 3
widget        12
gadget         5
widget        20

%s string, %d integer, %.2f two decimals, %-10s left-aligned in 10 columns, %5d right- aligned in 5. Unlike print, printf adds no newline: you write \n yourself.

Associative arrays: grouping

awk arrays are indexed by strings (associative arrays, also called maps or dictionaries). They make "per group" totals a one-liner:

$ awk -F, 'NR > 1 { units[$3] += $4 } END { for (p in units) print p, units[p] }' sales.csv | sort
doohickey 4
gadget 31
gizmo 11
sprocket 30
widget 60

units[$3] += $4 creates the entry for that product on first use (as empty, so 0) and adds to it. In END, for (p in units) visits each key. The order of for (k in arr) is unspecified (it depends on awk's internal hash table), so pipe the result through sort when order matters.

Count occurrences the same way: awk '{ count[$1]++ } END { for (k in count) print count[k], k }' is the awk version of sort | uniq -c, in one pass and without sorting the input.

When awk, when the others?

• Selecting lines by text: grep. Simple substitutions: sed. Fixed columns with a single clean delimiter: cut.
• Anything with numbers, comparisons, sums, per-group results, reordering fields, or columns separated by runs of spaces: awk.
• Logic longer than a few lines: a real script (bash, or Python).

awk programs can be put in a file and run with awk -f prog.awk, and support if, while, for, functions and string functions (length, substr, split, tolower, sub, gsub). This lesson covers the one-liner use, which is most of what you will need.

Common mistakes

• Double quotes around the program. awk "{ print $1 }" lets bash replace $1 (usually with nothing) and awk prints whole lines. Use single quotes.
• Forgetting -F, on CSV. The default separator is whitespace; the whole line becomes $1.
• Counting the header. Use NR > 1.
• print $1 $2 expecting a space. Use a comma: print $1, $2.
• Printing an empty total. With no input lines, print total prints an empty line; print total + 0 prints 0.
• Relying on for (k in arr) order. It is not sorted, and may differ between awk versions.
• CSV with quoted commas. awk's -F, knows nothing about quoting, like cut. For real CSV with quoted fields, use a CSV-aware tool (Python's csv module, or GNU awk's --csv option in version 5.3 and newer).

Exercises

A quiz on fields, a summing script with an edge case, two tasks on sales.csv, and a per-group report script.

Recap

• awk runs pattern { action } rules for each record; missing pattern = every line, missing action = print.
• Fields: $1..$NF, $0 whole line; NF field count, NR record number; -F sets the separator.
• Patterns: comparisons, /regex/, $n ~ /re/, &&, BEGIN, END.
• Variables start empty (0); accumulate in the main rule, report in END; printf formats.
• arr[key] += value groups; pipe for (k in arr) output through sort.

Going further: ps -eo user,rss | awk 'NR > 1 { m[$1] += $2 } END { for (u in m) print u, m[u] }' shows memory use per user. GNU awk's manual is extensive; awk --help lists its options here.