Text Processing · about 45 min

Slicing and counting: sort, uniq, cut, tr, wc

Why this matters

Data arrives as text all the time: CSV exports, logs, /etc/passwd, the output of other commands. A handful of tiny programs, each doing one simple thing to lines of text, can answer a surprising number of questions about such data: how many, which distinct values, the top ten, which column. This lesson covers the five most common ones. Combined with grep from the previous lesson and the pipes from module 5, they form the everyday Unix data toolkit.

All of them are filters: they read the files named as arguments, or standard input if there are none, and write to standard output. None of them modifies its input file.

Open the shell for this lesson; /work/sales.csv is ready:

$ head -n 4 sales.csv
date,region,product,units,price
2026-10-01,north,widget,12,2.50
2026-10-01,south,gadget,5,9.99
2026-10-02,east,widget,20,2.50

wc: counting

wc (word count) counts lines, words and bytes:

$ wc sales.csv
 16  16 505 sales.csv
$ wc -l sales.csv
16 sales.csv
$ wc -l < sales.csv
16

-l lines, -w words (runs of non-whitespace), -c bytes, -m characters (differs from bytes for non-ASCII text). Precisely, -l counts newline characters, so a last line without a trailing newline is not counted. 16 lines here means 15 sales plus the header.

head and tail, as slicers

You know head -n 5 and tail -n 5. Two more forms are useful for data:

$ tail -n +2 sales.csv | head -n 2
2026-10-01,north,widget,12,2.50
2026-10-01,south,gadget,5,9.99

tail -n +2 means "starting at line 2", the standard way to drop a header line. (head -n -1, with a minus, means "all but the last line".)

cut: columns

cut extracts parts of each line. With -d (delimiter) and -f (fields):

$ cut -d, -f3 sales.csv | head -n 3
product
widget
gadget
$ cut -d, -f2,4 sales.csv | head -n 3
region,units
north,12
south,5
$ cut -d: -f1,7 /etc/passwd | head -n 2
root:/bin/bash
daemon:/usr/sbin/nologin

Fields are numbered from 1. Lists and ranges work: -f1,3, -f2-4, -f3- (3 to the end). cut -c1-10 cuts characters instead of fields, handy for fixed-width text such as the timestamp at the start of a log line.

cut is simple on purpose, and has two limits: the delimiter is a single character, and every occurrence counts, so two spaces in a row mean an empty field between them. It also knows nothing about CSV quoting: a field like "Smith, John" is split at its comma. For space-aligned output (like ps or ls -l) use awk instead (lesson 4).

sort: ordering

sort reads all its input (it has to, to sort it) and prints the lines in order.

$ printf '%s\n' banana Apple cherry apple | sort
Apple
apple
banana
cherry

The important options:

Option	Effect
`-n`	numeric: compare numbers by value (`9` before `10`)
`-r`	reverse
`-u`	unique: output each distinct line once
`-t,`	field separator (like cut's `-d`)
`-k2,2`	sort by field 2 only (from field 2 to field 2)
`-f`	fold case: ignore upper/lower
`-h`	human sizes: `2K` < `1M` < `3G`

Sorting by a column, for example sales by units, highest first:

$ tail -n +2 sales.csv | sort -t, -k4,4nr | head -n 3
2026-10-06,east,sprocket,30,0.75
2026-10-02,east,widget,20,2.50
2026-10-03,south,widget,15,2.50

-k4,4nr means "the key is field 4 only, compared numerically, reversed". Always write the end of the key (-k4,4); -k4 alone means "from field 4 to the end of the line", which surprises people when the later fields differ. You can give several keys; later keys break ties: sort -t, -k2,2 -k4,4nr sorts by region, then by units within each region.

Why does text sorting put 100 before 20? It compares character by character: 1 sorts before 2, and the comparison is over. -n parses the leading number instead.

Note: The order of letters depends on the locale (language settings). This sandbox uses C.UTF-8, which sorts by character code: all uppercase letters before lowercase. Other systems may sort apple and Apple together. In scripts that must behave the same everywhere, people write LC_ALL=C sort.

uniq: collapsing repeats

uniq removes a line if it is identical to the line just before it. It only looks at adjacent lines, so you almost always sort first:

$ cut -d, -f3 sales.csv | tail -n +2 | uniq | head -n 4
widget
gadget
widget
gizmo
$ cut -d, -f3 sales.csv | tail -n +2 | sort | uniq
doohickey
gadget
gizmo
sprocket
widget

Its real power is -c, which prefixes each line with how many times it occurred:

$ tail -n +2 sales.csv | cut -d, -f3 | sort | uniq -c
      2 doohickey
      4 gadget
      3 gizmo
      1 sprocket
      5 widget
$ tail -n +2 sales.csv | cut -d, -f3 | sort | uniq -c | sort -nr
      5 widget
      4 gadget
      3 gizmo
      2 doohickey
      1 sprocket

sort | uniq -c | sort -nr is the frequency-count idiom. You will type it hundreds of times. Also useful: uniq -d prints only lines that were repeated, uniq -u only lines that were not.

tr: translating characters

tr works on characters, not lines or fields. It reads only standard input (it takes no file names). tr SET1 SET2 replaces each character of SET1 with the corresponding one of SET2:

$ echo 'Hello World' | tr 'a-z' 'A-Z'
HELLO WORLD
$ echo '2026-10-30' | tr '-' '/'
2026/10/30
$ echo 'a,b,,c' | tr ',' '\n'
a
b

c

Options:

-d SET deletes characters: tr -d '\r' removes the carriage returns of Windows text files, tr -d '[:digit:]' removes digits.
-s SET squeezes runs of repeated characters into one: tr -s ' ' collapses multiple spaces. With two sets, it squeezes after translating: tr -s ' ' '\n' turns any run of spaces into a single newline.
-c complements SET1 ("every character not in"): tr -cs 'a-zA-Z' '\n' puts each word on its own line, treating every non-letter as a separator.

'\n' inside the quotes is tr's own escape for newline; bash passes the backslash through unchanged because of the single quotes.

Putting them together

How many distinct products, and how many units were sold in the north?

$ tail -n +2 sales.csv | cut -d, -f3 | sort -u | wc -l
5
$ grep ',north,' sales.csv | cut -d, -f4
12
8
6
4

Summing those numbers is beyond these tools (that is awk's job, lesson 4), but finding and counting is not.

Common mistakes

uniq without sort. Duplicates that are not adjacent survive. uniq -c then reports the same value several times with small counts.
Counting the header. Data files often start with a header line; tail -n +2 removes it.
Text sort on numbers. Use -n (or -h for sizes like 4K).
-k2 instead of -k2,2. The key runs to the end of the line.
tr with a file name. tr a-z A-Z file.txt fails with tr: extra operand 'file.txt'. Use tr a-z A-Z < file.txt.
cut on space-aligned columns. ps aux | cut -d' ' -f2 returns empty fields, because the columns are padded with several spaces. Use awk '{print $2}'.

Exercises

Two tasks on sales.csv, a quiz on uniq, and two filter scripts that read standard input.

Recap

wc -l counts lines; head/tail -n +2 slice; cut -d, -fN picks columns.
sort with -n -r -u -t -k N,N orders lines; set -k ranges explicitly.
uniq collapses adjacent duplicates; sort | uniq -c | sort -nr counts frequencies.
tr translates (a-z A-Z), deletes (-d) and squeezes (-s) characters from stdin.

Going further: tr -cs 'A-Za-z' '\n' < /etc/services | sort | uniq -c | sort -nr | head shows the most common words in a real system file. Also look at paste and column -t -s, to join files side by side and pretty-print CSV.

Exercises

Exercise 1 · task in a shell

Unique regions

Open the shell. /work/sales.csv has a header line and one sale per line: date,region,product,units,price.

Save the list of distinct regions into /work/regions.txt: one region per line, sorted alphabetically, each appearing once, and without the header word region.

Hint 1

cut -d, -f2 prints field 2 of comma-separated lines. tail -n +2 prints from line 2 on, dropping the header.

Hint 2

tail -n +2 sales.csv | cut -d, -f2 | sort -u > regions.txt

Reference solution

tail -n +2 sales.csv | cut -d, -f2 | sort -u > regions.txt

Exercise 2 · task in a shell

How many sales per product?

Using the same sales.csv, count how many sales (lines) there are for each product, and save the result in /work/product-counts.txt, most frequent first, one product per line in the form count product, e.g. 5 widget.

The output of uniq -c has leading spaces; that is fine, the check ignores extra spaces.

Hint 1

Extract the product column, then sort | uniq -c gives count product lines.

Hint 2

Finish with sort -nr to put the largest counts first: tail -n +2 sales.csv | cut -d, -f3 | sort | uniq -c | sort -nr > product-counts.txt

Reference solution

tail -n +2 sales.csv | cut -d, -f3 | sort | uniq -c | sort -nr > product-counts.txt

Exercise 3 · quiz

Why sort before uniq?

A file contains the lines b, a, b, b, a (in that order). What does uniq print when given this file unsorted?

a then b
b then a
b, a, b, a
b, a, b, b, a (unchanged)

Answer

b, a, b, a

uniq only compares each line with the one directly before it, and drops a line when it is identical to the previous one. So it collapses runs: b, a, b b (to one b), a. It does this so it can work on endless streams with almost no memory. Sorting first puts all equal lines next to each other, which turns "collapse runs" into "remove all duplicates". sort -u does both in one step when you do not need counts.

Exercise 4 · write code

One word per line

Write a script that reads text from standard input, converts all uppercase letters to lowercase, and prints one word per line, where words are separated by one or more spaces (and by line ends). There are no tabs, punctuation or leading spaces in the input.

Example: the input The quick Brown prints the, quick, brown on three lines. Use tr (twice is fine).

File: script.sh

Starting point

tr 'A-Z' 'a-z'

Example

input

The  quick Brown fox
JUMPS   over the lazy dog

expected output

the
quick
brown
fox
jumps
over
the
lazy
dog

Example

input

Hello World   
BYE

expected output

hello
world
bye

Hint 1

tr ' ' '\n' turns every space into a newline. With two spaces in a row you would get an empty line.

Hint 2

tr -s squeezes repeated characters of the last set into one: tr 'A-Z' 'a-z' | tr -s ' ' '\n'

Reference solution

tr 'A-Z' 'a-z' | tr -s ' ' '\n'

Exercise 5 · write code

Count distinct values

Write a script that reads a comma-separated file with a header line from standard input and prints the number of distinct values in its second column (excluding the header), as a bare number.

For example, for

name,city
ana,Lisbon
ben,Oslo
chloe,Lisbon

it prints 2.

File: script.sh

Starting point

cut -d, -f2 | wc -l

Example

input

name,city
ana,Lisbon
ben,Oslo
chloe,Lisbon

expected output

Example

input

id,color,size
1,red,S
2,blue,M
3,green,L
4,red,M
5,blue,S
6,red,L

expected output

Hint 1

Four steps: drop the header, take column 2, keep each value once, count lines.

Hint 2

tail -n +2 | cut -d, -f2 | sort -u | wc -l. With no file name, each tool reads standard input.

Reference solution

tail -n +2 | cut -d, -f2 | sort -u | wc -l