Redirection and Pipes · about 40 min

Pipes and pipelines

Why this matters

The single most powerful idea in Unix is this: small programs that each do one thing, connected so that the output of one becomes the input of the next. "Which 10 IP addresses hit our web server most often today?" is not a program anyone wrote; it is five small tools in a row, typed in thirty seconds. The connector is the pipe, |. This lesson explains how it works, because the details (concurrency, buffers, what happens when one side stops) explain how pipelines behave on real data.

The idea

$ ls /usr/bin | wc -l
512

| connects the standard output of ls to the standard input of wc. ls writes names; wc counts lines of whatever arrives on its stdin. Neither program knows the other exists. You could do the same with a temporary file:

$ ls /usr/bin > /tmp/names.txt
$ wc -l < /tmp/names.txt
512

but the pipe needs no file, no cleanup, and it starts producing results immediately.

A pipeline is any number of commands joined by |:

$ sort -nr sizes.txt | head -n 3
15360 video.mp4
4096 database.sqlite
2048 backup.tar

Programs written to read stdin and write stdout, so that they can sit in the middle of a pipeline, are called filters: sort, head, tail, wc, grep, cut, tr, sed, awk, uniq. The next module is all about them.

What happens underneath

A pipe is a small buffer inside the kernel with two ends: a write end and a read end, each a file descriptor. Bytes written into one end come out of the other in the same order. bash sets up ls /usr/bin | wc -l like this:

  bash
   │ pipe()              → kernel creates a buffer; bash gets fds: 3 = read end, 4 = write end
   │ fork ─▶ child A:  dup2(4, 1)   stdout becomes the write end
   │                   close 3, 4   (tidy up the originals)
   │                   exec ls
   │ fork ─▶ child B:  dup2(3, 0)   stdin becomes the read end
   │                   close 3, 4
   │                   exec wc -l
   │ close 3, 4         the parent keeps no ends open
   │ wait for both
   ▼
          ┌──────┐  fd 1   ┌───────────────┐   fd 0  ┌──────┐
          │  ls  │ ──────▶ │ kernel buffer │ ──────▶ │  wc  │ ──▶ terminal (fd 1)
          └──────┘         └───────────────┘         └──────┘

It is the same trick as redirection: rearrange file descriptors between fork and exec. And it is the reason any program can join a pipeline without special support.

Several things follow from this design:

All commands run at the same time. bash starts every process of the pipeline at once. They are concurrent, not sequential. grep handles the first lines while cat is still reading the rest of the file.

The buffer is small, and that is fine. On Linux a pipe holds 64 KiB by default. If the writer is faster than the reader and the buffer fills up, the kernel simply pauses the writer (its write call blocks) until the reader catches up. If the reader is faster and the buffer is empty, the reader waits. This automatic speed matching is called backpressure, and it is why a pipeline uses little memory even on a 50 GB log file.

End of input is "all writers closed". The reader sees end-of-file when the buffer is empty and no process holds the write end any more. That is why bash closes its own copies: if the parent kept the write end open, wc would wait forever.

When the reader quits early, the writer is killed. If the last reader closes the pipe and the writer writes again, the kernel sends the writer the signal SIGPIPE, which terminates it by default. That makes this instant, even though seq would produce a billion lines:

$ seq 1 1000000000 | head -n 3
1
2
3

Only stdout goes into the pipe. Standard error is not connected, so errors still reach the terminal:

$ ls /etc/hostname /nope | wc -l
ls: cannot access '/nope': No such file or directory
1

To pipe both, merge them first: ls /etc/hostname /nope 2>&1 | wc -l (prints 2). bash also accepts |& as shorthand for 2>&1 |.

Every part runs in its own process. This matters later: a variable set inside a pipeline stage vanishes when that stage's process ends. Module 10 shows the classic bug this causes with while read loops.

Programs behave differently in a pipe

Notice that ls prints names in columns on your terminal but one per line into a pipe. Programs can ask the kernel whether their stdout is a terminal (the isatty check), and many adjust: ls switches to one name per line, grep and ls drop their colors, and many programs switch from line buffering to block buffering. You can see it:

$ ls /etc/apt
apt.conf.d  auth.conf.d  keyrings  preferences.d  sources.list.d  trusted.gpg.d
$ ls /etc/apt | cat
apt.conf.d
auth.conf.d
keyrings
preferences.d
sources.list.d
trusted.gpg.d

This is helpful (one name per line is exactly what wc -l wants), but remember it when output in a pipeline looks different from the screen.

Building a pipeline step by step

Write pipelines incrementally. Run the first command and look at its output, then add one stage at a time. For example, "the three most common shells in /etc/passwd":

$ head -n 3 /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
$ cut -d: -f7 /etc/passwd | head -n 3
/bin/bash
/usr/sbin/nologin
/usr/sbin/nologin
$ cut -d: -f7 /etc/passwd | sort | uniq -c | sort -nr
     16 /usr/sbin/nologin
      2 /bin/bash
      1 /bin/sync

You will learn cut and uniq properly in the next module; the habit to take away now is: look at the data after every stage.

Common mistakes

Piping into a program that does not read stdin. ls | rm does nothing useful: rm takes file names as arguments, not on stdin. (Module 14 shows xargs, which converts one into the other.)
Expecting errors in the pipe. cmd | grep error only searches stdout. Use cmd 2>&1 | grep error.
sort without -n for numbers. Text order puts 100 before 20 and 9 after 15360.
Reading and writing the same file in one pipeline. sort data.txt | uniq > data.txt truncates data.txt immediately, possibly before sort has read it. Same rule as before: write to a new file.
Useless use of cat. cat file | grep x works, but grep x file or grep x < file saves a process. Harmless, but you will see people point it out.

Exercises

Two quizzes on the mechanism, two tasks in the shell, and one script that acts as a filter: it reads its standard input, which the checker fills with numbers.

Recap

a | b connects a's stdout to b's stdin through a kernel pipe buffer.
bash creates the pipe, forks one child per command, rewires fds 0 and 1, and execs; all commands run concurrently.
A full buffer pauses the writer; an empty one pauses the reader; a closed reader kills the writer with SIGPIPE.
Only stdout goes through the pipe; use 2>&1 | to include stderr.
Build pipelines one stage at a time, looking at the output after each.

Going further: run sleep 30 | sleep 30 & and then ps -o pid,ppid,stat,comm: both sleeps are alive at the same time. Then ls -l /proc/self/fd | cat and look at what fd 1 of ls points to: pipe:[...], with a number identifying the pipe.

Exercises

Exercise 1 · quiz

One after another?

How does bash run the pipeline cat huge.log | grep ERROR | head -n 5?

cat runs to completion and saves its output in a temporary file, then grep runs on that file, then head.
All three processes are started at once and run concurrently, connected by kernel pipe buffers.
bash reads the output of each command into its own memory and passes it to the next one.
grep runs first, because it is in the middle, and asks cat for data when needed.

Answer

All three processes are started at once and run concurrently, connected by kernel pipe buffers.

bash creates two pipes, forks three children, wires each child's stdout or stdin to a pipe end, and execs all three. They run at the same time: grep processes lines as soon as cat produces them. No temporary files, and bash does not touch the data at all. This is why pipelines can process files far larger than memory, and why head gets its first line almost instantly.

Exercise 2 · quiz

When the reader leaves

In seq 1 100000000 | head -n 3, head prints three lines and exits after a fraction of a second. seq was asked to print a hundred million numbers. What happens to seq?

It keeps running in the background until it has printed all hundred million numbers.
Its next write to the pipe fails because nobody is reading anymore; the kernel sends it SIGPIPE and it dies.
Its output fills up /tmp until the disk is full.
bash waits for head, then prints the remaining numbers on the terminal.

Answer

Its next write to the pipe fails because nobody is reading anymore; the kernel sends it SIGPIPE and it dies.

Once the last reader of a pipe has closed it, any write to the pipe makes the kernel deliver the signal SIGPIPE to the writer, whose default action is to terminate it. That is how head can stop an infinite producer. (Before that, while head was still reading, seq was simply put to sleep whenever the pipe buffer was full.) The whole pipeline finishes almost at once.

Exercise 3 · task in a shell

How many programs?

Open the shell. Using a pipeline of ls and wc, count how many entries there are in /usr/bin, and save just the number in /work/nbin.txt.

Hint 1

ls /usr/bin produces the names; wc -l counts lines on its standard input.

Hint 2

Put the redirection on the last command: ls /usr/bin | wc -l > nbin.txt

Reference solution

ls /usr/bin | wc -l > nbin.txt

Exercise 4 · task in a shell

Largest files first

/work/sizes.txt lists files with their size in kilobytes, one per line, like 2048 backup.tar. Write a pipeline that saves the 3 largest entries, largest first, into /work/top3.txt (the lines unchanged).

You need sort with two options: -n (compare as numbers, not as text) and -r (reverse order), followed by head.

Hint 1

sort -nr sizes.txt sorts numerically, largest first. Look at its output before adding the rest.

Hint 2

sort -nr sizes.txt | head -n 3 > top3.txt

Reference solution

sort -nr sizes.txt | head -n 3 > top3.txt

Exercise 5 · write code

Top three numbers

Write a script that reads whole numbers from standard input, one per line, and prints the three largest, largest first, one per line. If there are fewer than three numbers, print all of them, largest first.

Example: input 5 12 7 3 40 (one per line) prints 40, 12, 7.

File: script.sh

Starting point

# read numbers from standard input and print the three largest
sort

Example

input

expected output

40
12
7

Example

input

expected output

1000
100
20

Hint 1

With no file name, sort reads standard input, which is the script's standard input.

Hint 2

sort -nr | head -n 3. Without -n, 100 sorts before 20 because '1' comes before '2'.

Reference solution

sort -nr | head -n 3