Pipeline exit status, tee and process substitution

Why this matters

Pipelines are so convenient that they end up in scripts that run unattended, and that raises questions the interactive user never notices. If the first command of a pipeline fails, does anyone find out? How do you keep a copy of data flowing through the middle of a pipeline? And what do you do with a program like diff that wants two files, when what you have is the output of two commands? This lesson answers all three.

The exit status of a pipeline

A pipeline runs several processes, so which status ends up in $?? The answer: the last command's.

$ false | true
$ echo $?
0
$ grep ERROR /nope | wc -l
grep: /nope: No such file or directory
0
$ echo $?
0

The second example is the dangerous one: grep failed to open the file, but the pipeline "succeeded" because wc did. In a script, if grep ... | wc -l or cmd | tee log will happily report success after the important part failed.

bash keeps the status of every stage in a special array variable, PIPESTATUS (arrays come in module 15; for now just use this syntax):

$ grep ERROR /nope | wc -l
grep: /nope: No such file or directory
0
$ echo "${PIPESTATUS[@]}"
2 0

${PIPESTATUS[0]} is the first command's status, ${PIPESTATUS[1]} the second's, and so on. Like $?, it is overwritten by the next command, so read it immediately. (Even echo resets it: the second echo "${PIPESTATUS[@]}" in a row would print just 0.)

The other tool is a shell option:

$ set -o pipefail
$ false | true
$ echo $?
1
$ set +o pipefail

With pipefail on, the pipeline's status is that of the last command that failed (or 0 if all succeeded). Most careful scripts turn it on at the top; module 12 discusses it with its siblings set -e and set -u, including one surprise: with pipefail, producer | head -n 1 may fail because head quitting early kills the producer with SIGPIPE (status 141).

Why does bash default to the last command? Because the shell's own status logic predates pipefail, and because often the last command really is the one that decides: grep -q at the end of a pipeline answers "was it found?".

tee: a T-junction for data

tee copies its standard input to standard output and to one or more files. The name comes from a T-shaped pipe fitting:

              ┌──────────▶ sorted.txt
  sort -u ───▶│ tee
              └──────────▶ stdout ───▶ wc -l

$ sort -u names.txt | tee sorted.txt | wc -l
8
$ head -n 3 sorted.txt
alice
bob
carol

Uses:

• Keep an intermediate result while the pipeline continues, as above.
• Watch and save at once: ./long-build 2>&1 | tee build.log shows the output live and keeps a log.
• Append with tee -a file.
• Write a file you cannot redirect into. On a normal system, sudo echo x > /etc/file fails because the redirection is done by your shell, as you, before sudo runs. echo x | sudo tee /etc/file works because tee itself opens the file with root's rights. (There is no sudo in this sandbox, but you will meet this idiom everywhere.)

Process substitution: commands that look like files

Some programs need file names, not stdin. diff compares two files; it has only one standard input. You could write temporary files:

sort expected.txt > /tmp/a
sort installed.txt > /tmp/b
diff /tmp/a /tmp/b
rm /tmp/a /tmp/b

bash has a better way, process substitution:

$ diff <(sort expected.txt) <(sort installed.txt)
3d2
< htop
5a5
> nano

<(command) runs command with its stdout connected to a pipe, and is replaced on the command line by a file name that refers to the read end of that pipe. You can see the name:

$ echo <(true)
/dev/fd/63

So diff actually received the arguments /dev/fd/63 and /dev/fd/62. When diff opens /dev/fd/63, Linux gives it the pipe that bash attached to fd 63, and diff reads sort's output as if it were a file. diff never knows. In the diff output, 3d2 < htop means "line 3 of the first input (htop) would have to be deleted", 5a5 > nano means "nano was added".

The mirror form >(command) gives a file name to write to, feeding a command's stdin:

./generate-report | tee >(gzip > report.gz) > report.txt

Here tee writes one copy into the gzip process and one into report.txt.

Limitations worth knowing:

• It is a pipe, not a real file: programs that need to seek back and forth or read the file twice may fail.
• It is a bash (and zsh, ksh) feature, not POSIX sh.
• <(...) and < file are different things. wc -l < <(ls) is valid (redirect stdin from the substituted name) but clumsy; ls | wc -l is the normal way.

Common mistakes

• Trusting $? after a pipeline. It only reflects the last stage. Use PIPESTATUS or set -o pipefail.
• Reading PIPESTATUS too late. Any command in between, even echo, resets it.
• A space inside the operator. < (sort a) is a syntax error (bash: syntax error near unexpected token '('). It is <(, no space.
• Using process substitution in a sh script. sh script.sh (dash on Debian) reports Syntax error: "(" unexpected. Run bash scripts with bash.
• tee overwriting a file you wanted to extend. Like >, tee truncates unless you add -a.

Exercises

A quiz on pipeline status, two tasks (tee and process substitution) and a script that has to report the status of the first stage of a pipeline.

Recap

• A pipeline's exit status is its last command's; PIPESTATUS holds every stage's status.
• set -o pipefail makes the pipeline fail if any stage fails.
• tee file duplicates a stream: one copy to the file, one onward; tee -a appends.
• <(cmd) turns a command's output into a readable file name (/dev/fd/N); >(cmd) gives a writable one.

Going further: compare ls /etc | head -n 2; echo "${PIPESTATUS[@]}" with seq 1 1000000 | head -n 2; echo "${PIPESTATUS[@]}". The second shows 141 for seq: 128 plus signal 13, SIGPIPE.