Standard error, /dev/null and ordering

Why this matters

Imagine a script that produces a report and saves it with > report.txt. One input file is missing. If the error message went into the report, the report would be corrupted and nobody would see the problem. Unix avoids this by giving every program two output channels: one for results, one for complaints. Knowing how to steer each of them separately, merge them, or throw one away is something you will do every day, and the syntax (2>&1) is famously confusing until you see what it does to the file descriptor table.

Two outputs, on purpose

From the previous lesson: fd 1 is standard output, fd 2 is standard error. Both start out pointing at the terminal, so you see them mixed. But they are separate descriptors, and > only moves fd 1:

$ ls /etc/hostname /nope > out.txt
ls: cannot access '/nope': No such file or directory
$ cat out.txt
/etc/hostname

The listing went to the file; the error stayed on the screen, where a human would notice it. That is the design: results on stdout, diagnostics on stderr. It also means a pipeline (next lesson) passes only real data to the next program, while errors still reach you.

Redirecting by number

Every redirection operator can take a file descriptor number in front of it:

No space between the number and the operator: 2> is one token, while 2 > means "the argument 2, then >".

$ ls /etc/hostname /nope > out.txt 2> err.txt
$ cat out.txt
/etc/hostname
$ cat err.txt
ls: cannot access '/nope': No such file or directory

2>&1: duplication, not a link

N>&M means: make fd N refer to whatever fd M refers to right now. Underneath, bash calls the dup2(M, N) system call. It copies the table entry; it does not create a lasting link between the two numbers.

Watch what happens to the table, step by step, for ls /etc /nope > all.txt 2>&1:

                       fd 1        fd 2
  start                terminal    terminal
  > all.txt            all.txt     terminal
  2>&1                 all.txt     all.txt      (copy of fd 1, which is the file)
  → both streams go into all.txt

Now the other order, ls /etc /nope 2>&1 > all.txt:

                       fd 1        fd 2
  start                terminal    terminal
  2>&1                 terminal    terminal     (copy of fd 1, which is still the terminal)
  > all.txt            all.txt     terminal
  → only stdout goes into the file; errors still on screen

Redirections are processed left to right, like assignments. "> file 2>&1" reads as "send stdout to the file, then send stderr wherever stdout goes now". Say it like that and you will never get the order wrong.

bash also has &> file (and &>> file to append) which does both at once. It is convenient interactively, but it is not part of the POSIX standard and fails in sh scripts, so prefer > file 2>&1 in scripts (module 16 covers portability).

When both fds share one open file, the writes land in the order the program makes them. One subtlety: many programs buffer standard output when it is not a terminal (collecting it and writing in big chunks) but write standard error immediately. So the interleaving in a merged file can differ from what you saw on screen. Shell builtins like echo write immediately, so scripts made of echos keep their order.

Writing to standard error yourself

Your own scripts should follow the convention too: errors and warnings to stderr. echo always writes to fd 1, so redirect that one command:

echo "error: config file not found" >&2

>&2 makes fd 1 of this echo a copy of fd 2, so its output goes wherever the shell's stderr goes. If someone runs your script with > report.txt, the error still reaches their terminal. You will use this pattern in nearly every script from module 8 on.

/dev/null: the bottomless file

/dev/null is a special device file. Anything written to it is accepted and discarded; reading from it gives end-of-file immediately.

$ ls -l /dev/null
crw-rw-rw- 1 root root 1, 3 Nov  1 14:00 /dev/null

The c at the start means character device: not data on disk, but a door into a kernel driver. The driver for "1, 3" simply throws bytes away. It is writable by everyone.

Common uses:

grep -r TODO . 2>/dev/null          # hide "Permission denied" noise, keep results
command -v jq > /dev/null 2>&1      # only care about the exit status
noisy_tool > /dev/null              # keep the errors, drop the chatter
sort < /dev/null                    # run with empty input

The second line is the classic "is this program installed?" test: all output discarded, and the exit status tells the answer.

Warning: Be careful about hiding stderr. 2>/dev/null makes a failing command look silent. Discard errors only when you know exactly which messages you are hiding and why.

Common mistakes

• 2>&1 > file instead of > file 2>&1. The errors stay on the terminal. Read it left to right.
• A space after the number. ls /nope 2 > err.txt passes 2 as an argument to ls (which then complains that 2 does not exist) and redirects stdout.
• 2>&1 written as 2>1. Without the &, bash creates a file literally named 1.
• Writing errors to stdout in scripts. They then end up in the output file or the next program in a pipeline, instead of in front of a human.
• Sending both streams to two separate > of the same file. cmd > f 2> f opens the file twice, with two independent write positions, and the streams overwrite each other. Use > f 2>&1 so both share one open file.

Exercises

The quiz is about order. The two tasks use a small program, noisy.sh, that writes to both streams; the code exercise has you write to stderr yourself.

Recap

• stdout (fd 1) is for results, stderr (fd 2) for diagnostics; > only moves fd 1.
• 2> file redirects errors; N>&M copies fd M into fd N at that moment (dup2).
• Redirections are applied left to right: > file 2>&1 captures both, 2>&1 > file does not.
• >&2 sends a command's output to stderr; scripts should report errors that way.
• /dev/null discards everything written to it.

Going further: try ls /etc /nope 3>&1 1>&2 2>&3 and work out on paper what each step does to the table (this swaps stdout and stderr). Then ls -l /proc/self/fd 2>/dev/null and see what fd 2 points to.