File descriptors and output redirection

Why this matters

So far, output has gone to your screen. Real work needs it elsewhere: a report saved to a file, a log appended to every night, the contents of a file fed into a program, errors kept apart from results. The shell can connect any program's input and output to files, and the program does not need to know or care. This lesson explains the mechanism, file descriptors, because once you understand it, all the redirection syntax (including the confusing 2>&1 in the next lesson) is just bookkeeping.

Files are opened, then used by number

When a program wants to use a file, it asks the kernel to open it (the open system call). The kernel does the work of finding the file on disk and checking permissions, and hands back a small integer, a file descriptor (fd). From then on the program says "write these bytes to fd 3", "read from fd 3", "close fd 3". Each process has its own file descriptor table:

  process: ls (PID 214)
  ┌────┬─────────────────────────────┐
  │ fd │ refers to                   │
  ├────┼─────────────────────────────┤
  │ 0  │ /dev/pts/0  (the terminal)  │  standard input   (stdin)
  │ 1  │ /dev/pts/0  (the terminal)  │  standard output  (stdout)
  │ 2  │ /dev/pts/0  (the terminal)  │  standard error   (stderr)
  │ 3  │ /etc  (opened by ls itself) │
  └────┴─────────────────────────────┘

By a convention every Unix program follows, the first three are always there when a program starts:

• fd 0, standard input: where the program reads its input from.
• fd 1, standard output: where it writes its normal results.
• fd 2, standard error: where it writes error messages and diagnostics.

A program does not open these; it inherits them. Remember from module 1 that a forked child gets copies of its parent's open files, and that exec replaces the program but keeps the file descriptor table. Your interactive bash has fds 0, 1 and 2 pointing at the terminal, so every program it starts inherits a table pointing at the terminal. That is the only reason output "appears on the screen".

You can look at a real table. Linux exposes each process's descriptors under /proc:

$ ls -l /proc/self/fd
lrwx------ 1 learner learner 64 Nov  1 14:20 0 -> /dev/pts/0
lrwx------ 1 learner learner 64 Nov  1 14:20 1 -> /dev/pts/0
lrwx------ 1 learner learner 64 Nov  1 14:20 2 -> /dev/pts/0
lr-x------ 1 learner learner 64 Nov  1 14:20 3 -> /proc/214/fd

/proc/self always means "the process looking at it", here the ls process. Its fd 3 is the directory it opened to produce this very listing.

Redirection: changing the table before exec

Redirection means: in the child, after fork and before exec, bash rearranges the file descriptor table. Then the program starts with its fds already pointing somewhere else.

  bash: ls > list.txt
    fork ──▶ child bash
               open("list.txt", write|create|truncate) → gets fd 3
               dup2(3, 1)      make fd 1 refer to the same open file as fd 3
               close(3)
               execve("/usr/bin/ls", ["ls"])
             ls writes to fd 1 as usual → bytes land in list.txt

Two consequences:

1. > list.txt is not an argument. bash removes it from the word list; ls receives no arguments at all. You can even put it first: > list.txt ls works identically.
2. The file is opened before the program runs. If bash cannot open it (no permission, missing directory), the command is never started.

$ ls > /nope/list.txt
bash: /nope/list.txt: No such file or directory

Notice the message comes from bash, not ls: ls never ran.

Open the shell for this lesson and try ls -l /proc/self/fd > fds.txt, then cat fds.txt. You will see fd 1 pointing at /work/fds.txt: the program that wrote the listing was itself writing into the file.

The operators

> write to a file

$ echo hello > greeting.txt
$ cat greeting.txt
hello
$ echo goodbye > greeting.txt
$ cat greeting.txt
goodbye

> opens the file with truncate: if it exists, it is emptied to zero bytes first; if it does not exist, it is created. The old contents are gone, with no warning and no undo.

>> append

$ echo "line one" >> journal.txt
$ echo "line two" >> journal.txt
$ cat journal.txt
line one
line two

>> opens the file in append mode: every write goes to the current end of the file. It creates the file if needed. Logs are written this way. Append mode is enforced by the kernel per write, so even two processes appending at once will not overwrite each other's lines (though short lines may interleave).

< read from a file

< connects a file to standard input (fd 0):

$ wc -l names.txt
11 names.txt
$ wc -l < names.txt
11

Same count, different output. In the first form wc received the argument names.txt and opened the file itself, so it knows the name and prints it. In the second, bash opened the file and wc just read fd 0; it has no idea a file called names.txt exists. This is a practical trick for getting "just the number".

Many programs read standard input when you give them no file name: cat, sort, wc, head, grep. That is why cat alone waits for typing: its fd 0 is the terminal. (Ctrl-D ends the input.)

Combining them

Redirections can be combined and appear anywhere in the command:

$ sort < names.txt > sorted.txt
$ head -n 3 sorted.txt
Ada
Barbara
Dennis

Creating an empty file, and emptying one

A redirection with no command still opens (and so creates or truncates) the file:

$ > empty.txt
$ wc -c empty.txt
0 empty.txt

This is a common way to empty a log file without deleting it (programs that have it open keep writing to the same file).

Protecting files: noclobber

Because > silently destroys, bash has an optional safety switch:

$ set -o noclobber
$ echo oops > greeting.txt
bash: greeting.txt: cannot overwrite existing file
$ echo yes >| greeting.txt
$ set +o noclobber

With noclobber on, > refuses to overwrite an existing file; >| overrides it deliberately. Some people enable it in their interactive shell.

Common mistakes

• > when you meant >>. The file is truncated the moment bash opens it, before your command runs. A nightly job with > keeps only last night's log.
• Redirecting a file onto itself. sort names.txt > names.txt leaves names.txt empty: bash truncates it first, then sort reads an empty file. Write to a new file and rename it afterwards (sort names.txt > tmp && mv tmp names.txt).
• Thinking the program sees the redirection. echo hi > out.txt does not pass > or out.txt to echo. That is why echo "> out.txt" (quoted) just prints the text.
• A space in a file name. echo hi > my file.txt writes hi file.txt into a file named my. bash takes only the next word as the target; the rest become arguments. Quote it.
• Expecting < to work with programs that ignore stdin. ls < dir lists the current directory: ls never reads standard input.

Exercises

The quiz checks the mechanism. The tasks use the shell; the code exercise runs your script twice in the same directory, which is exactly the situation where > and >> differ.

Recap

• Every process has a file descriptor table; fds 0, 1, 2 are stdin, stdout, stderr, inherited from the parent across fork and exec.
• Redirection is done by bash in the child before exec, so it works for any program; the program never sees the redirection words.
• > file truncates or creates; >> file appends; < file feeds a file to standard input.
• wc -l < file prints only the number, because wc never learns the name.
• Never redirect output onto the file you are reading.

Going further: run ls -l /proc/$$/fd to see your shell's own table, then exec 3> extra.txt (this opens fd 3 in the shell itself), echo hi >&3, ls -l /proc/$$/fd again, and finally exec 3>&- to close it. You will meet exec with redirections again in module 13.