Streams, buffering and stderr

Why this matters

Unix programs are glued together with pipes and redirection: ./extract < log.txt | sort > report.txt. For your C programs to play well in such pipelines they need to read from standard input, write results to standard output, and send complaints somewhere else, so the complaints do not pollute the data. You also need to understand buffering, because it explains some of the most confusing behavior in C: output that appears in the wrong order, or not at all when a program crashes.

Three streams are open before main starts

When a process starts, it inherits three open file descriptors from whoever launched it (usually the shell). A file descriptor is just a small integer the kernel uses to identify an open file, pipe or terminal for that process:

The C library wraps each descriptor in a stream, an object of type FILE that you always handle through a pointer, FILE *. stdin, stdout and stderr are three such pointers, declared in <stdio.h>. printf(...) is shorthand for fprintf(stdout, ...), and getchar() is shorthand for getc(stdin).

   your program                       the kernel                 the outside world

   printf ---> FILE *stdout --buffer--> fd 1 ------------------> terminal, file or pipe
   fprintf(stderr,...) --> FILE *stderr --> fd 2 --------------> terminal (usually)
   scanf  <--- FILE *stdin  <--buffer-- fd 0 <------------------ keyboard, file or pipe

The shell decides where the descriptors point before your program starts:

$ ./prog < in.txt          # fd 0 reads from in.txt
$ ./prog > out.txt         # fd 1 writes to out.txt (truncating it)
$ ./prog >> out.txt        # fd 1 appends to out.txt
$ ./prog 2> err.txt        # fd 2 writes to err.txt
$ ./prog > all.txt 2>&1    # fd 2 goes wherever fd 1 goes
$ ./a | ./b                # a's fd 1 is the write end of a pipe, b's fd 0 the read end

Your program does not know or care. It writes to stdout, and the bytes land in a file, a pipe or a terminal. That is what makes small programs composable.

A filter: copy stdin to stdout

The classic Unix program reads input byte by byte and writes it out. Here is one that also reports what it did, on stderr:

#include <stdio.h>

int main(void)
{
    int c;
    long n = 0;
    while ((c = getchar()) != EOF) {
        putchar(c);
        n++;
    }
    fprintf(stderr, "copied %ld bytes\n", n);
    return 0;
}

$ printf 'hello\nworld\n' > in.txt
$ ./copy < in.txt > out.txt
copied 12 bytes
$ cat out.txt
hello
world
$ ./copy < in.txt 2>/dev/null
hello
world

Points to notice:

• getchar returns an int, not a char. It returns each byte as a value from 0 to 255, or the special value EOF (which is -1 in glibc) when there is no more input. A char variable could not tell byte 255 apart from EOF on many systems, so the result must be stored in an int.
• End of input is not a character in the file. It is the condition "a read returned nothing". On a terminal you signal it by pressing Ctrl-D at the start of a line.
• The data went into out.txt, but the byte count still appeared on the terminal, because it was written to stderr. With 2>/dev/null the diagnostic is thrown away and only data remains.

Buffering: why output is not written immediately

Every write to a file descriptor is a system call: the program stops, the kernel takes over, copies the bytes, and returns. System calls are expensive compared with ordinary function calls, so calling the kernel once per character would make output very slow. The C library therefore collects output in a buffer, an array inside the FILE object (8192 bytes, BUFSIZ, in glibc), and hands it to the kernel in large chunks.

There are three buffering modes, and the library picks one per stream:

Buffers are also flushed when the program calls exit or returns from main, when you call fflush(stream), when you close the stream with fclose, and (for line-buffered stdout) when the program reads from stdin, so prompts appear before the program waits for input.

The decision between line and full buffering is made at run time by asking "is this descriptor a terminal?". That is why the same program can behave differently with and without a pipe.

Watch it happen

#include <stdio.h>

int main(void)
{
    printf("a");
    fprintf(stderr, "b");
    printf("c\n");
    fprintf(stderr, "d\n");
    return 0;
}

Run in a terminal, stdout is line buffered. Trace each call:

 call                     stdout buffer      appears on screen
 printf("a")              "a"                (nothing)
 fprintf(stderr, "b")     "a"                b
 printf("c\n")            "" (flushed)       bac      <- "ac\n" written after "b"
 fprintf(stderr, "d\n")   ""                 bac
                                             d

Terminal output:

$ ./order
bac
d

Now send both streams through a pipe. stdout becomes fully buffered, so "a" and "c\n" wait until exit flushes them at the very end:

$ ./order 2>&1 | cat
bd
ac

Buffering and crashes

If the process dies abnormally (a segmentation fault, or abort()), exit never runs, and whatever is still in the buffer is lost:

#include <stdio.h>
#include <stdlib.h>

int main(void)
{
    printf("starting\n");
    abort();
}

$ ./crash
starting
Aborted (core dumped)
$ ./crash > out.txt
Aborted (core dumped)
$ cat out.txt
$

In the terminal you see "starting"; redirected to a file, nothing. This is a classic debugging trap: you add printf calls to find where a program crashes, redirect the output to a file, and the last messages before the crash vanish, pointing you at the wrong place. Two fixes:

• Print debugging messages to stderr, which is unbuffered.
• Call fflush(stdout) after messages that must not be lost.

You can also change a stream's mode with setvbuf, for example setvbuf(stdout, NULL, _IONBF, 0); to make stdout unbuffered, but that makes heavy output slow.

Input is buffered too

stdin is buffered in the other direction: the library asks the kernel for a big chunk, then getchar and scanf hand it out piece by piece. On a terminal, the kernel itself also holds your typing until you press Enter (so you can use Backspace), which is why a program calling getchar() does not react to a single key press.

Common mistakes

• Printing errors to stdout. ./prog > data.csv silently swallows them into the data file, and the next program in the pipeline tries to parse "Error: ..." as data. Use fprintf(stderr, ...).
• Storing getchar() in a char. The loop while ((c = getchar()) != EOF) with char c may never end (if char is unsigned) or stop early at a 0xFF byte. Always int c.
• Prompts that do not appear. printf("Name: "); without a newline, followed by heavy computation, shows nothing until later. Reading from stdin flushes it, but if you do something else first, call fflush(stdout).
• Debug output lost on a crash, as shown above. Debug on stderr.
• Expecting output order between stdout and stderr when both go to the same place. They are separate streams with separate buffering; only the order within each stream is guaranteed.

Recap

• Every process starts with stdin (0), stdout (1) and stderr (2); the shell decides where they point with <, >, 2> and |.
• In C they are FILE * streams; fprintf(stderr, ...) writes diagnostics that stay out of the data.
• getchar returns an int: a byte value or EOF.
• Output is buffered to save system calls: stderr unbuffered, stdout line buffered on a terminal and fully buffered otherwise. exit flushes; a crash does not.
• fflush(stdout) forces buffered output out now.

Going further: read man 3 setvbuf and man 3 stdin. Then try the order program with stdbuf -o0 ./order | cat, which uses a trick to change stdout's buffering from outside the program.