Pipes and redirection

Why this matters

grep error log.txt | sort | uniq -c > report.txt connects four processes and a file with two characters, | and >. That composability, small programs that each do one thing, glued together with pipes, is the core of the Unix design, and it is built from three system calls you can now use yourself: pipe, dup2 and the fork/exec/wait trio from the previous lesson. Knowing how it works lets you run other programs and collect their output from C, build pipelines in your own tools, and diagnose the classic bug where a program using pipes hangs forever.

The pipe

pipe(fds) asks the kernel for a one-way byte channel with a buffer (64 KiB on Linux) and gives you two file descriptors:

               write(fds[1], ...)             read(fds[0], ...)
   writer  ----------------------->  [ kernel buffer ]  ----------------------->  reader
                 fds[1]: write end                          fds[0]: read end

Bytes written to fds[1] come out of fds[0] in the same order. Like a file, a pipe has no message boundaries: three writes of 10 bytes may arrive as one read of 30 or as reads of 7 and 23. The behavior that makes pipes work:

• read blocks while the pipe is empty, and returns 0 (end of file) only when the pipe is empty and every write end is closed, in every process.
• write blocks while the pipe is full, until the reader catches up.
• write to a pipe with no readers left kills the writer with the signal SIGPIPE (or, if that is ignored, fails with EPIPE). That is why yes | head -1 terminates: when head exits, yes gets SIGPIPE.

A pipe inside one process is not very useful. Its power comes from fork: the child inherits copies of both descriptors, so parent and child share the channel.

Closing what you do not use

After fork, both processes hold both ends. Each must close the ends it does not use:

  after fork:                              after closing unused ends:
  parent: fds[0] fds[1]                    parent:        fds[1]  (writes)
  child:  fds[0] fds[1]                    child:  fds[0]         (reads)

Forgetting this is the number one pipe bug. If the reader keeps its own copy of the write end open, its read never returns 0, because a writer (itself) still exists, and it hangs forever after consuming the data. In the exercises, such a hang shows up as "timed out".

dup2: rewiring descriptors

dup2(oldfd, newfd) makes newfd refer to the same open file as oldfd, closing whatever newfd referred to before. Since programs simply write to descriptor 1 and read from descriptor 0, rewiring those numbers before exec redirects a program without it knowing:

  dup2(fd, STDOUT_FILENO):

  before                               after
  1 -> terminal                        1 -> out.txt
  3 -> out.txt   (fd)                  3 -> out.txt   (then close(3))

This is the whole of shell redirection: > is open + dup2(fd, 1), < is open + dup2(fd, 0), 2>&1 is dup2(1, 2). Open file descriptors survive exec, so the new program inherits them.

Redirecting output to a file

#define _POSIX_C_SOURCE 200809L
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>

int main(void)
{
    int fd = open("out.txt", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    if (fd < 0) {
        perror("open");
        return 1;
    }
    fflush(stdout);
    if (dup2(fd, STDOUT_FILENO) < 0) {
        perror("dup2");
        return 1;
    }
    close(fd);
    printf("this line goes to out.txt\n");
    fflush(stdout);
    execlp("date", "date", "+%Y", (char *)NULL);
    perror("exec");
    return 1;
}

Afterwards out.txt contains this line goes to out.txt followed by the year from date. Remove the second fflush and the printf line vanishes: stdout now points to a file, so stdio buffers it, and exec throws away the process's memory, buffer included, before it was ever written. (This example redirects its own stdout; real code usually does it in a child after fork, so the parent keeps its stdout.)

Building a pipeline

Here is printf 'b\na\nc\n' | sort done by hand: one pipe, two children, each with one end dup2'd onto stdout or stdin:

#define _POSIX_C_SOURCE 200809L
#include <stdio.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <unistd.h>

int main(void)
{
    int fds[2];
    if (pipe(fds) < 0) {
        perror("pipe");
        return 1;
    }

    pid_t writer = fork();
    if (writer == 0) {
        dup2(fds[1], STDOUT_FILENO);   /* stdout now goes into the pipe */
        close(fds[0]);
        close(fds[1]);
        execlp("printf", "printf", "b\\na\\nc\\n", (char *)NULL);
        perror("exec printf");
        _exit(127);
    }

    pid_t reader = fork();
    if (reader == 0) {
        dup2(fds[0], STDIN_FILENO);    /* stdin now comes from the pipe */
        close(fds[0]);
        close(fds[1]);
        execlp("sort", "sort", (char *)NULL);
        perror("exec sort");
        _exit(127);
    }

    close(fds[0]);                     /* the parent uses neither end */
    close(fds[1]);
    waitpid(writer, NULL, 0);
    waitpid(reader, NULL, 0);
    printf("pipeline finished\n");
    return 0;
}

a
b
c
pipeline finished

(The \\n in the C string passes a literal backslash-n to the printf command, which interprets it itself, just as the shell version does.) Note that each child closes both original pipe descriptors after dup2: the copy on 0 or 1 is all it needs. The parent closes both too. If the parent kept fds[1] open, sort would wait forever for more input, because sort cannot print anything until it has seen end of file.

A shell runs a | b | c the same way with two pipes and three children, all started before any is waited for, so they run concurrently, data streaming through the pipes.

popen: the shortcut

For the common case "run a command and read its output", stdio offers FILE *popen(const char *command, "r") and pclose, which do the pipe, fork, exec (through /bin/sh -c) and wait for you. It is convenient, but because it passes the command to a shell, never build that string from untrusted input: popen("grep " + user_input) is a command injection hole. Using fork and execvp with an argument array avoids the shell entirely.

Common mistakes

• Not closing unused pipe ends: the reader never sees end of file and hangs.
• Waiting for the writer before reading a large output. If the child writes more than the pipe buffer holds, it blocks until someone reads, while the parent blocks in waitpid: a deadlock. Read until end of file first, then wait.
• Calling dup2 with the arguments reversed. It is dup2(existing, target).
• Forgetting fflush before exec or fork, losing or duplicating buffered output.
• Mixing up fds[0] and fds[1]. Remember the order of stdin (0) and stdout (1): read end first, write end second.

Recap

• pipe(fds) creates a one-way channel: write into fds[1], read from fds[0].
• read returns 0 only when the pipe is empty and all write ends are closed everywhere; close every unused end right after fork.
• Writing to a pipe with no readers raises SIGPIPE.
• dup2(fd, 0 or 1) before exec is redirection; open + dup2 is > and <, pipe + dup2 is |.
• Read a child's output to end of file before waiting for it.

Going further: read pipe(7) in the sandbox (man 7 pipe, section 7 is installed) for the exact rules on capacity and atomic writes. Then extend the pipeline example to three commands, such as printf ... | sort | uniq -c.