stdin, stdout, stderr and exit codes

Why this matters

A program does not live alone. It is started by a shell, a script, a cron job, a CI pipeline, or another program, and those callers need three things from it: the data it produces, any diagnostics kept separate from that data, and a clear signal of success or failure. Unix provides exactly these channels: standard output, standard error, and the exit status, plus standard input and command-line arguments going in. Programs that use them properly can be combined in pipelines (cat log | python3 filter.py | sort | uniq -c) and automated reliably. This lesson connects what you learned about files to these process-level channels.

The process's view of the world

When a process starts, the kernel gives it:

            argv: ["main.py", "17", "5"]     (command-line arguments)
                         │
 stdin  (fd 0) ──>  ┌────────────┐ ──> stdout (fd 1)   data
                    │  process   │
                    └────────────┘ ──> stderr (fd 2)   diagnostics
                         │
                    exit status: 0..255  (returned to the parent when it ends)

The three standard streams are just file descriptors 0, 1 and 2, opened before your code runs. The shell decides where they point: the terminal by default, or files and pipes when you redirect:

$ python3 main.py < input.txt > output.txt 2> errors.txt
$ python3 main.py | sort        # stdout of python3 feeds stdin of sort
$ python3 main.py 2>/dev/null   # discard diagnostics

Python exposes them in the sys module as file objects: sys.stdin, sys.stdout and sys.stderr. They are text files exactly like the ones open returns, so everything from the files lesson applies.

Reading all of stdin

input() reads one line. To process an unknown number of lines, iterate sys.stdin; the loop ends at end of input (EOF: the end of a redirected file, the writer of a pipe closing it, or Ctrl-D at a terminal):

import sys

total = 0
for line in sys.stdin:
    total += int(line)
print(total)

$ printf '3\n4\n5\n' | python3 sum.py
12

Lines keep their newline, as with any file. int ignores surrounding whitespace, so it does not matter here, but when printing lines back out, strip the newline or use print(line, end=""). Other options: sys.stdin.read() returns everything as one string, and sys.stdin.read().split() gives all whitespace-separated tokens regardless of line structure.

A program that reads stdin, transforms it, and writes stdout is called a filter. Filters are the building blocks of shell pipelines.

stdout versus stderr

print writes to sys.stdout by default. Write diagnostics (errors, warnings, progress, summaries) to sys.stderr:

import sys

print("result data")                             # stdout
print("warning: skipped 3 rows", file=sys.stderr) # stderr
sys.stderr.write("also stderr\n")                # the file-object way

Why separate them? Because stdout is often redirected into a file or another program, and messages mixed into the data would corrupt it. python3 report.py > report.csv should produce a clean CSV while still showing warnings on the terminal. Tracebacks go to stderr for the same reason.

Buffering differences

When stdout is a terminal, Python flushes it at every newline (line buffering). When stdout is a pipe or file, it is block-buffered: output accumulates in memory and is written in chunks. stderr is always written promptly. So if you mix them, the order you see in a combined log may differ from the order of your print calls. Pass flush=True to print when the timing matters, or run Python with -u for unbuffered output.

Command-line arguments: sys.argv

sys.argv is a list of strings: the script name followed by the arguments the user typed.

import sys
print(sys.argv)

$ python3 args.py report.csv --verbose 42
['args.py', 'report.csv', '--verbose', '42']

Everything is a string; convert with int() as needed. The shell has already split the command line on spaces and handled quotes, so python3 args.py "two words" gives ['args.py', 'two words']. Check the length before indexing, or a missing argument becomes IndexError: list index out of range. For anything beyond a couple of positional arguments, the argparse module (module 15) builds proper option parsing and help text for you.

Exit status

When a process ends, it returns a small integer to its parent: the exit status (or exit code). 0 means success; any other value, 1 to 255, means failure. Shells and CI systems branch on it:

$ python3 check.py && echo "all good"   # runs echo only on status 0
$ python3 check.py; echo "status was $?"

How a Python program's status is determined:

Conventions worth following: 1 for general errors, 2 for incorrect usage (wrong arguments), as grep, ls and Python's own argparse do. grep also uses 1 to mean "ran fine, found nothing", which shows that a non-zero status signals "not success" for the caller's purposes, not necessarily a crash.

How sys.exit works

sys.exit(n) does not kill the process on the spot. It raises the exception SystemExit(n). It propagates like any exception, so finally blocks and with statements still clean up on the way out. When it reaches the top level uncaught, the interpreter exits with the code. Because SystemExit inherits from BaseException, not Exception, an except Exception: handler does not catch it. (A bare except: would, which is one more reason not to write one.)

A common pattern for scripts:

import sys

def main():
    if len(sys.argv) != 2:
        print("usage: count.py FILE", file=sys.stderr)
        return 2
    try:
        with open(sys.argv[1], encoding="utf-8") as f:
            n = sum(1 for _ in f)
    except OSError as e:
        print(f"error: {e}", file=sys.stderr)
        return 1
    print(n)
    return 0

sys.exit(main())

main() returns a status code, and one sys.exit at the bottom turns it into the exit status. Errors become one clear line on stderr instead of a traceback, which is the right experience for a user (tracebacks are for developers). The OSError handler covers missing files, permission problems and directories in one place; str(e) already reads well, for example [Errno 2] No such file or directory: 'nope.txt'.

Common mistakes

Diagnostics on stdout. print("Error: bad file") goes into the data stream, breaking whatever consumes your output. Use file=sys.stderr.

Exiting 0 on failure. A script that prints an error and then ends normally reports success; a CI pipeline will happily continue. Exit non-zero.

Indexing argv blindly. sys.argv[1] with no arguments gives IndexError: list index out of range. Check len(sys.argv) and print a usage line.

Double newlines. for line in sys.stdin: print(line) prints blank lines between lines, since each line already has \n. Use end="" or strip.

Waiting forever. A program that reads stdin, run in a terminal with no redirection, waits for you to type. Press Ctrl-D to send EOF.

Catching SystemExit by accident. A bare except: around code that calls sys.exit swallows the exit.

Recap

• Every process has stdin (0), stdout (1), stderr (2) and arguments in; an exit status out. In Python: sys.stdin, sys.stdout, sys.stderr, sys.argv, sys.exit.
• Iterate sys.stdin to read all input until EOF; programs that read stdin and write stdout are filters.
• Data goes to stdout; diagnostics to stderr with print(..., file=sys.stderr).
• sys.argv[0] is the script; arguments follow, all strings.
• Exit 0 for success, non-zero for failure (1 errors, 2 usage); sys.exit raises SystemExit, uncaught exceptions exit with 1.

Going further: run python3 -c 'import sys; sys.exit(3)'; echo $? in a terminal, then try python3 main.py 2>&1 | less to see both streams in one pager. Read the sys module documentation for sys.stdin, sys.exit and sys.argv.