Files: open, read, write and with

Why this matters

Variables vanish when a program exits. Files are how data survives: configuration, logs, reports, saved games, the CSV export from your bank. Reading and writing text files is one of the most common things real programs do, and it is where several hidden layers meet: the operating system's file descriptors, buffering, character encodings, and line endings. Getting the details right means no lost data, no corrupted accented letters, and no files left open. The with statement, introduced here, is the idiomatic way to guarantee files are closed even when errors happen.

Opening a file

The built-in open(path, mode="r", encoding=None) opens a file and returns a file object:

f = open("notes.txt", encoding="utf-8")
text = f.read()
f.close()
print(text)

path is relative to the program's current working directory (in learnprog's sandbox, /work, where your program lives), or absolute like /etc/hostname.

What happens underneath

open asks the operating system kernel to open the file with the open() system call. The kernel checks permissions, finds the file, and returns a small integer, the file descriptor, an index into the process's table of open files. Remember that stdin, stdout and stderr are descriptors 0, 1 and 2; your file will typically get 3. Python wraps that descriptor in layers:

your code:   f.read()  /  f.write("é\n")
                 │  text layer (TextIOWrapper): str <-> bytes via the encoding,
                 │  newline translation
                 │  buffer layer (BufferedReader/Writer): collects bytes in memory,
                 │  typically 8 KiB at a time
                 │  raw layer (FileIO): read()/write() system calls on fd 3
                 ▼
             kernel  ──>  disk

You can peek: f.fileno() returns the descriptor number, and f.buffer is the binary layer underneath.

Modes

Add "b" for binary mode ("rb", "wb"): you read and write bytes instead of str, with no encoding and no newline translation. Use binary for images, archives and anything that is not text.

Warning: "w" empties the file the instant it is opened. Opening the file you are reading from with "w" destroys your input.

Always give an encoding

Files hold bytes; Python strings hold Unicode characters. In text mode Python decodes bytes to str when reading and encodes when writing, using encoding. If you omit it, Python uses the platform's default (UTF-8 on modern Linux and macOS, but historically a legacy code page on Windows), so the same program can corrupt text on another machine. Write encoding="utf-8" every time. Reading a file with the wrong encoding gives either mojibake (é instead of é) or UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3.

Closing, and the with statement

Every open file holds an OS resource (a descriptor; a process may have only so many, often 1024) and, when writing, data sitting in Python's buffer that has not reached the kernel yet. close() flushes the buffer and releases the descriptor. Forgetting it can lose the last chunk of written data if the program crashes, and leaks descriptors in long-running programs.

The problem with calling close() yourself is exceptions: if anything between open and close raises, close never runs. You could use try/finally, but Python has a better tool:

with open("notes.txt", encoding="utf-8") as f:
    text = f.read()
# here the file is closed, no matter what happened inside
print(len(text))

The with statement calls open(...), binds the result to f, runs the block, and then always closes the file when the block is left: normally, through return, or by an exception. It is equivalent to:

f = open("notes.txt", encoding="utf-8")
try:
    text = f.read()
finally:
    f.close()

File objects are context managers, objects that define what happens on entering and leaving a with block (module 13 shows how to write your own). One with can manage several files:

with open("in.txt", encoding="utf-8") as src, open("out.txt", "w", encoding="utf-8") as dst:
    dst.write(src.read().upper())

Use with for every file you open.

Reading

Several ways, for different needs:

with open("poem.txt", encoding="utf-8") as f:
    whole = f.read()          # the entire file as one str

with open("poem.txt", encoding="utf-8") as f:
    first = f.readline()      # one line, including its "\n" ("" at end of file)

with open("poem.txt", encoding="utf-8") as f:
    lines = f.readlines()     # list of lines, each with its "\n"

with open("poem.txt", encoding="utf-8") as f:
    for line in f:            # one line at a time: the idiomatic way
        print(line.rstrip("\n"))

A file object is an iterator over its lines (lesson 1 of the iteration module). Looping over it reads one buffered chunk at a time, so it works on a 50 GB log file with constant memory, whereas read() and readlines() load everything at once.

Two consequences of the iterator nature:

• Each line keeps its newline. "alpha\n".rstrip("\n") removes it; strip() also removes surrounding spaces. f.read().splitlines() gives a newline-free list in one step.
• The file object has a position. After you have read to the end, further reads return "" and a second loop gets nothing, until you reopen or call f.seek(0) to rewind.

Writing

names = ["ann", "bo", "cy"]
with open("names.txt", "w", encoding="utf-8") as f:
    for name in names:
        f.write(name + "\n")

write(s) writes the string exactly as given and returns the number of characters written; unlike print, it does not add a newline. print can write to a file too, which adds the newline and converts non-strings for you:

with open("squares.txt", "w", encoding="utf-8") as f:
    for n in range(1, 4):
        print(n, n * n, file=f)

squares.txt now contains 1 1, 2 4, 3 9 on three lines.

Buffering

write usually does not touch the disk immediately. It appends to an in-memory buffer, and Python calls the write() system call when the buffer fills or the file is closed. This makes writing many small strings fast (one system call per few thousand bytes instead of one per string). If another program needs to see your data now (say, a log another process is tailing), call f.flush(). Even after a flush, the kernel may hold the data in its own page cache before it reaches the physical disk; os.fsync(f.fileno()) forces that, which databases care about and most scripts do not.

Newlines across platforms

Linux and macOS end lines with \n; Windows uses \r\n. In text mode Python handles this for you: when reading, \r\n and \r are translated to \n ("universal newlines"); when writing, \n is written as the platform's line ending. Your code only ever sees \n. Pass newline="" to disable translation (the csv module needs it, as module 15 explains), and use binary mode to see the raw bytes.

Common mistakes

FileNotFoundError from a relative path.

FileNotFoundError: [Errno 2] No such file or directory: 'data.txt'

Relative paths are resolved against the current working directory, which is where you ran the program from, not necessarily where the script is. Print os.getcwd() to check.

Destroying the input with "w". Opening a file for writing truncates it immediately.

Forgetting newlines when writing. f.write("a"); f.write("b") produces ab on one line.

Reading twice. After f.read(), a second f.read() returns "". The position is at the end.

Writing non-strings. f.write(42) raises TypeError: write() argument must be str, not int. Use str(42) or print(42, file=f).

Mixing str and bytes. In "rb" mode read() returns b'...' bytes; comparing them to "..." strings is always false, and concatenating them raises TypeError.

Recap

• open(path, mode, encoding="utf-8") asks the kernel for a file descriptor and wraps it in text, buffer and raw layers.
• Modes: r read, w truncate and write, a append, x create new; add b for bytes.
• Always use with open(...) as f:; it closes (and flushes) the file on every exit path.
• Iterate the file object to read line by line with constant memory; lines keep their \n.
• write adds no newline; print(..., file=f) does. Writes are buffered until flush or close.
• Text mode translates line endings so your code sees only \n.

Going further: run python3 -c 'f = open("/etc/hostname"); print(f.fileno()); import os; print(os.listdir("/proc/self/fd"))' on Linux to see your process's open descriptors, and read the io module documentation's overview section.