Reading lines and fields safely

Why this matters

So much of the data you will handle in scripts is line-oriented text: log files, CSV exports, /etc/passwd, lists of host names, the output of other commands. "Do something for every line" is a while read loop. It looks simple, but the naive version silently mangles data: it strips leading spaces, eats backslashes, drops the last line of files that lack a final newline, and can swallow the input of commands inside the loop. This lesson gives you the correct idiom and explains every part of it.

The idiom

while IFS= read -r line; do
    printf '%s\n' "$line"
done < input.txt

Piece by piece:

• read line reads one line from standard input into the variable line. It returns 0 if it read a line, and 1 at end of file, which ends the loop.
• -r means raw: without it, read treats \ as an escape character, removing backslashes and joining lines that end in \. You almost never want that.
• IFS= sets the variable IFS to empty for this read command only (the VAR=value command form puts a variable in that one command's environment). read uses IFS to split and trim; an empty IFS means "keep the line exactly".
• done < input.txt redirects the file into the whole loop. The file is opened once, and each read continues where the last one stopped, because they all share the same open file descriptor and its read position. Contrast read line < input.txt inside a loop, which would reopen the file every time and read the first line forever.

Here is the difference IFS and -r make, on a line with leading spaces and backslashes:

$ printf '  path: C:\\temp\\new\n' > t.txt
$ cat t.txt
  path: C:\temp\new
$ read line < t.txt; echo "[$line]"
[path: C:tempnew]
$ IFS= read -r line < t.txt; echo "[$line]"
[  path: C:\temp\new]

Splitting fields with read

Give read several variable names and it splits the line into fields using IFS, one field per variable; the last variable gets the rest of the line:

$ echo "alice 42 likes tea and cake" | { read -r name age rest; echo "$name|$age|$rest"; }
alice|42|likes tea and cake

Set IFS to a delimiter to parse structured lines. /etc/passwd uses colons:

while IFS=: read -r user _ uid _ _ home shell; do
    echo "$user uses $shell"
done < /etc/passwd

root uses /bin/bash
daemon uses /usr/sbin/nologin
...
learner uses /bin/bash

_ is a conventional throwaway name for fields you do not need. For a CSV without quoted fields, IFS=, does the same. (CSV with quoted commas inside fields needs a real parser; awk or Python handle those better.)

A detail of IFS splitting: whitespace characters in IFS (space, tab, newline) are trimmed and runs of them count as one separator; other characters (like : or ,) separate exactly, so a,,b gives an empty middle field. That is what you want for data files.

The missing last newline

A text file should end with a newline, but many do not (files written by some editors, Windows tools, or printf without \n). On such a final line, read fills the variable but returns 1, so the loop ends without running the body for it:

$ printf 'one\ntwo' | while IFS= read -r l; do echo "got $l"; done
got one

two vanished. The fix is to also run the body when read failed but did read something:

while IFS= read -r line || [[ -n $line ]]; do
    ...
done

Feeding loops: files, pipes and process substitution

Three ways to give a loop its input:

while IFS= read -r line; do ...; done < file.txt          # a file
grep ERROR app.log | while IFS= read -r line; do ...; done  # a pipe
while IFS= read -r line; do ...; done < <(grep ERROR app.log)  # process substitution

The pipe version has a catch: each side of a pipeline runs in its own subshell (a forked copy of the shell), so variables set inside the loop are lost when it ends:

count=0
grep ERROR app.log | while IFS= read -r line; do (( count += 1 )); done
echo "$count"        # prints 0!

The loop incremented the copy's count, and the copy exited. The next lesson looks at this trap in detail; the short fix is process substitution, < <(command), which keeps the loop in the current shell. (<(command) gives a file name like /dev/fd/63 connected to the command's output, and the first < redirects it into the loop.)

Commands inside the loop that read stdin

Everything in the loop body inherits the loop's standard input. A command in the body that reads stdin will eat the rest of your file:

while IFS= read -r host; do
    ssh "$host" uptime        # ssh reads stdin and swallows the remaining host names
done < hosts.txt

Only the first host gets processed. Fix it by giving that command its own input (ssh "$host" uptime < /dev/null, or ssh -n), or by reading the file on another file descriptor:

while IFS= read -r -u 3 host; do
    ssh "$host" uptime
done 3< hosts.txt

3< hosts.txt opens the file as descriptor 3 for the loop, and read -u 3 reads from it, leaving stdin alone.

A worked example

Open the shell for this lesson. sales.csv is already there. Summing a column per region:

#!/bin/bash
north=0 other=0
{
    read -r _header
    while IFS=, read -r region rep amount || [[ -n $region ]]; do
        if [[ $region == north ]]; then
            (( north += amount ))
        else
            (( other += amount ))
        fi
    done
} < sales.csv
echo "north: $north"
echo "other: $other"

$ bash sums.sh
north: 1699
other: 1550

The { ...; } group shares one redirection: the first read consumes the header line, and the loop continues from line 2 on the same open file. Groups run in the current shell, so north and other survive.

Common mistakes

• for line in $(cat file). Iterates over words, not lines, and globs them. Use while IFS= read -r line.
• Forgetting -r. Backslashes disappear: C:\temp becomes C:temp.
• Forgetting IFS=. Leading and trailing spaces disappear. (Fine when you split into fields on purpose.)
• Dropping the last line. Add || [[ -n $line ]].
• Redirecting inside the loop. read line < file in the body rereads line 1 forever.
• Counters lost after a pipe. Use done < file or done < <(cmd) instead of cmd | while.
• A body command eating stdin. ssh, ffmpeg, read itself. Use < /dev/null or read -u 3.

Exercises

ifs-r checks what each part of the idiom protects. users parses passwd-format data; one test case has no newline at the end. The csv-report task has a header to skip, names with spaces, and an unterminated last line.

Recap

• The safe line loop: while IFS= read -r line || [[ -n $line ]]; do ...; done < file.
• read -r a b rest splits on IFS; the last variable gets the remainder. IFS=: or IFS=, for delimited data.
• Redirect into the whole loop (done < file) so one open file is read line by line.
• cmd | while runs the loop in a subshell; use < <(cmd) to keep variables.
• Body commands share the loop's stdin; protect it with < /dev/null or read -u 3 ... 3< file.

Going further: help read documents -d (another delimiter instead of newline), -a (into an array, module 15), -n and -t. Try while IFS= read -r -d '' f; do ...; done < <(find . -print0) again after module 14: it is the fully safe way to read file names.