for loops

Why this matters

The point of scripting is doing the same thing to many things: every log file, every user, every server, every argument. A for loop runs a block of commands once per item in a list, with a variable set to the current item. Combined with globs it is the everyday tool for batch work: convert every image, check every config file, rename a hundred files. Getting it right also depends on everything you learned about word splitting, which is where most loop bugs come from.

The shape

for NAME in WORD1 WORD2 WORD3; do
    COMMANDS     # run once per word, with $NAME set to it
done

bash expands the words after in (with all the usual expansions: variables, globs, braces, command substitution), producing a list. Then, for each item, it assigns the item to NAME and runs the body. do and done delimit the body; do can go on its own line, just like then.

$ for color in red green blue; do echo "I like $color"; done
I like red
I like green
I like blue

After the loop, the variable keeps its last value (echo $color prints blue). The loop's exit status is that of the last command run in the body.

Where the list comes from

Literal words and brace expansion

for server in web1 web2 db1; do echo "checking $server"; done
for n in {1..5}; do echo "attempt $n"; done
for d in {a..c}; do mkdir -p "test-$d"; done
for n in {0..20..5}; do echo $n; done          # 0 5 10 15 20

Brace expansion happens first and only works with literal numbers: {1..$n} does not expand $n (it produces the literal {1..5} when n=5), because braces are expanded before variables. For a range that comes from a variable, use seq or the C-style loop below.

Globs: looping over files

for f in *.log; do
    echo "$f: $(wc -l < "$f") lines"
done

Each file name matched by the glob becomes one item, even if it contains spaces, newlines or other strange characters. That is the key reason to loop over globs: the shell builds the list from the directory itself, not from text output. But remember to quote "$f" inside the body; the loop variable is safe, but using it unquoted in a command splits it again.

When nothing matches, the pattern stays as literal text (*.log), so the body runs once with a name that does not exist. Two defenses:

for f in *.log; do
    [[ -e $f ]] || continue       # skip the unmatched pattern (continue: lesson 4)
    ...
done

shopt -s nullglob                 # unmatched globs now expand to nothing
for f in *.log; do ...; done      # body simply does not run

"$@": looping over arguments

for arg in "$@"; do
    echo "argument: [$arg]"
done

"$@" expands to each argument as a separate word, exactly as given, so "two words" stays one item. Unquoted $@ or $* would split it. This loop is so common that for arg; do (with no in at all) means the same thing.

Command output, carefully

for user in $(cut -d: -f1 /etc/passwd); do echo "$user"; done

This works because user names contain no spaces. The unquoted $(...) is split on whitespace and globbed. For file names or anything that might contain spaces, do not do this; use a glob, find (module 14) or a while read loop (lesson 3).

The C-style for loop

bash also has a counting loop borrowed from C:

for (( i = 1; i <= 5; i++ )); do
    echo "line $i"
done

Inside (( )) are three arithmetic expressions: initialization (run once), condition (checked before each round; the loop stops when it is false) and step (run after each round). Because it is arithmetic, variables work where brace expansion would not:

n=${1:-3}
for (( i = n; i > 0; i-- )); do echo "$i..."; done
echo "liftoff"

$ bash countdown.sh 3
3...
2...
1...
liftoff

A worked example: a quick report

Open the shell for this lesson and try this on the shots directory created for the task (or any directory):

#!/bin/bash
# report.sh: summarize the files in a directory
dir=${1:-.}
count=0
bytes=0
for f in "$dir"/*; do
    [[ -f $f ]] || continue
    size=$(stat -c %s "$f")
    (( count += 1, bytes += size ))
    printf '%6d  %s\n' "$size" "${f##*/}"
done
echo "$count files, $bytes bytes"

$ bash report.sh shots
     8  Screenshot 1.png
     9  Screenshot 12.png
     8  Screenshot 2.png
...
7 files, 51 bytes

New pieces: "$dir"/* quotes the variable part and leaves the * unquoted so it still globs; stat -c %s prints a file's size in bytes; the comma in (( count += 1, bytes += size )) runs two expressions; printf '%6d %s\n' prints the number right-aligned in 6 columns; and ${f##*/} strips everything up to the last /, leaving the bare file name. Note the order: glob results are sorted as strings, so Screenshot 12 comes before Screenshot 2.

What the shell does with a loop

A for loop runs inside the current shell; no new process is created for the loop itself. bash reads the whole for ... done block first (that is why an interactive shell shows a > continuation prompt until you type done), expands the list once, and then executes the body repeatedly. Changing the list variable inside the loop does not change the iterations: the list was already built. External commands inside the body are forked and executed each round, so a loop that runs date 10,000 times starts 10,000 processes. Builtins and expansions are much cheaper; that is worth knowing when a loop feels slow.

Common mistakes

• Looping over $(ls). Breaks on spaces and is slower. Use a glob: for f in *.
• Unquoted $@. Splits arguments with spaces. Use "$@".
• Forgetting the unmatched-glob case. Body runs once with the literal pattern. Use [[ -e $f ]] || continue or shopt -s nullglob.
• {1..$n}. Brace expansion does not see variables. Use for (( i = 1; i <= n; i++ )).
• Quoting the glob. for f in "*.txt" loops once over the literal string *.txt.
• Missing do or ;. for f in a b c echo $f is all one list. bash reports syntax error near unexpected token 'done' if do is missing.

Exercises

how-many and no-match test the two classic surprises. args is checked with arguments that contain several spaces and a literal *, which only survive correct quoting. The rename task gives you a directory of files with spaces in their names; preview your loop with echo mv before running it for real.

Recap

• for x in LIST; do ...; done runs the body once per item; the list is expanded once, up front.
• Loop over files with globs (*.log), never over ls output; quote "$f" in the body.
• Unmatched globs stay literal unless shopt -s nullglob.
• Loop over arguments with "$@" (or for arg; do).
• Brace ranges {1..5} need literal numbers; the C-style for (( ; ; )) takes variables.

Going further: help for. Try shopt -s nullglob dotglob and loop over * in a directory with hidden files to see what dotglob changes. Then time for i in {1..2000}; do /bin/true; done against for i in {1..2000}; do true; done with time to feel the cost of fork and exec.