Word splitting, IFS and why you quote

Why this matters

If you remember one rule from this module, make it this: put double quotes around every variable and command substitution, unless you have a specific reason not to. This lesson explains why, by looking closely at step 5 of the expansion order, word splitting, and at its partner, globbing of expansion results. These two steps cause more broken scripts than anything else in bash: scripts that work for months and then fail on the first file name containing a space, or delete the wrong files when a variable is empty.

A tool for seeing arguments

echo hides the problem because it joins its arguments with single spaces. Use printf with a format that brackets each argument instead:

$ printf '[%s]\n' one "two  three" four
[one]
[two  three]
[four]

printf reuses its format for every remaining argument, so each argument appears on its own line inside brackets. You now see exactly what a program would receive. Keep this trick; you will use it to debug quoting for years.

What word splitting does

After parameter expansion, command substitution and arithmetic expansion, bash looks at the results that were not inside double quotes and splits them into words, using the characters in the IFS variable (internal field separator) as delimiters. By default IFS is space, tab and newline.

$ v='alpha beta   gamma'
$ printf '[%s]\n' $v
[alpha]
[beta]
[gamma]
$ printf '[%s]\n' "$v"
[alpha beta   gamma]

With the default IFS, whitespace splitting has these rules:

• leading and trailing whitespace is dropped,
• any run of spaces, tabs and newlines counts as one separator.

Newlines are whitespace too, which is why the output of a command with many lines turns into many words:

$ printf '[%s]\n' $(printf 'a b\nc\n')
[a]
[b]
[c]

Two things are not split: text you typed literally (it was tokenized once, in step 1), and anything inside double quotes.

Empty results disappear

An unquoted expansion that produces nothing, or only whitespace, produces no word at all. It does not become an empty argument; it vanishes:

$ empty=''
$ printf '[%s]\n' a $empty b
[a]
[b]
$ printf '[%s]\n' a "$empty" b
[a]
[]
[b]

That is harmless in printf and dangerous elsewhere:

backup_dir=$(cat /etc/backup-dir-config)   # suppose the file is missing: empty
rm -rf $backup_dir/*                         # becomes: rm -rf /*

With "$backup_dir"/*, the empty value gives /* too, so quoting alone does not save you here, but it does in many cases, and module 12 adds the checks (${var:?}, set -u) that catch the rest.

Globbing of the results

After splitting, each resulting word that contains an unquoted *, ? or [ is treated as a file name pattern. So the content of a variable can trigger globbing:

$ ls
a.txt  b.txt
$ msg='Warning: * is not allowed'
$ echo $msg
Warning: a.txt b.txt is not allowed
$ echo "$msg"
Warning: * is not allowed

User input, data from files and command output all go through this if you leave them unquoted.

Changing IFS

Because word splitting uses IFS, you can split on other characters by changing it:

$ line='alice:x:1000:1000:Alice:/home/alice:/bin/bash'
$ IFS=:
$ printf '[%s]\n' $line
[alice]
[x]
[1000]
[1000]
[Alice]
[/home/alice]
[/bin/bash]
$ unset IFS

unset IFS restores the default behavior. Non-whitespace IFS characters follow stricter rules: each one is a separator on its own, so two colons in a row produce an empty field between them:

$ IFS=,
$ v='a,,b'
$ printf '[%s]\n' $v
[a]
[]
[b]
$ unset IFS

Setting IFS for a whole interactive shell is confusing, since it affects every unquoted expansion afterwards. In scripts, change it briefly or only for one command. The cleanest place is the read builtin (next module), which accepts its own IFS: IFS=: read -r user _ uid rest.

Setting IFS='' (empty) turns word splitting off completely.

Turning globbing off: set -f

If you deliberately split an unquoted value but do not want globbing, set -f disables pathname expansion (set +f turns it back on):

$ v='*.txt|*.csv'
$ IFS='|'
$ set -f
$ printf '[%s]\n' $v
[*.txt]
[*.csv]
$ set +f; unset IFS

When quoting is the wrong answer

Sometimes you want splitting, for example a variable holding a few options:

opts='-l -h'
ls $opts /etc      # deliberately unquoted: two separate options

This works for simple, space-free words that you control. It breaks as soon as one of the items contains a space (a file name, say), because there is no way to quote inside a string variable and have the shell respect it later: quotes inside a variable's value are just characters.

$ args='"my file.txt" other.txt'
$ printf '[%s]\n' $args
["my]
[file.txt"]
[other.txt]

The right tool for lists of things is an array (module 15), or "$@" for script arguments (next module).

The rule, restated

Assignments are a special case: x=$y and x=$(cmd) are not split or globbed, so quotes there are optional. Many people quote anyway for consistency, which never hurts.

Common mistakes

• Unquoted variables holding file names. cat $file with file='my notes.txt' gives cat: my: No such file or directory and cat: notes.txt: No such file or directory.
• ambiguous redirect. wc -l < $f with a space in $f gives bash: $f: ambiguous redirect, because a redirection needs exactly one word. Quote: < "$f".
• Storing quotes in a variable and expecting the shell to honor them later. It will not.
• Empty variables vanishing. ls $dir lists the current directory if dir is empty.
• Forgetting to restore IFS. Later unrelated commands split oddly. unset IFS afterwards.

Exercises

Use printf '[%s]\n' in the shell to check your answers to the quizzes. The code exercises run as bash script.sh in a directory that contains the files they need.

Recap

• Unquoted results of $var, $(...) and $((...)) are split on IFS characters and then globbed.
• Default IFS: whitespace runs are one separator; leading and trailing whitespace vanish.
• An unquoted empty expansion produces no argument at all.
• printf '[%s]\n' ... shows the exact arguments.
• IFS=: splits on colons; set -f turns globbing off; unset IFS restores the default.
• Double-quote expansions by default. Lists belong in arrays or "$@", not in strings.

Going further: read the "Word Splitting" part of the bash manual (man bash, search /Word Splitting). Try IFS=$'\n' with printf '[%s]\n' $(ls) in a directory with spaced file names, and think about why that is still fragile (hint: newlines are legal in file names).