Globbing in depth

Why this matters

You have used *.txt since module 3. Globs are the shell's way of naming groups of files, and they are the right way: they produce real file names as separate words, spaces and all, with no text parsing in between (unlike $(ls), the pitfall from module 12). Knowing the full pattern language, and the handful of options that change how it behaves, lets you select exactly the files you mean, avoid the "no match" surprises, and often replace a find command with something simpler.

Who does the globbing

Globbing, officially pathname expansion, is done by bash, not by the command. It is one of the expansion steps from module 7, after word splitting and before quote removal:

$ ls
a.txt  b.txt  notes.md
$ echo *.txt
a.txt b.txt
$ echo '*.txt'
*.txt

echo never sees a star in the first case: bash read the directory, matched the names, sorted them, and passed two arguments. Quoting a pattern turns globbing off, which is how you pass a literal pattern to a program that does its own matching (find -name '*.txt', lesson 2).

Underneath, bash opens the directory, reads its entries (the getdents system call), keeps the names that match, and sorts them according to the locale (LC_COLLATE). Patterns with slashes are matched one directory level at a time: src/*/test_*.py lists src, keeps the subdirectories, and lists each of those.

The pattern language

Examples, with a directory holding IMG_0001.jpg IMG_12.jpg IMG_0010.JPG report-2024.txt report-final.txt:

$ echo IMG_????.jpg
IMG_0001.jpg
$ echo IMG_[0-9][0-9][0-9][0-9].*
IMG_0001.jpg IMG_0010.JPG
$ echo report-[0-9]*.txt
report-2024.txt
$ echo *[[:upper:]]
IMG_0010.JPG
$ echo report-[!0-9]*
report-final.txt

Important differences from regular expressions (module 6): in a glob * alone means "anything" (in a regex it means "repeat the previous item"), ? is one character (in a regex, "optional"), and a glob must match the whole name, not just part of it. *.txt does not match a.txt.bak.

Note: ranges like [a-z] depend on the locale; in some locales they include uppercase letters. The classes [[:lower:]] and [[:digit:]] always mean what they say. This sandbox uses C.UTF-8, where ranges behave as expected.

Rules about slashes and dots

• * and ? never match a /. * stays inside one directory level.
• A name starting with . (a hidden file) is matched only if the pattern itself starts with an explicit dot: * skips .bashrc, .* matches it. . and .. are never matched by * or by .* in bash 5.2 (older versions matched them with .*, which made rm -r .* dangerous).

When nothing matches

By default an unmatched pattern is passed through unchanged:

$ echo *.log
*.log
$ for f in *.log; do echo "processing $f"; done
processing *.log

Three options change that, set with shopt -s NAME (and unset with shopt -u NAME):

$ shopt -s nullglob
$ for f in *.log; do echo "processing $f"; done
$                                         (loop body never runs)
$ shopt -u nullglob; shopt -s failglob
$ echo *.log
bash: no match: *.log

nullglob is great in loops and with set -- counting, but has a trap: a command given only an unmatched glob gets no arguments, and many commands then do something different. ls *.log with nullglob and no logs runs plain ls, listing everything; cat *.log waits for keyboard input. Without nullglob, the portable idiom is to test each match:

for f in *.log; do
  [ -e "$f" ] || continue     # skips the literal pattern when nothing matched
  gzip -- "$f"
done

More options: dotglob, nocaseglob, globstar

• shopt -s dotglob: * also matches hidden names (still never . and ..).
• shopt -s nocaseglob: case-insensitive matching, so *.jpg also matches IMG.JPG.
• shopt -s globstar: ** matches any number of directory levels, including none.

$ shopt -s globstar
$ echo src/**/*.py
src/main.py src/lib/util.py src/lib/net/http.py src/tests/test_util.py

**/ is a simple alternative to find when you just want all files of a type under a tree. For very large trees, or tests on size, time or type, find (lesson 2) is better: globstar builds the whole list in memory before the command starts.

Extended globs: extglob

With shopt -s extglob, five extra operators take a |-separated list of patterns:

• ?(p1|p2) matches zero or one of the patterns
• *(p1|p2) matches zero or more
• +(p1|p2) matches one or more
• @(p1|p2) matches exactly one
• !(p1|p2) matches anything except the patterns

$ shopt -s extglob
$ ls
a.c  a.h  a.o  b.c  b.o  Makefile  README
$ echo *.@(c|h)
a.c a.h b.c
$ echo !(*.o)
a.c a.h b.c Makefile README
$ echo +([[:digit:]]).txt         # names made only of digits

rm !(*.keep) (remove everything except) is the classic use. extglob must be enabled before bash parses the line containing the pattern, so put shopt -s extglob on its own line, near the top of a script. (Inside [[ ... == pattern ]] extended patterns always work in recent bash.)

Brace expansion is not globbing

{a,b,c} and {1..5} look similar but are a different step: brace expansion generates words whether or not files exist, before any globbing.

$ echo file{1..3}.txt
file1.txt file2.txt file3.txt         (even if none exist)
$ echo [0-9].txt
[0-9].txt                             (no such files: left as is)
$ mkdir -p project/{src,tests,docs}   # creates three directories

Globs elsewhere

The same pattern language appears in case patterns (module 9), in [[ $x == *.txt ]], in parameter expansion like ${file%.*} (module 15), and in find -name. In all of those, matching is done against a string, not against the filesystem.

Common mistakes

• Expecting * to match hidden files. It does not; .* or dotglob.
• Unmatched pattern taken literally: rm: cannot remove '*.tmp': No such file or directory, or a loop running once with the pattern. Use nullglob or a [ -e "$f" ] check.
• nullglob plus a command with no other arguments: ls, cat, grep pat behave very differently with zero file arguments.
• Quoting the glob: "*.txt" is a literal star. Quote the variable parts only: "$dir"/*.txt.
• Regex habits: .* in a glob means "names starting with a dot", not "anything".
• extglob enabled on the same line as its use in a script or function: syntax error near unexpected token '('.
• Argument list too long: a glob in a huge directory can exceed the kernel's limit on arguments. Lesson 3 explains it and the fix.

Exercises

A quiz on unmatched patterns, two hands-on tasks in prepared directories (precise character classes, then extglob's "everything except"), and a counting script that must handle "no matches".

Recap

• bash expands globs before running the command; quoting disables them.
• * any string, ? one character, [...] one character from a set, [!...] not in it, [[:class:]] named classes. Globs match whole names, never /, and skip leading dots.
• Unmatched globs stay literal; nullglob removes them, failglob makes them errors.
• dotglob, nocaseglob, globstar (**) and extglob (!(...), @(...), ...) extend matching.
• Brace expansion generates words regardless of files.

Going further: man 7 glob (installed here) describes the POSIX rules. Run shopt with no arguments to see every option and its state.