Globbing, find and xargs · about 45 min

xargs and null-separated names

Why this matters

find produces lists of files; many tools want those files as arguments. Joining the two correctly is where scripts traditionally fall apart: a name with a space, a quote or a newline splits into pieces, and the pieces get passed to rm. And when the list is huge, a different failure appears: Argument list too long. This lesson explains both problems down at the kernel level and shows the robust tools: xargs, find -exec +, NUL separators, and bash's NUL-aware read.

Why xargs exists

Programs receive input in two different ways: as arguments (rm a b c) or on stdin (sort < file). Pipes connect stdout to stdin, so they can only feed the second kind:

find . -name '*.tmp' | rm        # does nothing useful: rm ignores stdin

xargs bridges the gap: it reads items from stdin and runs a command with those items appended as arguments.

$ printf 'a\nb\nc\n' | xargs echo rm
rm a b c
$ printf 'a\nb\nc\n' | xargs -n 1 echo rm
rm a
rm b
rm c

(echo in front is a good dry run: it prints the command instead of running it.) With no command, xargs runs echo.

Useful options:

Option	Meaning
`-0`	items are separated by NUL bytes; no quote or backslash processing
`-n N`	at most N items per command
`-I {}`	run once per item, replacing `{}` in the command (`-I {} mv {} {}.bak`)
`-r`	do not run the command at all if the input is empty (GNU)
`-P N`	run up to N commands in parallel
`-t`	print each command to stderr before running it

Without -r, GNU xargs runs the command once even with empty input: find . -name '*.x' | xargs rm with no matches runs plain rm, which complains rm: missing operand.

The kernel limit: ARG_MAX

When bash runs an external program, it passes the arguments and the environment to the kernel's execve system call, which copies them into the new process's memory. The kernel caps the total:

$ getconf ARG_MAX
2097152

About 2 MB for arguments plus environment together (on Linux it is a quarter of the stack size limit), and each single argument can be at most 128 KiB. Exceed it and execve fails with the error E2BIG, which bash reports as:

bash: /usr/bin/rm: Argument list too long

Note who complains: bash expanded *.tmp into half a million words without trouble; it is the exec of rm that fails. Builtins are never exec'd, so echo *.tmp and for f in *.tmp work with any number of files, and so does printf '%s\0' *.tmp | xargs -0 rm, because printf is a builtin.

xargs knows the limit: it packs as many items into each command as fit, then starts another command for the rest. find -exec cmd {} + does the same. So both handle any number of files.

The separator problem, and NUL

By default xargs splits on blanks and newlines and interprets quotes and backslashes:

$ printf "my file.txt\nit's.txt\n" | xargs echo
xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option

And find's normal output separates names with newlines, which are legal in file names. There is exactly one byte that can never appear in a Unix path: the NUL byte (value 0), because the kernel's interfaces use it to mark the end of a string. So the robust protocol is:

  find ... -print0          writes   name1\0name2\0name\nwith newline\0
  xargs -0 cmd              splits   on \0 only, no quote processing

find . -type f -name '*.log' -print0 | xargs -0 gzip --
find . -name '*.tmp' -print0 | xargs -0 -r rm --

Other tools that speak NUL: sort -z, grep -z (lines end in NUL) and grep -lZ (print matching file names NUL-terminated), du --files0-from, tar --null -T -, and printf '%s\0'.

The -- protects against names starting with -, which find does not produce when started from . (./-g.log), but which you get when the starting point is a bare name like data.

Reading NUL-separated input in bash

When the per-file work is shell code rather than one command, use a loop. read -d '' makes read stop at a NUL byte instead of a newline:

while IFS= read -r -d '' file; do
  size=$(stat -c %s -- "$file")
  printf '%8d  %s\n' "$size" "$file"
done < <(find . -type f -name '*.log' -print0)

Each piece has a purpose:

-d '': the delimiter is the empty string, whose first byte is NUL.
-r: backslashes are kept literally.
IFS=: leading and trailing spaces are kept.
< <(find ...): process substitution, so the loop runs in the current shell and variables set inside it survive (module 13, lesson 4). find ... | while ... also works if you do not need them afterwards.

Choosing between them

One command per batch of files: find ... -exec cmd {} +
Need xargs features (parallelism, -n, a list that comes from elsewhere): find ... -print0 | xargs -0 cmd
Shell logic per file: while IFS= read -r -d '' f; do ...; done < <(find ... -print0)
Files in one directory: a glob, for f in *.log

-exec + needs no extra process and is the simplest safe choice; reach for xargs when you need what it adds.

Parallel work with xargs -P

find photos -name '*.png' -print0 | xargs -0 -P 4 -n 10 optipng -quiet --

Up to 4 optipng processes at once, 10 files each. This is the bounded parallelism that module 13 warned you to use instead of an unbounded & loop. Output from parallel commands can interleave, so let each command write to its own file when the output matters.

Common mistakes

Plain find | xargs: breaks on spaces, quotes and newlines (xargs: unmatched single quote). Always pair -print0 with -0.
-print0 without -0 (or the reverse): one giant argument, or names glued together.
Missing -r: with empty input the command runs once with no file arguments.
xargs -I {} is safe for spaces (it splits on newlines only) but runs one command per item, so it is slow, and still breaks on newlines unless combined with -0.
$(find ...) as an argument list: word splitting plus ARG_MAX, both at once.
Forgetting -- when names may start with -.

Exercises

Two quizzes on the kernel limit and xargs's splitting, a task in the sandbox where file names include a newline (look at ls -b data to see it), and a script that reads NUL-separated input.

Recap

xargs turns stdin items into command arguments, in batches that respect the kernel's ARG_MAX.
Argument list too long is execve's E2BIG; builtins are immune; -exec + and xargs avoid it.
NUL is the only byte not allowed in file names: find -print0 | xargs -0 and while IFS= read -r -d '' handle every possible name.
xargs -r skips empty input; -n, -I, -P control batching and parallelism.
Prefer find -exec ... {} + when one command per batch is enough.

Going further: xargs --help, and getconf -a | grep ARG_MAX. Try printf '%s\0' * | xargs -0 -t -n 2 echo in a directory with a few files to watch how xargs batches.

Exercises

Exercise 1 · quiz

Argument list too long

In a directory with 500,000 files, rm *.tmp fails with bash: /usr/bin/rm: Argument list too long. Which statements are true? Select all that apply.

The error comes from the kernel's execve call: the arguments (plus the environment) exceed its size limit.
bash could not expand the glob because the directory has too many entries.
echo *.tmp still works, because echo is a builtin and no execve happens.
find . -maxdepth 1 -name '*.tmp' -exec rm {} + avoids the problem.
Increasing ulimit -n fixes it.

Answer

The error comes from the kernel's execve call: the arguments (plus the environment) exceed its size limit.
echo *.tmp still works, because echo is a builtin and no execve happens.
find . -maxdepth 1 -name '*.tmp' -exec rm {} + avoids the problem.

bash expands the glob without trouble; the list only becomes a problem when it is handed to a new program, because execve copies all arguments and environment strings into the new process and refuses (E2BIG) beyond a limit (getconf ARG_MAX, about 2 MB here, further limited per string). Builtins never exec, so echo *.tmp or a for loop over the glob are fine. find ... -exec rm {} + and xargs split the list into several commands that each fit. ulimit -n is the limit on open files, unrelated.

Exercise 2 · quiz

How xargs reads its input

printf 'my file.txt\nother.txt\n' | xargs rm is run in a directory containing my file.txt and other.txt. What happens?

Both files are removed.
rm is asked to remove my, file.txt and other.txt; other.txt is removed and rm complains about the other two.
Nothing happens, because xargs needs -0 to work at all.
xargs refuses to run because of the space.

Answer

rm is asked to remove my, file.txt and other.txt; other.txt is removed and rm complains about the other two.

Without options, xargs splits its input on any whitespace (and also interprets quotes and backslashes), so the name my file.txt becomes two arguments. That makes plain xargs unsafe for file names. With NUL-separated input (find ... -print0 | xargs -0 rm) every name arrives intact, because NUL is the one byte that cannot occur in a file name. xargs -d '\n' (split on newlines only) is a weaker fix that still fails on names containing newlines.

Exercise 3 · task in a shell

Compress awkward log files

Under /work/data there are .log files at several depths, with awkward names: one with a space, one starting with a dash, and one with a newline inside its name (ls shows it as 'd'$'\n''e.log').

Compress every .log file under data with gzip, in one pipeline using find ... -print0 and xargs -0, so that each X.log is replaced by X.log.gz. Do not touch keep.txt.

Hint 1

find data -type f -name '*.log' -print0 prints each name followed by a NUL byte.

Hint 2

xargs -0 splits its input on NUL bytes only. Add -- before the names so -g.log is not taken as an option.

Hint 3

find data -type f -name '*.log' -print0 | xargs -0 gzip --

Reference solution

find data -type f -name '*.log' -print0 | xargs -0 gzip --

Exercise 4 · write code

Read NUL-separated names in bash

script.sh reads file names from stdin, separated by NUL bytes (as find -print0 writes them), and prints two lines:

names: N
with spaces: M

N is the number of names, M the number of names containing at least one space. Names may contain newlines; a name with a newline is still one name.

The test feeds it, for example, the three names a, b c and d<newline>e, expecting names: 3 and with spaces: 1; and empty input, expecting names: 0 and with spaces: 0. The starter reads lines, which miscounts.

File: script.sh

Starting point

names=0
spaces=0
while IFS= read -r name; do
  names=$((names + 1))
  case $name in *" "*) spaces=$((spaces + 1)) ;; esac
done
echo "names: $names"
echo "with spaces: $spaces"

Hint 1

read -d X stops at the character X instead of a newline. An empty string means the NUL byte.

Hint 2

while IFS= read -r -d '' name; do ...; done

Reference solution

names=0
spaces=0
while IFS= read -r -d '' name; do
  names=$((names + 1))
  case $name in *" "*) spaces=$((spaces + 1)) ;; esac
done
echo "names: $names"
echo "with spaces: $spaces"