Debugging scripts

Why this matters

Shell scripts fail in quiet ways. A variable is empty because of a typo in its name, a file name splits into two words, a comparison is not a comparison at all, and the script just carries on printing wrong numbers. Staring at the source rarely helps, because the bug is in what the line becomes after expansion, which you cannot see in the source. bash can show you exactly that, and a few habits turn hour-long hunts into minutes.

The concept: make bash show its work

Tracing with -x

The xtrace option makes bash print every simple command to standard error, after all expansions and just before running it, prefixed with the value of PS4 (default +):

$ cat greet.sh
name="Ada Lovelace"
file=$name.txt
touch $file
$ bash -x greet.sh
+ name='Ada Lovelace'
+ file='Ada Lovelace.txt'
+ touch Ada Lovelace.txt

The trace quotes words that contain spaces, so touch Ada Lovelace.txt without quotes tells you touch received two arguments: the unquoted $file was split. That one line shows the bug.

Ways to turn it on:

Tracing only the suspicious part keeps the output readable:

set -x
result=$(compute "$input")
set +x

Better trace lines with PS4

PS4 is expanded each time a trace line is printed, so it can include live information:

export PS4='+ ${BASH_SOURCE}:${LINENO}:${FUNCNAME[0]:-main}: '

$ bash -x deploy.sh
+ deploy.sh:12:main: target=/srv/app
+ deploy.sh:30:copy_files: cp -r build /srv/app

Use single quotes so the variables are expanded at trace time, not once when PS4 is assigned. LINENO is the current line number, BASH_SOURCE the file (useful with sourced libraries), FUNCNAME[0] the current function. The first character is repeated to show nesting depth: ++ is a command inside a command substitution or subshell.

To keep the trace out of the program's own error output, send it elsewhere with BASH_XTRACEFD:

exec 5> trace.log      # open file descriptor 5 on trace.log
BASH_XTRACEFD=5
set -x

Syntax check with -n

bash -n script.sh parses without executing. It finds unbalanced quotes, a missing fi or done, a stray (:

$ bash -n broken.sh
broken.sh: line 14: syntax error: unexpected end of file

"Unexpected end of file" means bash was still waiting for something to close, usually a quote, fi, done, esac or } that starts much earlier than line 14.

Inspecting values

• declare -p var prints a variable exactly, as code: declare -- x="a b " reveals a trailing space, declare -a arr=([0]="x" [1]="y z") shows array elements.
• printf '[%s]\n' "$var" shows boundaries; printf '%q\n' "$var" shows hidden characters, for instance $'line\r' for a Windows line ending, a classic cause of $'\r': command not found and of files that "exist but cannot be found".
• cat -A file shows tabs as ^I, carriage returns as ^M and line ends as $.

Stopping at the first error, with a location

Combine strict mode from the robust scripts module with a trap that reports where things went wrong:

set -Eeuo pipefail
trap 'echo "error: status $? at ${BASH_SOURCE}:${LINENO}: $BASH_COMMAND" >&2' ERR

BASH_COMMAND holds the command being executed when the trap fired. -E makes the ERR trap apply inside functions too.

What happens underneath

xtrace is not a separate debugger. It is a hook in bash's main execution path: after bash has parsed a simple command and performed all expansions (parameters, command substitution, word splitting, globbing, quote removal) it has the final list of words it is about to pass to a builtin or to execve. With -x set, it first expands PS4, then prints those words, quoting any that would not survive being re-read as shell input. That is why the trace is the ground truth about word splitting: it is literally the argument vector the command receives.

Because tracing happens per simple command, compound commands appear as their pieces: you see the condition of an if and the commands of the branch taken, but not the if keyword itself. Assignments are shown too, and redirections are not: echo hi > out.txt traces as + echo hi. Keep that in mind when output "disappears".

A method, not just tools

1. Reproduce reliably, with the smallest input that fails.
2. Read the actual error message. line 12: [: too many arguments says the test on line 12 received more words than it expected: an unquoted variable containing spaces.
3. Trace the region around the failure; compare each traced line with what you expected.
4. Check one assumption at a time: does the file exist (ls -l -- "$f")? Is the variable what I think (declare -p)? Does the command behave like that on its own (run it by hand)?
5. Fix, then re-run the original failing case and a passing case.

Common mistakes and the messages they produce

Exercises

Two quizzes on the tools, a task producing a trace with line numbers, and a broken script with three bugs to find and fix using bash -x.

Recap

• bash -x / set -x print each command after expansion: the real arguments, quoted.
• PS4='+ ${BASH_SOURCE}:${LINENO}: ' adds file and line; ++ means nested.
• bash -n checks syntax without running anything.
• declare -p, printf '%q' and cat -A show what a value really contains.
• An ERR trap with $LINENO and $BASH_COMMAND reports where a strict-mode script died.
• Debug by reproducing, reading the message, tracing, and checking one assumption at a time.

Going further: help set lists -x, -v, -n, -u and -E. Try PS4='+ $(date +%s.%N) ${LINENO}: ' bash -x yourscript.sh to get timestamps, a cheap way to find the slow line in a script.