Signals and kill

Why this matters

Signals are how the world talks to a running process from the outside: Ctrl-C at the keyboard, kill from another shell, a container runtime stopping a container, systemd shutting down a service, a pipe whose reader went away. A script that ignores signals leaves half-written files and orphaned children when it is stopped. A script that handles them can stop cleanly. You met trap for cleanup in module 12; this lesson explains signals themselves, how they are delivered, and how to send them.

What a signal is

A signal is a small numbered notification that the kernel delivers to a process asynchronously, that is, at any moment, between any two instructions. Each process has, for each signal, a disposition: default action, ignore, or "call this handler". When a signal arrives the kernel interrupts the process and applies the disposition.

The ones you will use:

kill -l lists them all with numbers. Numbers can differ between operating systems (these are for Linux on x86 and ARM), which is why scripts should use names.

Exit statuses from signals

When a process dies from a signal, the kernel tells the parent "killed by signal N" rather than an exit code; bash reports that in $? as 128 + N:

$ sleep 100 &
[1] 61
$ kill 61; wait 61; echo $?
143
[1]+  Terminated              sleep 100
$ sleep 100 &
[1] 62
$ kill -9 62; wait 62; echo $?
137
[1]+  Killed                  sleep 100

143 = 128 + 15 (TERM), 137 = 128 + 9 (KILL), 130 = Ctrl-C (INT). When you see 137 in a container log, something (often the out-of-memory killer) sent SIGKILL.

Sending signals: kill, pkill, killall

kill PID               # SIGTERM
kill -TERM PID         # the same, explicitly; also kill -s TERM PID, kill -15 PID
kill -INT PID          # as if Ctrl-C
kill -KILL PID         # last resort
kill -0 PID            # sends nothing: only checks the process exists and you may signal it
kill %1                # interactive: a job number
pkill -TERM -f 'python3 server.py'   # by name or full command line (-f)
pgrep -a sleep         # list matching PIDs and command lines first: a dry run for pkill

kill is a bash builtin (so it understands %1), and there is also /usr/bin/kill. You may signal only your own processes, unless you are root. Here the sandbox user cannot touch processes of other users; on a real system kill 1 as a normal user gives bash: kill: (1) - Operation not permitted.

kill -0 is handy in scripts: if kill -0 "$pid" 2>/dev/null; then echo running; fi.

TERM first, KILL last

SIGTERM asks a process to stop and gives it the chance to finish writing, delete temporary files, tell its children and exit with a sensible status. SIGKILL gives it no chance: open files are closed, half-written data stays half-written, temporary files stay, child processes are orphaned. The professional sequence, used by systemd and container runtimes:

kill -TERM "$pid"
for i in 1 2 3 4 5 6 7 8 9 10; do
  kill -0 "$pid" 2>/dev/null || break   # gone?
  sleep 1
done
kill -0 "$pid" 2>/dev/null && kill -KILL "$pid"

kill -9 as a first resort is a habit worth breaking.

How delivery works, and what bash does with it

The kernel marks the signal as pending for the process. The next time that process is about to run in user space, the kernel checks for pending signals and, for a caught signal, makes the process jump into its handler; for an uncaught terminating signal it ends the process. A process blocked in a system call (reading, waiting) is woken up so the signal can be handled.

bash's handlers are your trap strings, and bash runs them between commands. When bash is waiting for a foreground child, a trapped signal is remembered and the handler runs after the child exits. Two consequences:

trap 'echo "got TERM"; exit' TERM
sleep 60        # kill -TERM <script PID>: the handler runs only after sleep finishes!

If only the script receives the signal (not its child), the handler waits up to 60 seconds. The fix is to wait in a way that can be interrupted: run the long command in the background and use the wait builtin, which returns immediately when a trapped signal arrives:

trap 'echo "got TERM"; kill "$child" 2>/dev/null; exit 143' TERM
sleep 60 &
child=$!
wait "$child"

Or keep the foreground steps short, as in the exercises (sleep 0.1 per loop iteration).

Signals are not queued per occurrence: two SIGTERMs arriving before the first is handled count as one. And signals sent to a process do not automatically reach its children; Ctrl-C reaches all of a foreground job only because the terminal signals the whole process group (lesson 2). kill -TERM -- -PGID (a negative number) signals a whole process group yourself.

What children inherit

After fork, a child has the same dispositions as its parent. After exec, caught signals are reset to default (the handler code is gone, replaced by the new program), but ignored signals stay ignored. That is how nohup works: it sets SIGHUP to ignore and then execs your command. bash also resets traps in subshells, so a ( ... ) child starts without the parent's traps, which is why the exercise installs the trap inside the subshell.

SIGPIPE: the signal you meet every day

$ yes | head -n 2
y
y
$ echo "${PIPESTATUS[@]}"
141 0

yes prints y forever. head exits after two lines, closing the read end of the pipe. The next time yes writes, the kernel sends it SIGPIPE, and it dies with 141 (128 + 13). This is a feature: it is how producer | head stops a producer that would otherwise run forever. It becomes a nuisance under set -o pipefail (module 12), where the 141 fails the pipeline. A handler or trap '' PIPE (ignore) makes the write fail with an EPIPE error instead, and well-behaved programs then exit quietly.

Writing a graceful script

#!/usr/bin/env bash
set -euo pipefail

running=1
on_signal() { echo "stopping after current item" >&2; running=0; }
trap on_signal TERM INT

for item in queue/*; do
  [ "$running" = 1 ] || break
  process "$item"           # each item is finished before we check the flag
done
echo "processed up to $item" >&2

Setting a flag in the handler and checking it at safe points is the most robust pattern: the work is never interrupted in the middle of an item.

Common mistakes

• kill -9 first. No cleanup happens. Send TERM, wait, then KILL.
• Trap set in the parent, expected in a ( ) child. Subshells reset traps; set them inside.
• Handler that does not exit. After the handler returns, execution continues where it was, so an endless loop keeps going.
• Long foreground commands delaying the handler. Use background plus wait, or short steps.
• Signal numbers in scripts. kill -15 is fine on Linux, but names (-TERM) are clearer and portable.
• Race at startup. Signalling a child before it has installed its trap kills it with the default action. Use a handshake (a ready file, a line on a pipe) as the exercise does.
• kill: (1234) - No such process: the process already exited; check with kill -0 first if that matters.

Exercises

A quiz on signal facts, then two scripts. In the first you add a handler to a child process that its parent terminates; in the second the checker itself plays the part of systemd and sends your script SIGTERM.

Recap

• Signals are asynchronous notifications; each has a default action, and most can be caught or ignored. SIGKILL and SIGSTOP cannot.
• kill [-SIGNAL] PID sends signals (default TERM); kill -0 tests for existence; pkill/pgrep work by name.
• Death by signal N shows as status 128+N: 130 INT, 137 KILL, 141 PIPE, 143 TERM.
• bash runs traps between commands; long foreground commands delay them; wait is interruptible.
• Exec resets caught signals but keeps ignored ones (nohup); subshells reset traps.
• TERM, wait, then KILL; set a flag in handlers and stop at safe points.

Going further: man 7 signal lists every signal with its default action (it is installed in this sandbox). Try kill -STOP and kill -CONT on a sleep 100 & and watch its state in ps -o pid,stat,cmd.