Processes, Jobs and Signals · about 45 min

Background jobs and wait

Why this matters

Some work is slow because it waits: for disks, for networks, for other programs. Running such jobs one after another wastes time when they could overlap. Ten servers to check, each taking two seconds to answer, take twenty seconds in a loop and two seconds in parallel. The shell can start commands in the background, keep track of them, and wait for them. At the interactive prompt the same machinery, called job control, lets you pause a long command, get your prompt back, and resume it later.

Starting a background job: `&`

A command followed by & is started and not waited for: bash forks, the child runs the command, and bash immediately goes on to the next line.

$ sleep 3 &
[1] 52
$ echo "I did not wait"
I did not wait
$ jobs
[1]+  Running                 sleep 3 &
$                                    (three seconds later, press Enter)
[1]+  Done                    sleep 3

[1] is the job number, local to this shell; 52 is the PID. $! holds the PID of the most recent background job, and is what scripts use:

long_task &
task_pid=$!

& is a command separator like ;, so cmd1 & cmd2 & wait starts two jobs and waits.

Waiting: `wait`

wait makes the shell perform the kernel wait call for its children:

Form	Behavior	Status
`wait`	waits for all background children	always 0
`wait PID`	waits for that child	that child's exit status
`wait -n`	waits for the next child to finish (bash 4.3+)	that child's status

download_a & pa=$!
download_b & pb=$!
wait "$pa" || echo "a failed"
wait "$pb" || echo "b failed"

The important detail: starting a job always "succeeds" ($? after cmd & is 0), and plain wait returns 0 even if every job failed. The only way to learn a job's status is wait PID. bash remembers the status of finished jobs, so it does not matter whether the job ends before or after you call wait on it.

A parallel pattern

pids=""
for host in alpha beta gamma; do
  check_host "$host" > "results/$host.txt" &
  pids="$pids $!"
done

failed=0
for pid in $pids; do
  wait "$pid" || failed=$((failed + 1))
done
echo "$failed checks failed"

$pids is deliberately unquoted in the for: it is a space-separated list of numbers, so word splitting is exactly what we want (an array, module 15, is the tidier way). Each job writes to its own file; if they all wrote to the terminal, their lines would interleave unpredictably.

Warning: Do not start an unbounded number of jobs. Looping & over 10,000 files creates 10,000 processes at once. This sandbox allows 256 processes in total and will refuse forks beyond that (fork: retry: Resource temporarily unavailable). Limit concurrency, for example by waiting with wait -n whenever a counter reaches N, or let xargs -P do it (module 14).

Job control at the interactive prompt

These commands only make sense in an interactive shell (job control is switched off in scripts).

Action	Effect
`cmd &`	start in the background
Ctrl-Z	stop (pause) the foreground job
`jobs`	list jobs with their numbers and state
`fg %1`	bring job 1 to the foreground (plain `fg`: the current job, marked `+`)
`bg %1`	let stopped job 1 continue in the background
`kill %1`	send SIGTERM to job 1
`disown %1`	remove job 1 from the shell's job table

$ sleep 100
^Z
[1]+  Stopped                 sleep 100
$ jobs
[1]+  Stopped                 sleep 100
$ bg
[1]+ sleep 100 &
$ fg
sleep 100
^C
$

%1, %2 refer to job numbers; %sleep refers to the job whose command starts with sleep.

Underneath: process groups and the terminal

How does Ctrl-C reach sleep but not the shell? Through process groups. Every process belongs to a process group (PGID); an interactive bash puts each job (a single command or a whole pipeline) in a new group of its own. The terminal has exactly one foreground process group at a time.

  terminal (/dev/pts/0) ── foreground group: 71
  │
  session: bash (PID 1, group 1)
     ├── job [1]: sleep 100 &        group 60   (background)
     └── job [2]: grep x f | sort    group 71   (foreground: grep and sort)

When you press Ctrl-C, the kernel's terminal driver sends SIGINT to every process in the foreground group, so both grep and sort get it, and nobody else does. Ctrl-Z sends SIGTSTP the same way. fg tells the kernel to make the job's group the foreground group (and sends SIGCONT if it was stopped); bash then waits for it. A background job that tries to read from the terminal is stopped with SIGTTIN, which is why jobs sometimes shows Stopped (tty input).

You can see groups with ps -o pid,pgid,stat,cmd. The + in STAT marks members of the terminal's foreground group.

In a script, job control is off: all processes stay in the script's own group, background jobs get their stdin from /dev/null (so they cannot steal input), and they ignore SIGINT. That is why Ctrl-C on a running script stops the script and its foreground command but leaves its & jobs running, unless the script traps INT and kills them (lesson 3).

Outliving the shell: SIGHUP, nohup and disown

When you close a terminal or your SSH connection drops, the kernel sends SIGHUP ("hang up") to the session, and bash forwards it to its jobs. Their default action is to terminate. Options:

nohup cmd > log 2>&1 &: start cmd with SIGHUP ignored. (Without your redirection, nohup writes output to nohup.out.)
disown %1: remove an already-running job from bash's table, so bash does not send it SIGHUP.
setsid cmd: start cmd in a brand-new session with no terminal at all.

For anything long-running on a server, use a terminal multiplexer (tmux, screen) or a service manager (systemd) instead; they are designed for it.

Common mistakes

Forgetting to wait. A script that starts jobs and ends immediately may report before the work is done, and leaves the jobs running as orphans.
Using plain wait to detect failures. It always returns 0. Wait per PID.
$! read too late. It is overwritten by the next &. Save it immediately.
Interleaved output. Several jobs writing to one terminal or file mix their lines. Give each its own output file and combine afterwards.
Unbounded parallelism. fork: retry: Resource temporarily unavailable means you hit the process limit.
Expecting job control in scripts. fg, bg and %1 print no job control in a script.

Exercises

Two quizzes on job states and wait, then a real parallel script: the checker measures its running time, so the workers must genuinely overlap, and it checks that a failing worker is noticed.

Recap

cmd & starts a job without waiting; $! is its PID.
wait PID returns that job's status; plain wait waits for all and returns 0; wait -n waits for the next one.
Interactive job control: Ctrl-Z, jobs, fg, bg, kill %N, disown.
The terminal sends Ctrl-C and Ctrl-Z signals to its foreground process group; each interactive job is its own group.
SIGHUP ends jobs when the terminal goes away; nohup, disown, setsid, tmux or systemd avoid that.

Going further: help wait, help jobs. In the sandbox shell, start sleep 200 | sleep 300 & and look at ps -o pid,pgid,cmd: both sleeps share one process group. Then try wait -n with two jobs of different lengths.

Exercises

Exercise 1 · quiz

Ctrl-Z, then bg

You start tar czf big.tgz data/ in your interactive shell, then press Ctrl-Z, then type bg. What state is tar in now?

It was killed by Ctrl-Z; bg starts it again from the beginning.
It is stopped (paused) and stays stopped until you type fg.
It is running again, in the background, and you have your prompt back.
It runs in the background and will keep running even if you close the terminal.

Answer

It is running again, in the background, and you have your prompt back.

Ctrl-Z makes the terminal send SIGTSTP to the foreground process group, which stops (pauses) it; nothing is lost. bg sends SIGCONT and lets the job continue in the background, fg would bring it back to the foreground. It is still a child of your shell and still attached to the terminal session, so closing the terminal sends it SIGHUP and it normally dies; disown or nohup are needed to let it outlive the shell.

Exercise 2 · quiz

What does wait return?

A script runs false & p1=$! and true & p2=$!. Which statements are true? Select all that apply.

wait with no arguments waits for both and returns 0, whatever their statuses.
wait "$p1" returns 1, the status of false.
Without any wait, $? right after false & is 1.
wait "$p2" after wait "$p1" returns 0.

Answer

wait with no arguments waits for both and returns 0, whatever their statuses.
wait "$p1" returns 1, the status of false.
wait "$p2" after wait "$p1" returns 0.

Starting a background job always succeeds, so $? after cmd & is 0; the job's real status is only available through wait PID, which returns exactly that status (bash remembers it even if the job finished long ago). Plain wait waits for every child but returns 0, so it cannot tell you about failures. To detect any failure, wait for each PID separately.

Exercise 3 · write code

Count lines in parallel

Write script.sh FILE... that processes each file in a background worker, all at the same time, and reports in argument order.

Each worker must:

sleep 1 first (pretend the work is slow), then
print FILE: N where N is the number of lines (wc -l < FILE), or FILE: missing if the file cannot be read, in which case the worker exits with status 1.

The main script waits for every worker by PID, prints all the report lines in the order the files were given, and exits with status 1 if any worker failed, 0 otherwise.

Because the workers run in parallel, 4 files must take about 1 second, not 4; the checker fails runs that take 2.5 seconds or more. Workers finish in unpredictable order, so have each worker write its line to its own temporary file (numbered by position) and print the files in order at the end. Use mktemp -d and a trap to clean up.

File: script.sh

Starting point

# for each file: a background worker that sleeps 1, then reports "FILE: N" or "FILE: missing"
for f in "$@"; do
  sleep 1
  echo "$f: $(wc -l < "$f")"
done

Example with arguments a.txt b.txt

expected output

a.txt: 3
b.txt: 5

Example with arguments c.txt nope.txt

expected output

c.txt: 1
nope.txt: missing

Hint 1

Put the worker in a function and start it with worker "$f" > "$tmp/$i" &, where i counts the files.

Hint 2

Collect PIDs in a string: pids="$pids $!". Later for pid in $pids; do wait "$pid" || status=1; done (unquoted on purpose: PIDs are plain numbers).

Hint 3

Inside the worker: if n=$(wc -l < "$1" 2>/dev/null); then echo "$1: $n"; else echo "$1: missing"; return 1; fi.

Reference solution

tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT

worker() {
  sleep 1
  local n
  if n=$(wc -l < "$1" 2>/dev/null); then
    echo "$1: $n"
  else
    echo "$1: missing"
    return 1
  fi
}

pids=""
i=0
for f in "$@"; do
  i=$((i + 1))
  worker "$f" > "$tmp/$i" &
  pids="$pids $!"
done

status=0
for pid in $pids; do
  wait "$pid" || status=1
done

j=1
while [ "$j" -le "$i" ]; do
  cat "$tmp/$j"
  j=$((j + 1))
done
exit "$status"

Background jobs and wait

Why this matters

Starting a background job: &

Waiting: wait

A parallel pattern

Job control at the interactive prompt

Underneath: process groups and the terminal

Outliving the shell: SIGHUP, nohup and disown

Common mistakes

Exercises

Recap

Exercises

Ctrl-Z, then bg

What does wait return?

Count lines in parallel

Starting a background job: `&`

Waiting: `wait`