Processes, PIDs and /proc

Why this matters

Everything you run is a process, and a shell script is mostly a machine for creating them: each external command, each pipeline stage, each $(...) is at least one new process. When a script is slow, hangs, leaves things running after it exits, or cannot be stopped, the explanation is in how processes are created, related and ended. This module goes underneath the shell to the kernel's view, and this first lesson gives you the vocabulary and the tools to look.

What a process is, to the kernel

The kernel keeps a record for every process. The important fields:

fork, exec, wait: the whole cycle

Module 1 introduced fork and exec. Here is the full life cycle, now with wait and exit:

   bash (PID 40)
     │ fork()  ─────────────▶  bash (PID 41)   an exact copy: same memory, variables,
     │                          │               directory, open fds; fork returns 0 here
     │                          │ (child sets up redirections: open, dup2, close)
     │                          │ execve("/usr/bin/sort", argv, envp)
     │                          ▼
     │                        sort (PID 41)    same PID, same fds, new program
     │                          │ ... runs ...
     │                          │ exit(0)      memory freed; PID and status kept: a zombie
     │ wait(41) ◀───────────────┘              parent collects status 0; entry removed
     ▼ $? = 0

Points worth holding on to:

• exec keeps the PID and the file descriptors. That is how redirection works: the child opens out.txt as fd 1 before exec, and the new program simply writes to fd 1 without knowing where it goes.
• exec replaces the program completely. Nothing after a successful exec runs in that process. You will use this deliberately with the exec builtin in lesson 4.
• The status lives until someone waits. An exited process whose parent has not waited yet is a zombie (state Z, shown as <defunct>). It holds no memory, only its slot in the process table. A program that never waits for its children accumulates zombies.
• Orphans are adopted. If a parent exits before its child, the child is re-parented to PID 1 (the init process; in a container, whatever runs as PID 1), which waits for it eventually.

Builtins run inside bash with no fork. Subshells ( ... ), command substitutions $( ... ), pipeline stages and background jobs are forks of bash without an exec: a copy of the shell running shell code. This is why they cannot change the parent's variables or directory.

Looking at processes: ps

Open the shell for this lesson.

$ ps
    PID TTY          TIME CMD
      1 pts/0    00:00:00 bash
     23 pts/0    00:00:00 ps
$ ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
learner        1       0  0 14:02 pts/0    00:00:00 bash -l
learner       24       1  0 14:05 pts/0    00:00:00 ps -ef

Plain ps shows processes on your terminal; ps -ef shows every process (-e) in full format (-f). The sandbox is a container with its own PID namespace, so it sees only its own processes and your login shell is PID 1. On a normal machine you would see hundreds, starting with systemd as PID 1.

Choose your own columns with -o:

$ sleep 100 &
[1] 31
$ ps -o pid,ppid,stat,etime,cmd
    PID    PPID STAT     ELAPSED CMD
      1       0 Ss         03:10 bash -l
     31       1 S          00:04 sleep 100
     32       1 R+         00:00 ps -o pid,ppid,stat,etime,cmd

The STAT letters: R running or runnable, S sleeping (waiting for something: time passing, input, a child), D uninterruptible sleep (usually disk I/O), T stopped, Z zombie. Extra characters: s session leader, + in the foreground process group (lesson 2).

pstree -p draws the parent-child relationships as a tree, and pgrep sleep prints the PIDs of processes by name. top gives a live view; press q to leave it.

The shell gives you a few PIDs directly:

/proc: the kernel's process table as files

Linux exposes every process as a directory under /proc, named by its PID. These are not files on disk; the kernel generates their contents when you read them. /proc/self always points to the process doing the reading.

$ ls /proc/$$
attr  cmdline  cwd  environ  exe  fd  limits  maps  status  ...
$ cat /proc/$$/status | head -9
Name:    bash
Umask:    0022
State:    S (sleeping)
Tgid:    1
Ngid:    0
Pid:    1
PPid:    0
...
$ readlink /proc/$$/cwd
/work
$ readlink /proc/$$/exe
/usr/bin/bash
$ ls -l /proc/$$/fd
lrwx------ 1 learner learner 64 Nov  1 14:10 0 -> /dev/pts/0
lrwx------ 1 learner learner 64 Nov  1 14:10 1 -> /dev/pts/0
lrwx------ 1 learner learner 64 Nov  1 14:10 2 -> /dev/pts/0
lrwx------ 1 learner learner 64 Nov  1 14:10 255 -> /dev/pts/0

• cmdline: the argument vector, with each argument terminated by a NUL byte (\0). NUL is the only byte that cannot appear inside an argument, so this layout is unambiguous, unlike any text form. tr '\0' ' ' < /proc/$$/cmdline makes it readable.
• environ: the environment the process started with, also NUL-separated.
• cwd, exe: symbolic links to the current directory and the program file.
• fd/: one symbolic link per open file descriptor. Here 0, 1, 2 all point to the terminal. Number 255 is bash's own private copy of the terminal (in a script, the script file); bash keeps it high to stay out of your way.
• status: a human-readable summary: name, state, PIDs, user IDs, memory.

Try watching a redirection happen:

$ sleep 60 > /tmp/out.txt &
[1] 40
$ ls -l /proc/40/fd
lr-x------ 1 learner learner 64 Nov  1 14:12 0 -> /dev/null
l-wx------ 1 learner learner 64 Nov  1 14:12 1 -> /tmp/out.txt
lrwx------ 1 learner learner 64 Nov  1 14:12 2 -> /dev/pts/0

fd 1 of the sleep points at the file: bash set that up in the child before exec. (Background jobs in a non-interactive shell get stdin from /dev/null; you will see why in lesson 2.)

Tools like ps, top and pgrep are just programs that read /proc. You can do it yourself, as the exercises show.

How many processes does a line create?

count=$(grep -c error app.log | tr -d ' ')

bash forks a subshell for $( ); inside it, it forks twice more for the two pipeline stages, which exec grep and tr. Three forks, two execs, for one line. Forks are cheap on Linux (memory is shared copy-on-write until someone writes to it), but in a loop over 100,000 lines they add up: a builtin such as ${var//x/y} can be a thousand times faster than $(echo "$var" | sed ...). This is the real reason for preferring builtins inside loops.

Common mistakes

• Thinking $$ changes in a subshell. It does not; use $BASHPID.
• Reading /proc/PID/cmdline as text. Arguments are NUL-separated; cat runs them together.
• Expecting a child to change the parent. cd, variables and exit in a child process affect only the child.
• Leaving background processes behind. A script that starts cmd & and exits leaves cmd running, re-parented to PID 1. Kill or wait for what you start (lessons 2 and 3).
• "kill zombie" attempts. A zombie is already dead; kill -9 does nothing. Fix its parent.

Exercises

Two quizzes on the process model, then two scripts that look at real processes through /proc: one at itself, one at a child it creates.

Recap

• Every process has a PID, a parent (PPID), a program, an environment, a directory and file descriptors.
• fork copies, exec replaces the program (same PID and fds), exit leaves a zombie until the parent waits.
• Subshells, $( ), pipeline stages and & jobs are forks of bash without exec.
• ps -ef, ps -o ..., pstree -p, pgrep show processes; /proc/PID/ exposes the kernel's record: status, cmdline, environ, cwd, exe, fd/.
• $$ is the script, $BASHPID the current process, $! the last background job.

Going further: run man 5 proc and search for /proc/pid/status. Then compare tr '\0' '\n' < /proc/$$/environ with the output of env, and explain any differences (hint: environ is the environment at start).