The Shell, the Terminal and the Kernel · about 35 min

Anatomy of a command

Why this matters

Every command line you will ever write, from ls to a 200-line script, is processed by the same rules. If you know exactly how bash turns a line of text into "run this program with these arguments", most mysterious failures become obvious: the file name with a space that "is not found", the option that is silently ignored, the command that works in your shell but not in a script. This lesson builds that model.

Command, options, arguments

A simple command looks like this:

  ls    -l -a    /etc  /tmp
  ───   ──────   ──────────
  name  options  operands (files to work on)

bash splits the line into words at unquoted spaces and tabs. The first word is the command name; all the words, including the name, are handed to the program as its arguments (the name is argument number zero). That is all bash does with them. bash has no idea that -l is an option: the ls program looks at its arguments and decides that words starting with - are options. That is why every program's options are different, and why conventions matter.

The common conventions (followed by nearly all GNU tools):

Short options are a dash and one letter: -l, -a. They can usually be combined: ls -la means ls -l -a.
Long options are two dashes and a word: ls --all, ls --human-readable. Easier to read in scripts.
Some options take a value: head -n 5 file or head --lines=5 file.
A lone -- means "end of options": everything after it is an operand even if it starts with a dash. rm -- -weird-file deletes a file actually named -weird-file.

Try these in the shell:

$ ls /etc/apt
apt.conf.d  auth.conf.d  keyrings  preferences.d  sources.list.d  trusted.gpg.d
$ ls -a /etc/apt
.  ..  apt.conf.d  auth.conf.d  keyrings  preferences.d  sources.list.d  trusted.gpg.d
$ ls -l -d /etc
drwxr-xr-x. 1 root root 1262 Nov  1 09:01 /etc
$ ls -ld /etc
drwxr-xr-x. 1 root root 1262 Nov  1 09:01 /etc

The last two are the same command; -ld is -l -d combined (-d: show the directory itself, not its contents).

Word splitting, and how quotes change it

bash splits on runs of whitespace, so extra spaces vanish:

$ echo one      two
one two

echo received two arguments, one and two, and printed them separated by one space, which is what echo always does. The spaces you typed were separators, not data. To keep spaces, quote:

$ echo 'one      two'
one      two
$ echo "one      two"
one      two

Now echo received a single argument containing six spaces. bash removes the quotes themselves before running the program (a step called quote removal), so echo never sees them. The difference between single and double quotes matters a great deal later; for now, either keeps spaces together.

This is the number one source of shell bugs: a file called my notes.txt.

$ echo hi > 'my notes.txt'
$ cat my notes.txt
cat: my: No such file or directory
cat: notes.txt: No such file or directory
$ cat 'my notes.txt'
hi

Unquoted, cat received two arguments and looked for two files.

What kind of command is it?

The first word can name several kinds of thing, and bash checks in this order:

Alias: a text shortcut defined in your shell (alias ll='ls -l'). Interactive only.
Function: a named block of shell code (module 11).
Builtin: a command implemented inside bash itself: cd, echo, pwd, type, export.
External program: an executable file found by searching PATH.

The type builtin tells you which one you have:

$ type ll
ll is aliased to `ls -l'
$ type cd
cd is a shell builtin
$ type date
date is /usr/bin/date
$ type -t echo
builtin
$ type -a echo
echo is a shell builtin
echo is /usr/bin/echo

type -a shows every match; the first one wins. There is both a builtin echo and a program /usr/bin/echo, and bash uses its builtin.

command -v name prints just the path (or the name, for builtins), which is handy in scripts. You may also see which; it is an external program that only knows about PATH, so it can be wrong about aliases, functions and builtins. Prefer type.

PATH: how bash finds programs

PATH is a variable holding a colon-separated list of directories:

$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

For a name with no slash in it, bash tries each directory in order, /usr/local/sbin/date, /usr/local/bin/date, ... and runs the first executable file it finds. (It remembers the answer in a small table; hash shows it.) If no directory has it:

$ frobnicate
bash: frobnicate: command not found

If the name contains a slash, there is no search at all: /usr/bin/date or ./myscript runs exactly that file. This is why running a script in the current directory needs ./: the current directory is (deliberately, for safety) not in PATH.

Fork and exec, slightly closer

Underneath, starting an external program uses two system calls:

 bash (PID 100)
   │  fork()                       → kernel makes an almost exact copy of bash
   ├────────────▶ bash (PID 101)    child: same memory, same open files, same directory
   │                 │ execve("/usr/bin/date", ["date"], environment)
   │                 ▼               kernel replaces the program in PID 101 with date
   │              date (PID 101)     runs, writes output, exits with a status
   │  wait()  ◀──────┘
   ▼  prints the next prompt

The PID is the process ID, a number the kernel gives each process. The child inherits copies of the parent's environment variables, current directory and open files (which is how date's output reaches the same terminal), but they are copies: whatever the child changes dies with it. That is exactly why cd must be a builtin.

You can watch PIDs change. $$ expands to the shell's own PID; ps shows processes:

$ echo $$
1
$ ps -o pid,ppid,comm
    PID    PPID COMMAND
      1       0 bash
     12       1 ps

ps is a child of bash (its PPID, parent PID, is bash's PID). In a container bash happens to be PID 1; on a normal system the numbers are larger.

Common mistakes

command not found for a script in the current directory. myscript.sh alone searches PATH; use ./myscript.sh.
Unquoted names with spaces. cat my notes.txt means two files. Quote: cat 'my notes.txt'.
Options after --, or operands that look like options. cat -n waits for input instead of opening a file named -n. Use cat -- -n or cat ./-n.
Trusting which. It does not know about builtins or aliases. type does.
Assuming options are universal. -r means "recursive" for cp and rm, "reverse" for sort and ls. Check each program's documentation (lesson 4).

Recap

bash splits a line into words at unquoted whitespace; the first word is the command name. Quotes keep text together and are removed before the program runs.
Options are just arguments starting with -; each program interprets its own.
A name can be an alias, function, builtin or external program; type tells you which.
External programs are found by searching the directories in PATH, unless the name contains a /.
External programs run in a forked child process via exec, so they cannot change the shell's own state.

Going further: run help type to read the builtin's documentation, and try type -a ls pwd [. Yes, [ is a command; you will learn why in the conditionals module.

Exercises

Exercise 1 · quiz

Counting arguments

How many arguments (not counting the command name itself) does ls receive from this line?

ls -l -a /etc "my dir"

Type a number.

Answer

Four: -l, -a, /etc and my dir. bash splits on unquoted spaces and tabs, and any run of them counts as one separator, so the extra spaces vanish. The double quotes keep my dir together as a single word, and bash removes the quotes before ls ever sees it. Options are just arguments that happen to start with -; it is ls, not bash, that decides they mean something.

Exercise 2 · quiz

Why is cd a builtin?

type cd reports cd is a shell builtin. Why must cd be built into the shell rather than be a separate program like /usr/bin/ls?

Because builtins run faster, and cd is used often.
Because a separate program runs in its own child process, and changing the child's current directory would not change the shell's.
Because only the kernel may change directories, and builtins run inside the kernel.
It is historical accident; a /usr/bin/cd program would work just as well.

Answer

Because a separate program runs in its own child process, and changing the child's current directory would not change the shell's.

The current directory is a property of each process. An external program is started in a forked child; if it changed its own directory and then exited, the shell would be exactly where it was before. So cd has to run inside the shell process itself. The same reasoning applies to other builtins that change the shell's own state, such as export, exit, alias and source. (Builtins do also save a fork, but that is a bonus, not the reason.)

Exercise 3 · task in a shell

Find the program behind a name

In the shell, use type or command -v to find out which file runs when you type date. Save just that path (one line, for example /usr/bin/something) into /work/datepath.txt.

Then find out what kind of command echo is and save the one-word answer printed by type -t echo into /work/echo-kind.txt.

Hint 1

type date prints a sentence; command -v date prints only the path.

Hint 2

Redirect the output into the file: command -v date > datepath.txt, then type -t echo > echo-kind.txt.

Reference solution

command -v date > datepath.txt
type -t echo > echo-kind.txt

Exercise 4 · write code

Where did my spaces go?

The starter script is meant to print these two lines exactly:

one two
a   b

The second line has three spaces between a and b. The starter prints a b with only one. Fix the script so that both lines come out exactly as shown. Still use echo.

File: script.sh

Starting point

echo one two
echo a   b

Example

expected output

one two
a   b

Hint 1

Count the arguments echo receives on the second line. echo prints its arguments separated by a single space.

Hint 2

To make the spaces part of one argument, bash must not split there: put the text in quotes.

Reference solution

echo one two
echo 'a   b'