Script files, the shebang and chmod

Why this matters

Everything you have typed so far disappears when you close the shell. A script is a text file of shell commands that you can run again, give to someone else, schedule to run at night, and improve over time. Scripts are also where the precision you learned in the expansion module pays off, because a script runs unattended on data you have not seen. This lesson covers how a text file becomes something you can run, what the kernel does with the first line, and the three different ways of running a script.

A script is just a text file

Create a file hello.sh in your lesson shell. The easiest editor in the sandbox is nano:

$ nano hello.sh

Type:

#!/bin/bash
# hello.sh: my first script
echo "Hello from $(hostname)"
echo "Today is $(date +%A)"

Save with Ctrl-O then Enter, and exit with Ctrl-X. (You could also create it with printf and >; any way of writing text to a file works.)

Run it by handing it to bash:

$ bash hello.sh
Hello from sandbox
Today is Sunday

Here the program is bash, and hello.sh is just an argument: bash opens the file, reads it line by line, and runs each line exactly as if you had typed it. Lines starting with # are comments (the shebang line is a comment too, as far as bash is concerned).

Making it a command: the shebang and chmod

To run it as ./hello.sh, like a real program, two things are needed.

1. The execute permission. The kernel only executes files that have the x bit set for you. You learned chmod in the permissions module:

$ ./hello.sh
bash: ./hello.sh: Permission denied
$ chmod +x hello.sh
$ ls -l hello.sh
-rwxr-xr-x 1 learner learner 87 Nov  1 14:20 hello.sh
$ ./hello.sh
Hello from sandbox
Today is Sunday

2. The shebang. The first line, #!/bin/bash, is called the shebang (from "hash" and "bang", the nicknames of # and !). It tells the kernel which interpreter to use.

What the kernel does with #!

When bash runs ./hello.sh, it forks and calls execve("./hello.sh", ...) as for any program. The kernel opens the file and looks at its first bytes:

• If it finds the magic bytes of a compiled program (ELF, as in /usr/bin/ls), it loads it.
• If it finds #!, it reads the rest of that line as an interpreter path plus an optional single argument, and runs that program instead, with the script's path appended:

 ./hello.sh  with first line  #!/bin/bash
       │
       ▼   kernel rewrites the exec as:
 execve("/bin/bash", ["/bin/bash", "./hello.sh"], env)

So ./hello.sh ends up as /bin/bash ./hello.sh, which is the same as what you typed by hand. The same mechanism runs Python (#!/usr/bin/python3), awk (#!/usr/bin/awk -f) and others. You can see it with file:

$ file hello.sh /usr/bin/ls
hello.sh:    Bourne-Again shell script, ASCII text executable
/usr/bin/ls: ELF 64-bit LSB pie executable, x86-64, ...

You will often see #!/usr/bin/env bash instead. env searches PATH for bash, which helps on systems where bash is not at /bin/bash (some BSDs, macOS with a newer bash installed elsewhere). On Linux both work; this course uses #!/bin/bash.

Note: The shebang must be the very first line, starting at the very first byte. With a blank line or a space before #!, the kernel refuses the file (Exec format error). bash then quietly runs it itself as a bash script, which hides the problem, but other programs that start it, such as cron or a Python program, simply fail.

Why ./?

./hello.sh contains a slash, so bash does no PATH search and runs that exact file. Plain hello.sh would search PATH, which does not include the current directory, and fail with command not found. To run your scripts by name from anywhere, put them in a directory and add it to PATH: commonly ~/bin or ~/.local/bin:

export PATH="$HOME/bin:$PATH"

(You would put that line in ~/.bashrc so it applies to every new shell.)

File names

The .sh extension is only a hint for humans; the kernel and bash ignore it. Many installed commands are scripts without any extension. A common convention is .sh for files you run during development and no extension for commands installed into a PATH directory.

Three ways to run a script

The first two run the script in a child process. Its variables, cds and other changes disappear when it exits, exactly like the bash -c experiments in the expansion module. source (whose POSIX name is .) reads the file into your current shell. That is what you want for files that configure your shell:

$ cat settings.sh
export PROJECT=/work/project
cd "$PROJECT" 2>/dev/null || echo "no project dir yet"
$ bash settings.sh
no project dir yet
$ echo "[$PROJECT]"
[]
$ source settings.sh
no project dir yet
$ echo "[$PROJECT]"
[/work/project]

Your ~/.bashrc is sourced by every interactive bash at startup for the same reason.

The script's exit status

A script finishes with the status of the last command it ran, unless it calls exit N. The lesson after next covers this in depth; for now, note that the status is visible in $? after running it, just as for any program.

Common mistakes

These messages are worth recognizing; you will see all of them.

• Permission denied (status 126): no execute bit. chmod +x script.sh, or run it with bash script.sh.
• command not found (status 127): you typed script.sh without ./, or the directory is not in PATH.
• cannot execute: required file not found or bad interpreter: No such file or directory: the shebang names an interpreter that does not exist. Very often the cause is invisible: the file was saved with Windows line endings (CRLF), so the interpreter's name is /bin/bash plus a carriage return. cat -A script.sh shows ^M$ at line ends; fix with sed -i 's/\r$//' script.sh.
• Shebang not on line 1, or with a typo: #/bin/bash, !#/bin/bash, #! /bin/bsh.
• Expecting the script to change your shell. Variables and cd in a script vanish when it ends; use source if that is what you want.
• Running a bash script with sh script.sh. On Debian sh is dash, a smaller shell, and bash-only features fail with errors like Syntax error: "(" unexpected. Use bash or ./.

Exercises

The two quizzes cover the process model; the tasks are done in the lesson shell, and their checks actually run your script.

Recap

• A script is a text file of commands. bash file runs it with bash; no x permission needed.
• #!/bin/bash on the first line plus chmod +x lets you run it as ./file: the kernel reads the shebang and runs the interpreter with the file as an argument.
• ./ is needed because the current directory is not in PATH.
• source file (. file) runs it in the current shell, so its changes persist.
• Know the errors: Permission denied (126), command not found (127), bad interpreter / CRLF.

Going further: find out which installed commands are really scripts: grep -l '^#!' /usr/bin/* 2>/dev/null | head, then look at the first line of a few of them with head -n 1 to see which interpreters they use.