Debugging and undefined behavior · about 50 min

Debugging with gdb

Why this matters

Sooner or later a C program will crash with nothing but Segmentation fault (core dumped), or it will print a wrong number with no clue as to why. Adding printf calls everywhere works for small programs, but it is slow, it changes the program you are studying, and it is no help when the output never appears because the crash happened first. A debugger lets you stop a running program at any line, look at every variable, walk up and down the call stack, and watch memory change. gdb, the GNU debugger, is the standard on Linux, and the skills transfer directly to every other debugger.

How a debugger works

A debugger is a separate process that controls your program through a kernel facility called ptrace. It can stop the program, read and write its memory and registers, and resume it. To set a breakpoint, gdb overwrites the first instruction of a line with a special trap instruction; when the CPU reaches it, the kernel stops your program and tells gdb.

To map machine code back to your source, gdb needs debugging information, which -g adds to the executable: which addresses belong to which line, where each variable is stored, what type it has. That is why every build in this track uses -g. It is also why we use -O0: with optimization, variables live in registers or vanish entirely and lines get reordered, so the debugger shows <optimized out> and jumps around confusingly.

   gdb process                       your program (stopped)
  +------------------+   ptrace    +-----------------------------+
  | reads -g info:   | ----------> | memory: code, stack, heap   |
  | line 7 = 0x1189  |  read/write | registers: rip, rsp, ...    |
  | sum is at rbp-4  | <---------- |                             |
  +------------------+             +-----------------------------+

A first session

Take this program, which should print the average of four numbers and then crashes:

#include <stdio.h>

int average(const int *v, int n)
{
    int sum = 0;
    for (int i = 0; i <= n; i++)
        sum += v[i];
    return sum / n;
}

int main(void)
{
    int *p = NULL;
    int data[4] = {3, 5, 7, 9};
    printf("%d\n", average(data, 4));
    printf("%d\n", *p);
    return 0;
}

Start gdb on the executable (not the source) and run it:

$ gcc -std=c17 -Wall -Wextra -g -O0 -o prog main.c
$ gdb -q ./prog
Reading symbols from ./prog...
(gdb) run
Starting program: /work/prog

Program received signal SIGSEGV, Segmentation fault.
0x000055b4de34b1d9 in main () at main.c:16
16        printf("%d\n", *p);
(gdb) info locals
p = 0x0
data = {3, 5, 7, 9}

In one command gdb shows the exact line and that p is the null pointer (0x0). SIGSEGV is the signal the kernel sends when a program touches memory it has not mapped; address 0 is never mapped, precisely so that null pointer dereferences crash loudly. (Addresses like 0x000055b4... differ on every run; ignore them.) In the sandbox you will also see a harmless warning: Error disabling address space randomization.

Note: when stdout is a pipe or file rather than a terminal, output is buffered and a crash throws away whatever was not yet written. That is why a crashing program can appear to die before a printf that actually ran. A debugger sidesteps the question entirely.

Breakpoints and stepping

Now the subtler bug: average reads one element too many. Stop at the start of it:

(gdb) break average
Breakpoint 1 at 0x1144: file main.c, line 5.
(gdb) run
Breakpoint 1, average (v=0x7ffd89719910, n=4) at main.c:5
5        int sum = 0;
(gdb) next
6        for (int i = 0; i <= n; i++)
(gdb) next
7            sum += v[i];
(gdb) print v[0]@4
$2 = {3, 5, 7, 9}
(gdb) info args
v = 0x7ffd89719910
n = 4

Stepping through the loop with next and print i would show i reaching 4, one past the last valid index. The commands you will use most:

command	short	what it does
`run [args]`	`r`	start the program, optionally with arguments (`run < input.txt` works too)
`break FUNC` / `break FILE:LINE`	`b`	stop when execution reaches that point
`next`	`n`	run the current line; calls run to completion
`step`	`s`	run the current line, entering any function it calls
`finish`		run until the current function returns, print the return value
`continue`	`c`	resume until the next breakpoint or the end
`print EXPR`	`p`	evaluate a C expression: `p sum`, `p v[i] * 2`, `p *ptr`
`print ARR@N`		print N elements starting at ARR: `p v[0]@4`
`display EXPR`		print EXPR again after every step
`info locals` / `info args`		all local variables / the function's arguments
`backtrace`	`bt`	the call stack
`frame N` / `up` / `down`	`f`	select a stack frame to inspect
`list`	`l`	show source around the current line
`watch EXPR`		stop whenever the value of EXPR changes
`quit`	`q`	leave gdb

Pressing Enter on an empty line repeats the last command, so stepping through a loop is just next followed by Enter, Enter, Enter.

The call stack in the debugger

When a crash happens deep inside nested calls, bt shows how you got there:

(gdb) run
Program received signal SIGSEGV, Segmentation fault.
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
(gdb) bt
#0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
#1  0x0000561274dd5174 in field_length (i=2) at crash.c:8
#2  0x0000561274dd519c in total_length (n=4) at crash.c:15
#3  0x0000561274dd51c0 in main () at crash.c:21

Each numbered line is one stack frame, exactly the frames you learned about in the functions module. Frame 0 is inside the C library's optimized strlen (the exact name depends on your CPU). Library code is almost never the culprit: look for the deepest frame in your code, here field_length with i=2. Then frame 1 and print fields[i] show 0x0: someone passed a null pointer to strlen.

Watchpoints

A watchpoint stops the program whenever a value changes, which finds "who overwrote my variable?" bugs that are otherwise very hard to track down:

(gdb) break total_length
Breakpoint 1 at 0x1181: file crash.c, line 13.
(gdb) run
Breakpoint 1, total_length (n=4) at crash.c:13
13        size_t total = 0;
(gdb) next
14        for (int i = 0; i < n; i++)
(gdb) watch total
Hardware watchpoint 2: total
(gdb) continue
Hardware watchpoint 2: total

Old value = 0
New value = 5
total_length (n=4) at crash.c:14
(gdb) continue
Hardware watchpoint 2: total

Old value = 5
New value = 8

A watchpoint on a local variable only makes sense while that variable exists, so stop inside its function first (and step past its initialization). When the function returns, gdb deletes the watchpoint and tells you so.

Batch mode and core dumps

gdb can run commands non-interactively, handy in scripts:

$ gdb -q -batch -ex run -ex bt ./prog

A core dump is a snapshot of a crashed process's memory written to disk, which lets you run gdb ./prog core after the fact. Many systems disable or redirect core files (ulimit -c shows the size limit; 0 means none), so running the program under gdb directly is usually simpler.

Common mistakes

Running gdb on the source file. gdb main.c fails with not in executable format: file format not recognized. Give it the compiled program.
Forgetting -g. gdb prints (No debugging symbols found in ./prog), break on line numbers fails, and backtraces show only addresses. Rebuild with -g.
Debugging optimized code. Variables show <optimized out> and next jumps backwards. Rebuild with -O0 while debugging.
Blaming the library. If frame 0 is strlen, printf or free, the bug is almost always in the arguments your code passed. Walk up with bt and frame.
Expecting the crash line to be the bug. The crash is where bad data was used. The bug is where it was created, often much earlier. Watchpoints and breakpoints earlier in the flow help you go back in time.

Recap

Compile with -g -O0, then gdb ./prog and run.
break, next, step, finish and continue control execution; print, display, info locals and info args inspect it.
bt shows the call stack; frame N selects a frame; find the deepest frame in your code.
watch stops when a value changes.
A segmentation fault is the kernel stopping a program that touched unmapped memory, very often through a null or dangling pointer.

Going further: inside gdb, help lists command classes and help break explains one command. Try layout src in the sandbox terminal for a split view of source and commands (press Ctrl-x then a to leave it), and info frame to see the saved return address of the current frame.

Exercises

Exercise 1 · quiz

Pick the gdb command

Your program crashed under gdb with Program received signal SIGSEGV. Which command shows the chain of function calls that led to the crash?

Answer

bt (backtrace)

bt prints the call stack, innermost frame first: frame #0 is where the CPU was when the fault happened, #1 is its caller, and so on up to main. Each line shows the function, its argument values and the source file and line. frame N then selects a frame so that print and info locals work on that function's variables.

Exercise 2 · quiz

next versus step

gdb is stopped on the line total += field_length(i);. What is the difference between typing next and typing step?

There is none; they are aliases
next runs the whole line, including the call, and stops on the following line; step enters field_length and stops on its first line
step runs until the next breakpoint; next runs one machine instruction
next skips the line without executing it

Answer

next runs the whole line, including the call, and stops on the following line; step enters field_length and stops on its first line

next treats a function call as a single step: the function runs completely and gdb stops at the next line of the current function. step descends into the called function (if it has debugging information). finish runs until the current function returns and shows the return value, which is the way back out after stepping in too far.

Exercise 3 · task in a shell

Open the vault

Your workspace contains vault.c and the executable vault, built from it with -g. The vault opens when you run ./vault NUMBER with the right code. The code is computed from a secret phrase that was given on the gcc command line, so it is not in the source.

Use gdb to find the code: set a breakpoint on a line after expected has been computed, run the program, and print expected. Then check that ./vault <code> prints access granted, and save just the number in a file called answer.txt (for example echo 123456 > answer.txt).

Reference solution

gdb -q -batch -ex 'break vault.c:18' -ex run -ex 'printf "%lu\n", expected' ./vault 2>/dev/null | tail -1 > answer.txt
cat answer.txt

Exercise 4 · task in a shell

Find where it crashed

The workspace also has crash.c and its executable crash. Running ./crash ends with Segmentation fault. Run it under gdb (gdb ./crash, then run), get a backtrace with bt, and find the deepest frame that is in our code (crash.c), not inside the C library.

Write the name of that function, and nothing else, into crash.txt. Then, as a bonus, use frame 1 and print i to see which array element was the problem, and read the source to see why.

Reference solution

echo field_length > crash.txt

Exercise 5 · write code

Fix the crash

Here is the program from the previous exercise. Fix field_length so that a missing field (a NULL entry) counts as length 0, instead of being passed to strlen. Leave the data alone. The program must then print exactly:

total: 12

File: main.c

Starting point

#include <stdio.h>
#include <string.h>

static const char *fields[] = {"alice", "bob", NULL, "dave"};

static size_t field_length(int i)
{
    return strlen(fields[i]);
}

static size_t total_length(int n)
{
    size_t total = 0;
    for (int i = 0; i < n; i++)
        total += field_length(i);
    return total;
}

int main(void)
{
    printf("total: %zu\n", total_length(4));
    return 0;
}

Example

expected output

total: 12

Hint 1

strlen walks memory starting at the pointer it is given. With NULL, that is address 0, which is never mapped.

Hint 2

Check if (fields[i] == NULL) return 0; before calling strlen.

Reference solution

#include <stdio.h>
#include <string.h>

static const char *fields[] = {"alice", "bob", NULL, "dave"};

static size_t field_length(int i)
{
    if (fields[i] == NULL)
        return 0;
    return strlen(fields[i]);
}

static size_t total_length(int n)
{
    size_t total = 0;
    for (int i = 0; i < n; i++)
        total += field_length(i);
    return total;
}

int main(void)
{
    printf("total: %zu\n", total_length(4));
    return 0;
}