Undefined behavior

Why this matters

You have met the phrase "undefined behavior" many times in this track. It is the single most important idea separating C from safer languages, and the source of most C security holes. It does not mean "the program crashes" or "you get a random number". It means the C standard places no requirements at all on what the program does, and modern compilers take that literally: they assume undefined behavior never happens and optimize on that basis. The result can be code that works at -O0, breaks at -O2, deletes your safety checks, or loops forever. Understanding the rules is how you write C that keeps working when the compiler, the flags or the input change.

Three kinds of "not fully specified"

The C standard sorts loose ends into three categories:

• Implementation-defined behavior: the compiler chooses, and must document its choice. Examples: the size of int, whether plain char is signed, what >> does to a negative number (gcc: arithmetic shift). Portable code avoids relying on it; correct code can use it.
• Unspecified behavior: one of several outcomes, and the compiler need not tell you which, or even be consistent. Example: the order in which function arguments are evaluated in f(g(), h()). A correct program must work whichever order happens.
• Undefined behavior (UB): anything. The program has no meaning from that point on. In fact, the standard allows UB to affect what happens before it, because the compiler may rearrange code on the assumption that it never occurs.

The usual suspects

Here are a few of them in one program, compiled with warnings:

#include <stdio.h>

int sign(int x)
{
    if (x > 0)
        return 1;
    if (x < 0)
        return -1;
}

int main(void)
{
    int i = 1;
    i = i++ + 1;
    char *s = "hello";
    printf("%d %d\n", i, sign(5));
    s[0] = 'J';
    printf("%s\n", s);
    return 0;
}

$ gcc -std=c17 -Wall -Wextra -O0 -o prog seq.c
seq.c: In function 'main':
seq.c:14:7: warning: operation on 'i' may be undefined [-Wsequence-point]
   14 |     i = i++ + 1;
      |     ~~^~~~~~~~~
seq.c: In function 'sign':
seq.c:9:1: warning: control reaches end of non-void function [-Wreturn-type]
    9 | }
      | ^
$ ./prog
Segmentation fault (core dumped)

The compiler warned about two of the three bugs. The third, writing to the literal "hello", crashed because the linker put string literals in a read-only memory page. Notice also that the first printf ran but its output never appeared: when output goes to a pipe it is buffered, and the crash discarded the buffer.

How the optimizer uses UB

Compilers are not malicious; they are logical. Their job is to produce the fastest code that behaves as the standard requires for every valid execution. Executions that contain UB are not valid, so the compiler need not care what happens in them. Two consequences you can see in the sandbox:

Overflow checks vanish.

int will_overflow(int x)
{
    return x + 1 < x;
}

gcc turns this into return 0; even at -O0: if signed overflow cannot happen, x + 1 is always greater than x. The check is gone. Compiled with -fsanitize=undefined, the same call reports runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int'.

Loops stop terminating.

int count = 0;
for (int i = 1; i > 0; i *= 2)
    count++;
printf("%d\n", count);

$ gcc -std=c17 -O0 -o prog loop.c && ./prog
31
$ gcc -std=c17 -O2 -o prog loop.c && ./prog
^C          (runs forever)

At -O0 the doubling happens to wrap to a negative number and the loop ends after 31 steps. At -O2 gcc reasons that i starts positive and is only ever doubled, and since overflow "cannot happen" it stays positive, so i > 0 is always true. The loop becomes infinite. Both behaviors are permitted; the program was never meaningful.

Warning: "it works on my machine" means nothing for code with UB. A new compiler version, a different flag, or an unrelated change elsewhere in the function can change the outcome.

Writing UB-free code

You cannot test UB away; you avoid it by construction and catch slips with tools:

1. Check before you operate. Before a + b, test a > INT_MAX - b (for positive b). Before a[i], test i < n. Before x / y, test y != 0.
2. Use unsigned types for bit manipulation and wrapping counters. Unsigned overflow is defined (it wraps modulo 2^N), and shifts on unsigned values have no sign-bit trap. For shifts, write 1u << n, not 1 << n.
3. Initialize every variable at its declaration.
4. One side effect per variable per expression. i++ on its own line is always clear.
5. Turn on warnings (-Wall -Wextra) and treat them as errors.
6. Run tests under -fsanitize=address,undefined, which turns silent UB into loud reports.
7. Copy bytes with memcpy when you need to reinterpret an object as another type, instead of casting pointers (the "strict aliasing" rule lets the compiler assume an int * and a float * never point at the same memory).

Common mistakes

• Checking for overflow after it happened, as in if (a + b < a). The compiler may delete the check. Check the operands first.
• Assuming wrapping for int. It is true of the hardware, not of the C language.
• Thinking -O0 is safe. It is merely less surprising. The sandbox showed an overflow check removed even at -O0.
• Writing i = i++; or a[i] = i++;. These modify and use i without an ordering between them. Different compilers really do produce different results.
• Believing UB only matters for crashes. Many security exploits start with an out-of-bounds write that never crashed in testing.

Recap

• Implementation-defined: documented choice. Unspecified: one of several outcomes. Undefined: no requirements whatsoever.
• Common UB: signed overflow, out-of-bounds access, null or dangling dereference, uninitialized reads, bad shifts, modifying literals, unsequenced modifications, division by zero.
• Optimizers assume UB never happens: checks vanish and loops change meaning.
• Avoid UB by checking operands before operating, using unsigned for bits, initializing, and testing with sanitizers.

Going further: read John Regehr's "A Guide to Undefined Behavior in C and C++" (three blog posts) and the GCC manual's description of -fwrapv, a flag that makes signed overflow wrap, used by the Linux kernel. Try gcc -O2 -S on will_overflow and find the xorl %eax, %eax that returns 0.