Endianness and serialization

Why this matters

Inside one program, an integer is just an integer. The moment it leaves the program, written to a file, sent over a network, or shared with hardware, it becomes a sequence of bytes, and the two sides must agree on the order of those bytes. Get it wrong and a length of 256 arrives as 65536. Every file format (PNG, ZIP, WAV, ELF) and every network protocol (IP, TCP, DNS) specifies a byte order, and C programmers read and write these formats with the shift and mask techniques from the previous lessons.

Multi-byte values in byte-addressed memory

Memory is addressed byte by byte, but a uint32_t needs four bytes. Which byte goes at the lowest address? There are two conventions:

  value 0x12345678

                    address:  a     a+1   a+2   a+3
  big-endian                  12    34    56    78     most significant byte first
  little-endian               78    56    34    12     least significant byte first

Big-endian matches how we write numbers. Little-endian puts the "little end" first. x86 and x86-64 are little-endian, and so is ARM as used by almost every phone and Apple Silicon Mac. Some older and embedded architectures are big-endian, and network protocols use big-endian, which is why it is also called network byte order.

You can see your machine's order by looking at an integer's bytes through an unsigned char pointer (inspecting any object's bytes this way is always allowed in C):

#include <stdio.h>
#include <stdint.h>

int main(void)
{
    uint32_t x = 0x12345678;
    unsigned char *p = (unsigned char *)&x;
    for (size_t i = 0; i < sizeof x; i++)
        printf("%02X ", p[i]);
    printf("\n");
    return 0;
}

78 56 34 12

The sandbox is x86-64, so the least significant byte, 78, comes first.

The bug it causes

Suppose a file format says "a 4-byte big-endian length", and a file contains the bytes 00 00 01 00, meaning 256. The tempting code copies the bytes straight into an integer:

unsigned char raw[4] = {0x00, 0x00, 0x01, 0x00};
uint32_t naive;
memcpy(&naive, raw, 4);
printf("%u\n", naive);       /* 65536 on x86: the bytes were taken as little-endian */

The program reads the bytes in the machine's order, not the format's. It may even be correct on the developer's machine and wrong on the user's. You can look at raw bytes in a file with od:

$ od -A d -t x1 f.bin
0000000 00 00 01 00
0000004

The portable solution: shifts

Shifts and masks operate on values, not on memory layout, so they mean the same thing on every CPU. To write a value in big-endian order, extract each byte from the top down; to read it, shift each byte back into place:

void put_u32_be(unsigned char *buf, uint32_t v)
{
    buf[0] = (unsigned char)(v >> 24);
    buf[1] = (unsigned char)(v >> 16);
    buf[2] = (unsigned char)(v >> 8);
    buf[3] = (unsigned char)v;
}

uint32_t get_u32_be(const unsigned char *buf)
{
    return (uint32_t)buf[0] << 24 | (uint32_t)buf[1] << 16 |
           (uint32_t)buf[2] << 8  | (uint32_t)buf[3];
}

With these, get_u32_be(raw) returns 256 on any machine, and put_u32_be produces 12 34 56 78 for 0x12345678. No #ifdef for byte order is needed; the compiler turns the shifts into a single load and byte-swap instruction where that is faster.

Note the casts to uint32_t before shifting. Without them, buf[0] << 24 promotes the unsigned char to int, and a byte of 0x80 or more shifted by 24 lands in the sign bit: undefined behavior. The cast keeps the arithmetic unsigned.

The standard helpers

POSIX provides htonl, htons, ntohl and ntohs in <arpa/inet.h> ("host to network long", and so on). They convert a value between the host's order and big-endian, and you will see them in all networking code:

uint32_t n = htonl(0x12345678);   /* on x86 the bytes are swapped; on a big-endian CPU, no-op */

They return integers whose memory representation is big-endian, which you then memcpy into a packet. The shift functions are usually clearer because they go straight to bytes.

Serialization

Serialization means turning in-memory data into a sequence of bytes that can be stored or transmitted and turned back later (deserialization). For C structs, the robust approach is:

1. Define the format precisely: each field's offset, size, signedness and byte order.
2. Write each field with an explicit encoder (put_u32_le, put_u16_be, ...) into a byte buffer.
3. Write the buffer with fwrite or write.
4. Read the bytes and decode each field with the matching decoder, checking lengths.

Text formats (CSV, JSON) avoid the byte order question entirely at the cost of size and parsing speed, which is why many protocols are text based.

What you should not do is fwrite(&some_struct, sizeof some_struct, 1, f) for data that must outlive the build. As you saw in the structs module, structs contain padding chosen by the compiler, and types like int and long differ in size between platforms. A raw dump captures one compiler's layout, one CPU's byte order, and whatever garbage happens to be in the padding bytes.

Signed values and floating point add one more step: convert a signed value to the unsigned type of the same width before encoding ((uint32_t)x), and back afterwards. For a double, copy its bytes into a uint64_t with memcpy and encode that; nearly every modern system uses the IEEE 754 format, so only the byte order then needs agreeing.

Common mistakes

• Copying bytes straight into an integer (memcpy, or casting a char * buffer to uint32_t *) when the data has a defined byte order. The cast version is also undefined if the buffer is not suitably aligned.
• Shifting unsigned char values without a cast, overflowing int for bytes above 0x7F.
• Using char for bytes. Plain char is signed on x86, so a byte 0xFF becomes -1 and sign-extends when converted. Use unsigned char or uint8_t for raw bytes.
• Assuming everyone is little-endian. Today that is nearly true for CPUs, but never for network protocols and many file formats.

Recap

• Endianness is the order of a multi-byte value's bytes in memory: little-endian (x86, ARM) puts the least significant byte first; big-endian ("network order") the most significant.
• Look at an object's bytes with an unsigned char *.
• Encode and decode with shifts and masks on uint32_t; that is independent of the host's order.
• htonl/ntohl convert between host and network order.
• Never dump raw structs as a persistent format: padding, sizes and byte order are not portable.

Going further: run od -A x -t x1z /bin/ls | head -2 in the sandbox: the first bytes are the ELF magic 7f 45 4c 46 (\x7fELF), followed by a byte that says the file is 64-bit and one that says it is little-endian. man 3 endian describes htobe32, le32toh and friends, glibc's explicit-width versions of htonl.