Unicode, code points and bytes

Why this matters

Text is not bytes. Your program works with characters such as é, π and €, but files, network sockets and pipes carry only bytes. Every time text crosses that boundary it must be encoded (characters to bytes) or decoded (bytes to characters), and getting it wrong produces crashes like UnicodeDecodeError or garbled text like café. Python 3 makes the distinction explicit with two separate types, str and bytes. Understanding them once saves you hours of confused debugging whenever you read a file, parse network data or handle names from around the world.

Characters are numbers: code points

Computers store numbers, so every character is assigned a number. Unicode is the universal catalog: it gives each character a unique number called a code point, written U+ followed by hex digits. Some examples:

Unicode has room for over a million code points and covers nearly every writing system, plus symbols and emoji. The first 128 code points are identical to the old ASCII standard, which is why plain English text looks the same in both.

A Python str is a sequence of code points. len() counts code points. Two built-in functions convert between characters and their numbers:

print(ord("A"))
print(ord("é"))
print(chr(960))
print(chr(0x20AC))
print(f"{ord('€'):04X}")

65
233
π
€
20AC

ord takes a one-character string and returns its code point; chr does the reverse. 0x20AC is how you write a number in hexadecimal in Python source.

You can also write any character in a string literal by code point, using escapes: "é" is é, and "\N{GREEK SMALL LETTER PI}" is π.

Because characters are numbers, comparing strings compares code points, one position at a time. That is why "a" <= ch <= "z" tests for a lowercase ASCII letter, and also why "Zebra" < "apple" is True: uppercase Z (90) comes before lowercase a (97).

Bytes: what actually gets stored

A byte is 8 bits, a number from 0 to 255. Files and network connections are sequences of bytes. Python represents raw binary data with the bytes type, written with a b prefix:

data = b"hello"
print(data)
print(data[0])
print(len(data))
print(type(data))

b'hello'
104
5
<class 'bytes'>

Note that indexing a bytes object gives an integer (104 is h), not a one-character string. Bytes are immutable sequences of small integers. A bytes literal may contain only ASCII characters; other byte values are written with \x escapes, as in b"caf\xc3\xa9".

Encodings: from code points to bytes

An encoding is a rule for turning code points into bytes and back. Code points can be far larger than 255, so a scheme is needed. The dominant encoding today, used by the web, Linux and Python source files by default, is UTF-8. It is a variable-width encoding:

ASCII text encoded as UTF-8 is byte-for-byte identical to ASCII, which made the transition painless. Multi-byte sequences use bytes 128-255 only, so a UTF-8 byte below 128 is always an ASCII character.

encode and decode

str.encode(encoding) produces bytes; bytes.decode(encoding) produces a str:

          encode("utf-8")
   str  ------------------>  bytes
 "café"                     b'caf\xc3\xa9'
        <------------------
          decode("utf-8")

word = "café"
data = word.encode("utf-8")
print(data)
print(len(word), len(data))
print(data.hex(" "))
print(data.decode("utf-8"))

b'caf\xc3\xa9'
4 5
63 61 66 c3 a9
café

é became the two bytes c3 a9. The .hex(" ") method shows each byte as two hex digits, separated by the given string, a convenient way to inspect binary data.

The key rule: decode with the same encoding that was used to encode. Bytes do not carry a label saying how they were encoded; you must know (from a file format specification, an HTTP header, or a convention like "this project uses UTF-8").

When it goes wrong

Decoding bytes with the wrong encoding either fails or silently produces garbage.

data = "café".encode("utf-8")
print(data.decode("latin-1"))
print(data.decode("ascii"))

café
Traceback (most recent call last):
  File "main.py", line 3, in <module>
    print(data.decode("ascii"))
          ~~~~~~~~~~~^^^^^^^^^
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)

Latin-1 maps every byte to a character, so it never fails, but it read the two bytes of é as two separate characters, Ã and ©. This garbling is called mojibake, and you have surely seen it on web pages. ASCII decoding refuses outright, because byte 0xc3 is outside its range.

Encoding can fail too, when the target encoding has no way to represent a character:

"€5".encode("ascii")

UnicodeEncodeError: 'ascii' codec can't encode character '€' in position 0: ordinal not in range(128)

Error handlers

Both encode and decode take an errors argument that says what to do with problem characters. The default is "strict" (raise an exception). Others:

bad = b"caf\xe9"            # Latin-1 bytes, invalid as UTF-8
print(bad.decode("utf-8", errors="replace"))
print(bad.decode("utf-8", errors="ignore"))
print("€5".encode("ascii", errors="replace"))
print("€5".encode("ascii", errors="backslashreplace"))

caf�
caf
b'?5'
b'\\u20ac5'

replace substitutes the official replacement character U+FFFD (shown as �) when decoding, or ? when encoding; ignore drops the bad data silently. Use these deliberately, for example when salvaging a corrupted log file. Silently ignoring errors in important data is a way to lose information without noticing.

Where encoding happens in practice

You often do not call encode and decode yourself, but they still happen:

• print encodes your str to bytes for stdout, and input decodes the bytes arriving on stdin. Python picks the encoding from the environment; on modern Linux that is UTF-8.
• open("file.txt") in text mode decodes when reading and encodes when writing. You will learn in the files lesson to always pass encoding="utf-8" explicitly, so your program behaves the same on every machine.
• Network libraries hand you bytes; you decode them.

The healthy pattern is the "Unicode sandwich": decode bytes as early as possible when they enter your program, work only with str inside, and encode as late as possible on the way out.

A note on what "character" means

Code points are not always what a human would call one character. é can be written as one code point (U+00E9) or as e followed by a combining accent (U+0065 U+0301); both look identical, but len gives 1 and 2, and == says they differ. The unicodedata.normalize("NFC", s) function converts such text to a consistent form. Many emoji are also built from several code points. For most programs this does not matter, but it explains surprising lengths when it does.

Common mistakes

• Mixing str and bytes. "abc" + b"def" raises TypeError: can only concatenate str (not "bytes") to str. Decode or encode one side first.
• Indexing bytes and expecting a character. b"hi"[0] is 104. Use b"hi"[0:1] for a one-byte bytes object, or decode first.
• Assuming one character equals one byte. That holds only for ASCII. Use len(s.encode("utf-8")) when you need a byte count, for example for a database column limit.
• Fixing mojibake by guessing. If you see é, UTF-8 data was decoded as Latin-1 (or Windows-1252). Fix the decoding step, rather than patching the output.

Recap

• A str is a sequence of Unicode code points; ord and chr convert between characters and integers.
• A bytes object is an immutable sequence of integers 0-255; this is what files and networks carry.
• s.encode("utf-8") turns text into bytes; b.decode("utf-8") turns bytes into text. Always decode with the encoding that was used to encode.
• UTF-8 uses 1 to 4 bytes per code point, and plain ASCII is unchanged.
• Wrong encodings cause UnicodeDecodeError, UnicodeEncodeError or mojibake; errors="replace" and friends control what happens to bad data.

Going further: experiment with import unicodedata; print(unicodedata.name("€")), and read the Python "Unicode HOWTO" (docs.python.org, Python HOWTOs), which covers normalization and more.