Strings and Text · about 40 min

UTF-8: Bytes, chars and Why s[0] Does Not Compile

Why this matters

Many languages pretend a string is an array of characters, and that pretense breaks the moment real text arrives: names with accents, Chinese or Arabic script, emoji. Lengths come out wrong, reversals scramble characters, and truncation produces garbage. Rust refuses to pretend. Its strings are UTF-8 bytes, and the API makes you choose explicitly between bytes and characters. That costs a little convenience, and saves you from bugs that otherwise show up only when a user named Zoë signs up.

Unicode and code points

Unicode assigns a number, a code point, to every character of every writing system: A is U+0041 (65), é is U+00E9 (233), 日 is U+65E5, 😀 is U+1F600. There are about 1.1 million possible code points. A Rust char holds one of them (a "Unicode scalar value"), which is why it is 4 bytes.

UTF-8: a variable-length encoding

Storing every character in 4 bytes would quadruple the size of English text. UTF-8 encodes each code point in 1 to 4 bytes:

code pointsbytesexample
U+0000 to U+007F (ASCII)1A = 41
U+0080 to U+07FF2é = C3 A9
U+0800 to U+FFFF3日 = E6 97 A5
U+10000 and up4😀 = F0 9F 98 80

ASCII text is unchanged, which made UTF-8 the dominant encoding on the web and in files. The leading bits of each byte say whether it starts a character and how many bytes follow, so a decoder can always find the next character boundary.

fn main() {
    for s in ["A", "é", "日", "😀"] {
        println!("{s}: {} byte(s) {:02x?}, code point U+{:04X}", s.len(), s.as_bytes(), s.chars().next().unwrap() as u32);
    }
}
A: 1 byte(s) [41], code point U+0041
é: 2 byte(s) [c3, a9], code point U+00E9
日: 3 byte(s) [e6, 97, a5], code point U+65E5
😀: 4 byte(s) [f0, 9f, 98, 80], code point U+1F600

A Rust String or &str is always valid UTF-8. Functions that create strings from raw bytes check this (String::from_utf8 returns an error for invalid bytes), so code working with strings never has to worry about malformed text.

len() counts bytes

fn main() {
    let s = "héllo";
    println!("len {} chars {}", s.len(), s.chars().count());
    println!("{:?}", s.as_bytes());
    for (i, c) in s.char_indices() {
        println!("byte {i}: {c:?}");
    }
}
len 6 chars 5
[104, 195, 169, 108, 108, 111]
byte 0: 'h'
byte 1: 'é'
byte 3: 'l'
byte 4: 'l'
byte 5: 'o'

len() is the number of bytes, stored in the string's handle and therefore instant. chars() decodes the bytes into chars one at a time; counting them walks the whole string. char_indices() gives each character with the byte position where it starts. Note the jump from 1 to 3: é occupies bytes 1 and 2.

 bytes:   68 | c3 a9 | 6c | 6c | 6f
 index:    0 | 1  2  |  3 |  4 |  5
 chars:    h |   é   |  l |  l |  o

Why s[0] does not compile

fn main() {
    let s = String::from("héllo");
    let c = s[0];
}
error[E0277]: the type `str` cannot be indexed by `{integer}`
 --> main.rs:3:15
  |
3 |     let c = s[0];
  |               ^ string indices are ranges of `usize`
  |
  = note: you can use `.chars().nth()` or `.bytes().nth()`
          for more information, see chapter 8 in The Book: <https://doc.rust-lang.org/book/ch08-02-strings.html#indexing-into-strings>
  = help: the trait `SliceIndex<str>` is not implemented for `{integer}`
          but trait `SliceIndex<[_]>` is implemented for `usize`
  = help: for that trait implementation, expected `[_]`, found `str`
  = note: required for `String` to implement `Index<{integer}>`

E0277 "the type str cannot be indexed by {integer}". The notes point at the alternatives. Why refuse? What would s[1] be? The byte 0xC3, which is not a character at all? The char é? Getting the n-th char would require scanning from the start, which hides a linear cost behind syntax that looks constant-time. So Rust offers each meaning as an explicit operation instead:

fn main() {
    let s = "héllo";
    println!("{}", s.as_bytes()[1]);
    println!("{:?}", s.chars().nth(1));
    println!("{}", &s[1..3]);
}
195
Some('é')
é

Slicing in the middle of a character panics

fn main() {
    let s = String::from("héllo");
    let bad = &s[0..2];
    println!("{bad}");
}
thread 'main' panicked at main.rs:3:17:
byte index 2 is not a char boundary; it is inside 'é' (bytes 1..3) of `héllo`
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Byte 2 is inside é, and a &str must always be valid UTF-8, so the slice panics with a precise explanation. Byte-range slicing is safe when the positions come from the string itself: from find, char_indices, or because you know the text is ASCII. s.is_char_boundary(i) tests a position.

Iterating: bytes, chars, and beyond

fn main() {
    let s = "naïve 日本";
    let upper: String = s.chars().map(|c| c.to_ascii_uppercase()).collect();
    println!("{upper}");
    println!("{}", s.to_uppercase());
    let reversed: String = s.chars().rev().collect();
    println!("{reversed}");
    let letters = s.chars().filter(|c| c.is_alphabetic()).count();
    println!("{letters} letters");
}
NAïVE 日本
NAÏVE 日本
本日 evïan
7 letters

(map, filter and collect with |c| ... closures are module 11; read them as "transform each", "keep some", "gather into a String".) Character methods such as is_alphabetic, is_numeric and to_uppercase understand all of Unicode; the ascii variants only touch ASCII letters.

One more level exists. What a reader perceives as one character can be several code points: é can also be written as e followed by a combining accent (U+0301), and many emoji are sequences joined by invisible characters. These "grapheme clusters" are not handled by std; real text editors use the unicode-segmentation crate. For this course, chars() is the right level.

Common mistakes

Using len() as a character count. It counts bytes. For display width or "characters remaining" use chars().count(), and know that even that is an approximation.

Truncating with a byte slice. &s[..10] panics on non-ASCII text whenever byte 10 falls inside a character. Find the cut point with char_indices.

Reversing bytes. Reversing as_bytes() produces invalid UTF-8. Reverse chars().

Assuming one byte per char in loops. for i in 0..s.len() with &s[i..i+1] panics on the first multi-byte character.

Recap

Going further: try String::from_utf8(vec![0xff, 0x41]) and print the result with {:?}, and String::from_utf8_lossy on the same bytes. Read the "Storing UTF-8 Encoded Text with Strings" section of the Rust Book, chapter 8.

Exercises

Exercise 1 · quiz

How long is it?

What do "héllo".len() and "héllo".chars().count() return?

Answer

  • 6 and 5

len() counts bytes of the UTF-8 encoding. ASCII letters take one byte each, but é (U+00E9) takes two bytes in UTF-8 (0xC3 0xA9), so the total is 6. chars() decodes the bytes into Unicode scalar values, of which there are 5. len() is instant (stored in the String); counting chars walks the whole string.

Exercise 2 · write code

Bytes versus characters

Read one line from stdin (trim the trailing newline only, using trim_end_matches('\n')) and print its byte length, its character count, and its characters reversed:

bytes: 6
chars: 5
reversed: olléh

for the input héllo. Reverse by characters (chars().rev()), never by bytes.

File: main.rs

Starting point

use std::io;

fn main() {
    let mut line = String::new();
    io::stdin().read_line(&mut line).unwrap();
    let s = line.trim_end_matches('\n');
}

Example

input

héllo

expected output

bytes: 6
chars: 5
reversed: olléh

Example

input

abc

expected output

bytes: 3
chars: 3
reversed: cba
Hint 1

s.len() counts bytes, s.chars().count() counts characters.

Hint 2

Build the reversed String with a loop over s.chars().rev() and push, or s.chars().rev().collect::<String>().

Reference solution
use std::io;

fn main() {
    let mut line = String::new();
    io::stdin().read_line(&mut line).unwrap();
    let s = line.trim_end_matches('\n');
    println!("bytes: {}", s.len());
    println!("chars: {}", s.chars().count());
    let mut rev = String::new();
    for c in s.chars().rev() {
        rev.push(c);
    }
    println!("reversed: {rev}");
}

Exercise 3 · write code

A prefix that never panics

Write fn prefix(s: &str, n: usize) -> &str that returns the first n characters of s (or all of s if it has fewer). Slicing with &s[..n] would cut by bytes and can panic in the middle of a character. Find the byte position of the n-th character with char_indices() instead.

The given main prints:

hé
日本
abc

File: main.rs

Starting point

fn prefix(s: &str, n: usize) -> &str {
    &s[..n]
}

fn main() {
    println!("{}", prefix("héllo", 2));
    println!("{}", prefix("日本語", 2));
    println!("{}", prefix("abc", 10));
}

Example

expected output

hé
日本
abc
Hint 1

char_indices() yields (byte_position, char) for each character.

Hint 2

The byte position of character number n (counting from 0) is where the prefix ends. If the string runs out first, return all of s.

Reference solution
fn prefix(s: &str, n: usize) -> &str {
    for (count, (i, _)) in s.char_indices().enumerate() {
        if count == n {
            return &s[..i];
        }
    }
    s
}

fn main() {
    println!("{}", prefix("héllo", 2));
    println!("{}", prefix("日本語", 2));
    println!("{}", prefix("abc", 10));
}

Exercise 4 · quiz

Why no s[0]?

Why does Rust refuse to compile let c = s[0]; for a String s? Select all that apply.

Answer

  • A byte position does not always correspond to a whole character in UTF-8

  • Finding the n-th character requires scanning from the start, so it could not be the constant-time operation indexing promises

  • It would be ambiguous whether the result should be a byte, a char, or a user-perceived character

UTF-8 characters take 1 to 4 bytes, so byte 0 may be only part of a character, and the n-th character is at an unpredictable byte offset. Rather than pick one meaning silently, Rust makes you say what you want: s.as_bytes()[0] for a byte, s.chars().nth(0) for a char, or a slice &s[0..1] by byte range (which panics if it splits a character).