Why this matters
Every programming language, calculator, spreadsheet formula, query language and configuration format is processed the same way: text is split into tokens, tokens are parsed into a tree that captures the structure, and the tree is evaluated. Building a small interpreter end to end uses nearly everything in this course: enums with data, Box for recursive types, pattern matching, Result and ? for errors, iterators, a HashMap for variables, and careful ownership of the tree. It is also one of the most satisfying programs to write.
The finished program is a calculator with variables:
> 1 + 2 * 3
7
> let x = 10
x = 10
> let y = x * x - 1
y = 99
> y / (x - 10)
error: division by zero
> -x % 3
-1
> (4 + 5
error: expected ')'
The pipeline
"let y = x * x - 1"
|
| tokenize: characters -> tokens
v
[Let, Ident("y"), Assign, Ident("x"), Op('*'), Ident("x"), Op('-'), Num(1)]
|
| parse: tokens -> tree (respecting precedence: * before -)
v
Bin(-)
/ \
Bin(*) Num(1)
/ \
Var(x) Var(x)
|
| eval: tree + variables -> value
v
99 (and store y = 99)
Each stage has one job and its own error messages, and each can be tested on its own.
Stage 1: tokens
A token is the smallest meaningful unit. An enum describes them exactly:
#[derive(Debug, Clone, PartialEq)]
enum Token {
Num(i64),
Ident(String),
Op(char),
LParen,
RParen,
Let,
Assign,
}
The tokenizer walks the characters with an index. Digits accumulate into a number, letters into a name (which may turn out to be the keyword let), and single characters map to operators; anything else is an error. This is a small state machine like the previous lesson's, written with inner while loops instead of an explicit state enum:
while i < chars.len() {
let c = chars[i];
if c.is_whitespace() { i += 1; }
else if c.is_ascii_digit() { /* consume all digits, parse into i64 */ }
else if c.is_ascii_alphabetic() || c == '_' { /* consume a name; "let" becomes Token::Let */ }
else { /* + - * / % ( ) = become tokens; anything else: Err("unexpected character 'c'") */ }
}
The first exercise asks you to write it. Collecting the characters into a Vec<char> first makes indexing by character easy (strings cannot be indexed, module 6).
Stage 2: the tree
The structure of an expression is a tree, and a recursive enum with Box (module 13) represents it:
#[derive(Debug)]
enum Expr {
Num(i64),
Var(String),
Neg(Box<Expr>),
Bin(Box<Expr>, char, Box<Expr>),
}
Bin(left, op, right) is a binary operation; Neg is unary minus. The tree has no parentheses: they only influence its shape during parsing.
Stage 3: parsing with recursive descent
Precedence ("* binds tighter than +") is expressed as a grammar, one rule per precedence level:
expr := term (('+' | '-') term)*
term := factor (('*' | '/' | '%') factor)*
factor := '-' factor | NUMBER | IDENT | '(' expr ')'
Read it as: an expression is one or more terms joined by +/-; a term is one or more factors joined by *, /, %; a factor is a number, a variable, a negated factor, or a whole expression in parentheses. Recursive descent turns each rule into a method that calls the methods for the rules it mentions:
fn expr(&mut self) -> Result<Expr, String> {
let mut left = self.term()?;
while let Some(Token::Op(op @ ('+' | '-'))) = self.peek() {
let op = *op;
self.pos += 1;
let right = self.term()?;
left = Expr::Bin(Box::new(left), op, Box::new(right));
}
Ok(left)
}
peek looks at the next token without consuming it; next consumes it.- The
while let with the pattern Token::Op(op @ ('+' | '-')) loops as long as the next token is + or -, binding the operator (module 7's @ bindings). - Building
left = Bin(left, op, right) in a loop makes operators left-associative: 10 - 3 - 2 becomes (10 - 3) - 2. term is identical with *, /, % and calls factor. Because expr only ever sees whole terms, * automatically binds tighter than +.factor is a match self.next(): Op('-') recurses into factor (so --3 works), Num and Ident become leaves, and LParen parses a full expr and then requires an RParen. Anything else is an error.
Ownership is pleasantly simple: each Box::new(left) moves the subtree into its parent, so the final tree owns all its nodes, and dropping the root frees everything.
After parsing, check that all tokens were used: in 4 5, expr happily parses 4 and stops, so leftover tokens mean an error.
Stage 4: evaluation
Evaluation is a recursive match over the tree, which mirrors its definition exactly:
fn eval(e: &Expr, env: &HashMap<String, i64>) -> Result<i64, String> {
match e {
Expr::Num(n) => Ok(*n),
Expr::Var(name) => env.get(name).copied().ok_or(format!("unknown variable '{name}'")),
Expr::Neg(inner) => eval(inner, env)?.checked_neg().ok_or(String::from("overflow")),
Expr::Bin(l, op, r) => { /* evaluate both sides with ?, then apply op with checked arithmetic */ }
}
}
? propagates errors from deep inside the tree straight to the caller.- Checked arithmetic (
checked_add, checked_div, ...) returns None on overflow instead of panicking, which ok_or turns into an error message. Division by zero is checked explicitly first for a clearer message. - The environment is borrowed immutably; only the top level, handling
let, modifies it.
Statements: let
A line is either let NAME = EXPRESSION or an expression. Slice patterns (module 7) recognize the let form directly on the token list:
match tokens.as_slice() {
[Token::Let, Token::Ident(name), Token::Assign, rest @ ..] => /* parse rest, store under name */,
[Token::Let, ..] => Err("expected: let NAME = EXPRESSION"),
_ => /* parse all tokens as an expression and print its value */,
}
rest @ .. binds the remaining tokens as a slice. main then just loops over input lines, calls this for each, and prints the result or error: MESSAGE, keeping the variables in a HashMap across lines.
Errors are part of the design
Every stage reports problems as Err(String) with a specific message, and the program continues with the next line. Count the failure modes the exercise checks: unknown characters, missing ), missing operands, leftover input, malformed let, unknown variables, division by zero, overflow. None of them panics. That is the difference between a toy and a tool.
Extensions to try
- Add
^ for exponentiation with right associativity and higher precedence than * (a new grammar level whose method recurses instead of looping). - Use
f64 values and a sqrt(x) function call syntax. - Report error positions: store each token's column in the tokenizer and carry it into error messages.
- Print the tree with
{:#?} (derive Debug is already there) to see how precedence shaped it.
Recap
- Interpreters are pipelines: tokenize, parse, evaluate, each with its own errors.
- Tokens and trees are enums; recursive trees use
Box. - A grammar with one rule per precedence level maps directly onto recursive-descent methods; looping builds left-associative trees.
- Evaluation is a recursive
match; ? and checked arithmetic make every failure a clean error. - Slice patterns recognize statement forms like
let NAME = ....
Going further: read Robert Nystrom's free book "Crafting Interpreters" (craftinginterpreters.com); its first half builds a tree-walking interpreter with exactly this architecture.
Exercise 1 · write code
Stage 1: the tokenizer
Write fn tokenize(src: &str) -> Result<Vec<Token>, String> for the Token enum in the starter:
- whitespace is skipped
- a run of ASCII digits is a
Num(i64) (if it does not fit: Err("number too large: DIGITS")) - a run starting with a letter or
_, continuing with letters, digits or _, is an Ident(String), except that the word let is Token::Let + - * / % are Op(c), ( is LParen, ) is RParen, = is Assign- anything else:
Err("unexpected character 'C'")
The given main prints the tokens of each stdin line with {:?}, or error: MESSAGE:
[Let, Ident("rate"), Assign, Num(12), Op('*'), LParen, Ident("x_1"), Op('-'), Num(3), RParen]
[Num(7), Op('%'), Num(2)]
error: unexpected character '$'
for the lines let rate = 12 * (x_1 - 3), 7%2, a $ b.
File: main.rs
Starting point
use std::io;
#[derive(Debug, Clone, PartialEq)]
enum Token {
Num(i64),
Ident(String),
Op(char),
LParen,
RParen,
Let,
Assign,
}
fn tokenize(src: &str) -> Result<Vec<Token>, String> {
Ok(Vec::new())
}
fn main() {
for line in io::stdin().lines() {
let line = line.unwrap();
match tokenize(&line) {
Ok(tokens) => println!("{:?}", tokens),
Err(e) => println!("error: {e}"),
}
}
}
Example
input
let rate = 12 * (x_1 - 3)
7%2
a $ b
expected output
[Let, Ident("rate"), Assign, Num(12), Op('*'), LParen, Ident("x_1"), Op('-'), Num(3), RParen]
[Num(7), Op('%'), Num(2)]
error: unexpected character '$'
Example
input
letter let _x9
99999999999999999999
expected output
[Ident("letter"), Let, Ident("_x9")]
error: number too large: 99999999999999999999
Hint 1
Collect the characters first: let chars: Vec<char> = src.chars().collect(); and walk them with an index i.
Hint 2
For a number: remember start = i, advance while digits, then chars[start..i].iter().collect::<String>() and parse it.
Hint 3
For single-character tokens, match c { '+' | '-' | '*' | '/' | '%' => Token::Op(c), '(' => Token::LParen, ... } and advance by one.
Reference solution
use std::io;
#[derive(Debug, Clone, PartialEq)]
enum Token {
Num(i64),
Ident(String),
Op(char),
LParen,
RParen,
Let,
Assign,
}
fn tokenize(src: &str) -> Result<Vec<Token>, String> {
let chars: Vec<char> = src.chars().collect();
let mut tokens = Vec::new();
let mut i = 0;
while i < chars.len() {
let c = chars[i];
if c.is_whitespace() {
i += 1;
} else if c.is_ascii_digit() {
let start = i;
while i < chars.len() && chars[i].is_ascii_digit() {
i += 1;
}
let text: String = chars[start..i].iter().collect();
let n = text.parse().map_err(|_| format!("number too large: {text}"))?;
tokens.push(Token::Num(n));
} else if c.is_ascii_alphabetic() || c == '_' {
let start = i;
while i < chars.len() && (chars[i].is_ascii_alphanumeric() || chars[i] == '_') {
i += 1;
}
let word: String = chars[start..i].iter().collect();
tokens.push(if word == "let" { Token::Let } else { Token::Ident(word) });
} else {
tokens.push(match c {
'+' | '-' | '*' | '/' | '%' => Token::Op(c),
'(' => Token::LParen,
')' => Token::RParen,
'=' => Token::Assign,
_ => return Err(format!("unexpected character '{c}'")),
});
i += 1;
}
}
Ok(tokens)
}
fn main() {
for line in io::stdin().lines() {
let line = line.unwrap();
match tokenize(&line) {
Ok(tokens) => println!("{:?}", tokens),
Err(e) => println!("error: {e}"),
}
}
}
Exercise 2 · write code
Stages 2 to 4: parse and evaluate
The starter contains the tokenizer, the Expr tree type, run_line (handles let) and main. Implement the three parser methods expr, term and factor (recursive descent, following the grammar in the comments) and eval. Error messages, exactly:
- in
factor, a missing ): expected ')' - in
factor, any other token or the end of input where an operand is expected: expected a number, variable or '(' - an undefined variable:
unknown variable 'NAME' - division or remainder by zero:
division by zero - any overflow (including negating the minimum value):
overflow
Operators are left-associative (10 - 3 - 2 is 5); unary minus binds tighter than * (-2 * 3 is -6). / and % behave like Rust's integer operators. For the input
1 + 2 * 3
let x = 10
let y = x * x - 1
y / (x - 10)
-x % 3
(4 + 5
the program prints
7
x = 10
y = 99
error: division by zero
-1
error: expected ')'
File: main.rs
Starting point
use std::collections::HashMap;
use std::io;
#[derive(Debug, Clone, PartialEq)]
enum Token {
Num(i64),
Ident(String),
Op(char),
LParen,
RParen,
Let,
Assign,
}
fn tokenize(src: &str) -> Result<Vec<Token>, String> {
let chars: Vec<char> = src.chars().collect();
let mut tokens = Vec::new();
let mut i = 0;
while i < chars.len() {
let c = chars[i];
if c.is_whitespace() {
i += 1;
} else if c.is_ascii_digit() {
let start = i;
while i < chars.len() && chars[i].is_ascii_digit() {
i += 1;
}
let text: String = chars[start..i].iter().collect();
let n = text.parse().map_err(|_| format!("number too large: {text}"))?;
tokens.push(Token::Num(n));
} else if c.is_ascii_alphabetic() || c == '_' {
let start = i;
while i < chars.len() && (chars[i].is_ascii_alphanumeric() || chars[i] == '_') {
i += 1;
}
let word: String = chars[start..i].iter().collect();
tokens.push(if word == "let" { Token::Let } else { Token::Ident(word) });
} else {
tokens.push(match c {
'+' | '-' | '*' | '/' | '%' => Token::Op(c),
'(' => Token::LParen,
')' => Token::RParen,
'=' => Token::Assign,
_ => return Err(format!("unexpected character '{c}'")),
});
i += 1;
}
}
Ok(tokens)
}
#[derive(Debug)]
enum Expr {
Num(i64),
Var(String),
Neg(Box<Expr>),
Bin(Box<Expr>, char, Box<Expr>),
}
struct Parser {
tokens: Vec<Token>,
pos: usize,
}
impl Parser {
fn peek(&self) -> Option<&Token> {
self.tokens.get(self.pos)
}
fn next(&mut self) -> Option<Token> {
let t = self.tokens.get(self.pos).cloned();
self.pos += 1;
t
}
// expr := term (('+' | '-') term)*
fn expr(&mut self) -> Result<Expr, String> {
todo!("parse terms separated by + and -")
}
// term := factor (('*' | '/' | '%') factor)*
fn term(&mut self) -> Result<Expr, String> {
todo!("parse factors separated by *, / and %")
}
// factor := '-' factor | NUMBER | IDENT | '(' expr ')'
fn factor(&mut self) -> Result<Expr, String> {
todo!("parse a negation, number, variable or parenthesized expression")
}
}
fn eval(e: &Expr, env: &HashMap<String, i64>) -> Result<i64, String> {
todo!("evaluate the tree")
}
fn run_line(line: &str, env: &mut HashMap<String, i64>) -> Result<String, String> {
let tokens = tokenize(line)?;
let (name, expr_tokens) = match tokens.as_slice() {
[Token::Let, Token::Ident(name), Token::Assign, rest @ ..] => (Some(name.clone()), rest.to_vec()),
[Token::Let, ..] => return Err(String::from("expected: let NAME = EXPRESSION")),
_ => (None, tokens),
};
let mut parser = Parser { tokens: expr_tokens, pos: 0 };
let expr = parser.expr()?;
if parser.pos < parser.tokens.len() {
return Err(String::from("unexpected input after expression"));
}
let value = eval(&expr, env)?;
match name {
Some(n) => {
env.insert(n.clone(), value);
Ok(format!("{n} = {value}"))
}
None => Ok(value.to_string()),
}
}
fn main() {
let mut env = HashMap::new();
for line in io::stdin().lines() {
let line = line.unwrap();
if line.trim().is_empty() {
continue;
}
match run_line(&line, &mut env) {
Ok(out) => println!("{out}"),
Err(e) => println!("error: {e}"),
}
}
}
Example
input
1 + 2 * 3
let x = 10
let y = x * x - 1
y / (x - 10)
-x % 3
(4 + 5
expected output
7
x = 10
y = 99
error: division by zero
-1
error: expected ')'
Example
input
10 - 3 - 2
2 * 3 + 4 * 5
-2 * 3
--4
100 / 10 / 5
(1 + 2) * (3 + 4)
expected output
5
26
-6
4
2
21
Hint 1
expr: let mut left = self.term()?; while let Some(Token::Op(op @ ('+' | '-'))) = self.peek() { let op = *op; self.pos += 1; let right = self.term()?; left = Expr::Bin(Box::new(left), op, Box::new(right)); } Ok(left). term is the same with *, /, % and factor.
Hint 2
factor: match self.next() { Some(Token::Op('-')) => Ok(Expr::Neg(Box::new(self.factor()?))), Some(Token::Num(n)) => ..., Some(Token::LParen) => ..., _ => Err(...) }.
Hint 3
eval: recurse with ?; use checked_add, checked_sub, checked_mul, checked_div, checked_rem, checked_neg and turn None into overflow; check for a zero divisor first.
Reference solution
use std::collections::HashMap;
use std::io;
#[derive(Debug, Clone, PartialEq)]
enum Token {
Num(i64),
Ident(String),
Op(char),
LParen,
RParen,
Let,
Assign,
}
fn tokenize(src: &str) -> Result<Vec<Token>, String> {
let chars: Vec<char> = src.chars().collect();
let mut tokens = Vec::new();
let mut i = 0;
while i < chars.len() {
let c = chars[i];
if c.is_whitespace() {
i += 1;
} else if c.is_ascii_digit() {
let start = i;
while i < chars.len() && chars[i].is_ascii_digit() {
i += 1;
}
let text: String = chars[start..i].iter().collect();
let n = text.parse().map_err(|_| format!("number too large: {text}"))?;
tokens.push(Token::Num(n));
} else if c.is_ascii_alphabetic() || c == '_' {
let start = i;
while i < chars.len() && (chars[i].is_ascii_alphanumeric() || chars[i] == '_') {
i += 1;
}
let word: String = chars[start..i].iter().collect();
tokens.push(if word == "let" { Token::Let } else { Token::Ident(word) });
} else {
tokens.push(match c {
'+' | '-' | '*' | '/' | '%' => Token::Op(c),
'(' => Token::LParen,
')' => Token::RParen,
'=' => Token::Assign,
_ => return Err(format!("unexpected character '{c}'")),
});
i += 1;
}
}
Ok(tokens)
}
#[derive(Debug)]
enum Expr {
Num(i64),
Var(String),
Neg(Box<Expr>),
Bin(Box<Expr>, char, Box<Expr>),
}
struct Parser {
tokens: Vec<Token>,
pos: usize,
}
impl Parser {
fn peek(&self) -> Option<&Token> {
self.tokens.get(self.pos)
}
fn next(&mut self) -> Option<Token> {
let t = self.tokens.get(self.pos).cloned();
self.pos += 1;
t
}
// expr := term (('+' | '-') term)*
fn expr(&mut self) -> Result<Expr, String> {
let mut left = self.term()?;
while let Some(Token::Op(op @ ('+' | '-'))) = self.peek() {
let op = *op;
self.pos += 1;
let right = self.term()?;
left = Expr::Bin(Box::new(left), op, Box::new(right));
}
Ok(left)
}
// term := factor (('*' | '/' | '%') factor)*
fn term(&mut self) -> Result<Expr, String> {
let mut left = self.factor()?;
while let Some(Token::Op(op @ ('*' | '/' | '%'))) = self.peek() {
let op = *op;
self.pos += 1;
let right = self.factor()?;
left = Expr::Bin(Box::new(left), op, Box::new(right));
}
Ok(left)
}
// factor := '-' factor | NUMBER | IDENT | '(' expr ')'
fn factor(&mut self) -> Result<Expr, String> {
match self.next() {
Some(Token::Op('-')) => Ok(Expr::Neg(Box::new(self.factor()?))),
Some(Token::Num(n)) => Ok(Expr::Num(n)),
Some(Token::Ident(name)) => Ok(Expr::Var(name)),
Some(Token::LParen) => {
let e = self.expr()?;
match self.next() {
Some(Token::RParen) => Ok(e),
_ => Err(String::from("expected ')'")),
}
}
_ => Err(String::from("expected a number, variable or '('")),
}
}
}
fn eval(e: &Expr, env: &HashMap<String, i64>) -> Result<i64, String> {
match e {
Expr::Num(n) => Ok(*n),
Expr::Var(name) => env.get(name).copied().ok_or(format!("unknown variable '{name}'")),
Expr::Neg(inner) => eval(inner, env)?.checked_neg().ok_or(String::from("overflow")),
Expr::Bin(l, op, r) => {
let (a, b) = (eval(l, env)?, eval(r, env)?);
let result = match op {
'+' => a.checked_add(b),
'-' => a.checked_sub(b),
'*' => a.checked_mul(b),
_ if b == 0 => return Err(String::from("division by zero")),
'/' => a.checked_div(b),
_ => a.checked_rem(b),
};
result.ok_or(String::from("overflow"))
}
}
}
fn run_line(line: &str, env: &mut HashMap<String, i64>) -> Result<String, String> {
let tokens = tokenize(line)?;
let (name, expr_tokens) = match tokens.as_slice() {
[Token::Let, Token::Ident(name), Token::Assign, rest @ ..] => (Some(name.clone()), rest.to_vec()),
[Token::Let, ..] => return Err(String::from("expected: let NAME = EXPRESSION")),
_ => (None, tokens),
};
let mut parser = Parser { tokens: expr_tokens, pos: 0 };
let expr = parser.expr()?;
if parser.pos < parser.tokens.len() {
return Err(String::from("unexpected input after expression"));
}
let value = eval(&expr, env)?;
match name {
Some(n) => {
env.insert(n.clone(), value);
Ok(format!("{n} = {value}"))
}
None => Ok(value.to_string()),
}
}
fn main() {
let mut env = HashMap::new();
for line in io::stdin().lines() {
let line = line.unwrap();
if line.trim().is_empty() {
continue;
}
match run_line(&line, &mut env) {
Ok(out) => println!("{out}"),
Err(e) => println!("error: {e}"),
}
}
}