Projects · about 90 min

Project: An Expression Interpreter

Why this matters

Every programming language, calculator, spreadsheet formula, query language and configuration format is processed the same way: text is split into tokens, tokens are parsed into a tree that captures the structure, and the tree is evaluated. Building a small interpreter end to end uses nearly everything in this course: enums with data, Box for recursive types, pattern matching, Result and ? for errors, iterators, a HashMap for variables, and careful ownership of the tree. It is also one of the most satisfying programs to write.

The finished program is a calculator with variables:

> 1 + 2 * 3
7
> let x = 10
x = 10
> let y = x * x - 1
y = 99
> y / (x - 10)
error: division by zero
> -x % 3
-1
> (4 + 5
error: expected ')'

The pipeline

 "let y = x * x - 1"
        |
        |  tokenize: characters -> tokens
        v
 [Let, Ident("y"), Assign, Ident("x"), Op('*'), Ident("x"), Op('-'), Num(1)]
        |
        |  parse: tokens -> tree (respecting precedence: * before -)
        v
              Bin(-)
             /      \
         Bin(*)     Num(1)
         /    \
    Var(x)   Var(x)
        |
        |  eval: tree + variables -> value
        v
       99        (and store y = 99)

Each stage has one job and its own error messages, and each can be tested on its own.

Stage 1: tokens

A token is the smallest meaningful unit. An enum describes them exactly:

#[derive(Debug, Clone, PartialEq)]
enum Token {
    Num(i64),
    Ident(String),
    Op(char),
    LParen,
    RParen,
    Let,
    Assign,
}

The tokenizer walks the characters with an index. Digits accumulate into a number, letters into a name (which may turn out to be the keyword let), and single characters map to operators; anything else is an error. This is a small state machine like the previous lesson's, written with inner while loops instead of an explicit state enum:

while i < chars.len() {
    let c = chars[i];
    if c.is_whitespace() { i += 1; }
    else if c.is_ascii_digit() { /* consume all digits, parse into i64 */ }
    else if c.is_ascii_alphabetic() || c == '_' { /* consume a name; "let" becomes Token::Let */ }
    else { /* + - * / % ( ) = become tokens; anything else: Err("unexpected character 'c'") */ }
}

The first exercise asks you to write it. Collecting the characters into a Vec<char> first makes indexing by character easy (strings cannot be indexed, module 6).

Stage 2: the tree

The structure of an expression is a tree, and a recursive enum with Box (module 13) represents it:

#[derive(Debug)]
enum Expr {
    Num(i64),
    Var(String),
    Neg(Box<Expr>),
    Bin(Box<Expr>, char, Box<Expr>),
}

Bin(left, op, right) is a binary operation; Neg is unary minus. The tree has no parentheses: they only influence its shape during parsing.

Stage 3: parsing with recursive descent

Precedence ("* binds tighter than +") is expressed as a grammar, one rule per precedence level:

expr   := term   (('+' | '-') term)*
term   := factor (('*' | '/' | '%') factor)*
factor := '-' factor | NUMBER | IDENT | '(' expr ')'

Read it as: an expression is one or more terms joined by +/-; a term is one or more factors joined by *, /, %; a factor is a number, a variable, a negated factor, or a whole expression in parentheses. Recursive descent turns each rule into a method that calls the methods for the rules it mentions:

fn expr(&mut self) -> Result<Expr, String> {
    let mut left = self.term()?;
    while let Some(Token::Op(op @ ('+' | '-'))) = self.peek() {
        let op = *op;
        self.pos += 1;
        let right = self.term()?;
        left = Expr::Bin(Box::new(left), op, Box::new(right));
    }
    Ok(left)
}

Ownership is pleasantly simple: each Box::new(left) moves the subtree into its parent, so the final tree owns all its nodes, and dropping the root frees everything.

After parsing, check that all tokens were used: in 4 5, expr happily parses 4 and stops, so leftover tokens mean an error.

Stage 4: evaluation

Evaluation is a recursive match over the tree, which mirrors its definition exactly:

fn eval(e: &Expr, env: &HashMap<String, i64>) -> Result<i64, String> {
    match e {
        Expr::Num(n) => Ok(*n),
        Expr::Var(name) => env.get(name).copied().ok_or(format!("unknown variable '{name}'")),
        Expr::Neg(inner) => eval(inner, env)?.checked_neg().ok_or(String::from("overflow")),
        Expr::Bin(l, op, r) => { /* evaluate both sides with ?, then apply op with checked arithmetic */ }
    }
}

Statements: let

A line is either let NAME = EXPRESSION or an expression. Slice patterns (module 7) recognize the let form directly on the token list:

match tokens.as_slice() {
    [Token::Let, Token::Ident(name), Token::Assign, rest @ ..] => /* parse rest, store under name */,
    [Token::Let, ..] => Err("expected: let NAME = EXPRESSION"),
    _ => /* parse all tokens as an expression and print its value */,
}

rest @ .. binds the remaining tokens as a slice. main then just loops over input lines, calls this for each, and prints the result or error: MESSAGE, keeping the variables in a HashMap across lines.

Errors are part of the design

Every stage reports problems as Err(String) with a specific message, and the program continues with the next line. Count the failure modes the exercise checks: unknown characters, missing ), missing operands, leftover input, malformed let, unknown variables, division by zero, overflow. None of them panics. That is the difference between a toy and a tool.

Extensions to try

Recap

Going further: read Robert Nystrom's free book "Crafting Interpreters" (craftinginterpreters.com); its first half builds a tree-walking interpreter with exactly this architecture.

Exercises

Exercise 1 · write code

Stage 1: the tokenizer

Write fn tokenize(src: &str) -> Result<Vec<Token>, String> for the Token enum in the starter:

The given main prints the tokens of each stdin line with {:?}, or error: MESSAGE:

[Let, Ident("rate"), Assign, Num(12), Op('*'), LParen, Ident("x_1"), Op('-'), Num(3), RParen]
[Num(7), Op('%'), Num(2)]
error: unexpected character '$'

for the lines let rate = 12 * (x_1 - 3), 7%2, a $ b.

File: main.rs

Starting point

use std::io;

#[derive(Debug, Clone, PartialEq)]
enum Token {
    Num(i64),
    Ident(String),
    Op(char),
    LParen,
    RParen,
    Let,
    Assign,
}

fn tokenize(src: &str) -> Result<Vec<Token>, String> {
    Ok(Vec::new())
}

fn main() {
    for line in io::stdin().lines() {
        let line = line.unwrap();
        match tokenize(&line) {
            Ok(tokens) => println!("{:?}", tokens),
            Err(e) => println!("error: {e}"),
        }
    }
}

Example

input

let rate = 12 * (x_1 - 3)
7%2
a $ b

expected output

[Let, Ident("rate"), Assign, Num(12), Op('*'), LParen, Ident("x_1"), Op('-'), Num(3), RParen]
[Num(7), Op('%'), Num(2)]
error: unexpected character '$'

Example

input

letter let _x9
99999999999999999999

expected output

[Ident("letter"), Let, Ident("_x9")]
error: number too large: 99999999999999999999
Hint 1

Collect the characters first: let chars: Vec<char> = src.chars().collect(); and walk them with an index i.

Hint 2

For a number: remember start = i, advance while digits, then chars[start..i].iter().collect::<String>() and parse it.

Hint 3

For single-character tokens, match c { '+' | '-' | '*' | '/' | '%' => Token::Op(c), '(' => Token::LParen, ... } and advance by one.

Reference solution
use std::io;

#[derive(Debug, Clone, PartialEq)]
enum Token {
    Num(i64),
    Ident(String),
    Op(char),
    LParen,
    RParen,
    Let,
    Assign,
}

fn tokenize(src: &str) -> Result<Vec<Token>, String> {
    let chars: Vec<char> = src.chars().collect();
    let mut tokens = Vec::new();
    let mut i = 0;
    while i < chars.len() {
        let c = chars[i];
        if c.is_whitespace() {
            i += 1;
        } else if c.is_ascii_digit() {
            let start = i;
            while i < chars.len() && chars[i].is_ascii_digit() {
                i += 1;
            }
            let text: String = chars[start..i].iter().collect();
            let n = text.parse().map_err(|_| format!("number too large: {text}"))?;
            tokens.push(Token::Num(n));
        } else if c.is_ascii_alphabetic() || c == '_' {
            let start = i;
            while i < chars.len() && (chars[i].is_ascii_alphanumeric() || chars[i] == '_') {
                i += 1;
            }
            let word: String = chars[start..i].iter().collect();
            tokens.push(if word == "let" { Token::Let } else { Token::Ident(word) });
        } else {
            tokens.push(match c {
                '+' | '-' | '*' | '/' | '%' => Token::Op(c),
                '(' => Token::LParen,
                ')' => Token::RParen,
                '=' => Token::Assign,
                _ => return Err(format!("unexpected character '{c}'")),
            });
            i += 1;
        }
    }
    Ok(tokens)
}

fn main() {
    for line in io::stdin().lines() {
        let line = line.unwrap();
        match tokenize(&line) {
            Ok(tokens) => println!("{:?}", tokens),
            Err(e) => println!("error: {e}"),
        }
    }
}

Exercise 2 · write code

Stages 2 to 4: parse and evaluate

The starter contains the tokenizer, the Expr tree type, run_line (handles let) and main. Implement the three parser methods expr, term and factor (recursive descent, following the grammar in the comments) and eval. Error messages, exactly:

Operators are left-associative (10 - 3 - 2 is 5); unary minus binds tighter than * (-2 * 3 is -6). / and % behave like Rust's integer operators. For the input

1 + 2 * 3
let x = 10
let y = x * x - 1
y / (x - 10)
-x % 3
(4 + 5

the program prints

7
x = 10
y = 99
error: division by zero
-1
error: expected ')'

File: main.rs

Starting point

use std::collections::HashMap;
use std::io;

#[derive(Debug, Clone, PartialEq)]
enum Token {
    Num(i64),
    Ident(String),
    Op(char),
    LParen,
    RParen,
    Let,
    Assign,
}

fn tokenize(src: &str) -> Result<Vec<Token>, String> {
    let chars: Vec<char> = src.chars().collect();
    let mut tokens = Vec::new();
    let mut i = 0;
    while i < chars.len() {
        let c = chars[i];
        if c.is_whitespace() {
            i += 1;
        } else if c.is_ascii_digit() {
            let start = i;
            while i < chars.len() && chars[i].is_ascii_digit() {
                i += 1;
            }
            let text: String = chars[start..i].iter().collect();
            let n = text.parse().map_err(|_| format!("number too large: {text}"))?;
            tokens.push(Token::Num(n));
        } else if c.is_ascii_alphabetic() || c == '_' {
            let start = i;
            while i < chars.len() && (chars[i].is_ascii_alphanumeric() || chars[i] == '_') {
                i += 1;
            }
            let word: String = chars[start..i].iter().collect();
            tokens.push(if word == "let" { Token::Let } else { Token::Ident(word) });
        } else {
            tokens.push(match c {
                '+' | '-' | '*' | '/' | '%' => Token::Op(c),
                '(' => Token::LParen,
                ')' => Token::RParen,
                '=' => Token::Assign,
                _ => return Err(format!("unexpected character '{c}'")),
            });
            i += 1;
        }
    }
    Ok(tokens)
}

#[derive(Debug)]
enum Expr {
    Num(i64),
    Var(String),
    Neg(Box<Expr>),
    Bin(Box<Expr>, char, Box<Expr>),
}

struct Parser {
    tokens: Vec<Token>,
    pos: usize,
}

impl Parser {
    fn peek(&self) -> Option<&Token> {
        self.tokens.get(self.pos)
    }

    fn next(&mut self) -> Option<Token> {
        let t = self.tokens.get(self.pos).cloned();
        self.pos += 1;
        t
    }

    // expr := term (('+' | '-') term)*
    fn expr(&mut self) -> Result<Expr, String> {
        todo!("parse terms separated by + and -")
    }

    // term := factor (('*' | '/' | '%') factor)*
    fn term(&mut self) -> Result<Expr, String> {
        todo!("parse factors separated by *, / and %")
    }

    // factor := '-' factor | NUMBER | IDENT | '(' expr ')'
    fn factor(&mut self) -> Result<Expr, String> {
        todo!("parse a negation, number, variable or parenthesized expression")
    }
}

fn eval(e: &Expr, env: &HashMap<String, i64>) -> Result<i64, String> {
    todo!("evaluate the tree")
}

fn run_line(line: &str, env: &mut HashMap<String, i64>) -> Result<String, String> {
    let tokens = tokenize(line)?;
    let (name, expr_tokens) = match tokens.as_slice() {
        [Token::Let, Token::Ident(name), Token::Assign, rest @ ..] => (Some(name.clone()), rest.to_vec()),
        [Token::Let, ..] => return Err(String::from("expected: let NAME = EXPRESSION")),
        _ => (None, tokens),
    };
    let mut parser = Parser { tokens: expr_tokens, pos: 0 };
    let expr = parser.expr()?;
    if parser.pos < parser.tokens.len() {
        return Err(String::from("unexpected input after expression"));
    }
    let value = eval(&expr, env)?;
    match name {
        Some(n) => {
            env.insert(n.clone(), value);
            Ok(format!("{n} = {value}"))
        }
        None => Ok(value.to_string()),
    }
}

fn main() {
    let mut env = HashMap::new();
    for line in io::stdin().lines() {
        let line = line.unwrap();
        if line.trim().is_empty() {
            continue;
        }
        match run_line(&line, &mut env) {
            Ok(out) => println!("{out}"),
            Err(e) => println!("error: {e}"),
        }
    }
}

Example

input

1 + 2 * 3
let x = 10
let y = x * x - 1
y / (x - 10)
-x % 3
(4 + 5

expected output

7
x = 10
y = 99
error: division by zero
-1
error: expected ')'

Example

input

10 - 3 - 2
2 * 3 + 4 * 5
-2 * 3
--4
100 / 10 / 5
(1 + 2) * (3 + 4)

expected output

5
26
-6
4
2
21
Hint 1

expr: let mut left = self.term()?; while let Some(Token::Op(op @ ('+' | '-'))) = self.peek() { let op = *op; self.pos += 1; let right = self.term()?; left = Expr::Bin(Box::new(left), op, Box::new(right)); } Ok(left). term is the same with *, /, % and factor.

Hint 2

factor: match self.next() { Some(Token::Op('-')) => Ok(Expr::Neg(Box::new(self.factor()?))), Some(Token::Num(n)) => ..., Some(Token::LParen) => ..., _ => Err(...) }.

Hint 3

eval: recurse with ?; use checked_add, checked_sub, checked_mul, checked_div, checked_rem, checked_neg and turn None into overflow; check for a zero divisor first.

Reference solution
use std::collections::HashMap;
use std::io;

#[derive(Debug, Clone, PartialEq)]
enum Token {
    Num(i64),
    Ident(String),
    Op(char),
    LParen,
    RParen,
    Let,
    Assign,
}

fn tokenize(src: &str) -> Result<Vec<Token>, String> {
    let chars: Vec<char> = src.chars().collect();
    let mut tokens = Vec::new();
    let mut i = 0;
    while i < chars.len() {
        let c = chars[i];
        if c.is_whitespace() {
            i += 1;
        } else if c.is_ascii_digit() {
            let start = i;
            while i < chars.len() && chars[i].is_ascii_digit() {
                i += 1;
            }
            let text: String = chars[start..i].iter().collect();
            let n = text.parse().map_err(|_| format!("number too large: {text}"))?;
            tokens.push(Token::Num(n));
        } else if c.is_ascii_alphabetic() || c == '_' {
            let start = i;
            while i < chars.len() && (chars[i].is_ascii_alphanumeric() || chars[i] == '_') {
                i += 1;
            }
            let word: String = chars[start..i].iter().collect();
            tokens.push(if word == "let" { Token::Let } else { Token::Ident(word) });
        } else {
            tokens.push(match c {
                '+' | '-' | '*' | '/' | '%' => Token::Op(c),
                '(' => Token::LParen,
                ')' => Token::RParen,
                '=' => Token::Assign,
                _ => return Err(format!("unexpected character '{c}'")),
            });
            i += 1;
        }
    }
    Ok(tokens)
}

#[derive(Debug)]
enum Expr {
    Num(i64),
    Var(String),
    Neg(Box<Expr>),
    Bin(Box<Expr>, char, Box<Expr>),
}

struct Parser {
    tokens: Vec<Token>,
    pos: usize,
}

impl Parser {
    fn peek(&self) -> Option<&Token> {
        self.tokens.get(self.pos)
    }

    fn next(&mut self) -> Option<Token> {
        let t = self.tokens.get(self.pos).cloned();
        self.pos += 1;
        t
    }

    // expr := term (('+' | '-') term)*
    fn expr(&mut self) -> Result<Expr, String> {
        let mut left = self.term()?;
        while let Some(Token::Op(op @ ('+' | '-'))) = self.peek() {
            let op = *op;
            self.pos += 1;
            let right = self.term()?;
            left = Expr::Bin(Box::new(left), op, Box::new(right));
        }
        Ok(left)
    }

    // term := factor (('*' | '/' | '%') factor)*
    fn term(&mut self) -> Result<Expr, String> {
        let mut left = self.factor()?;
        while let Some(Token::Op(op @ ('*' | '/' | '%'))) = self.peek() {
            let op = *op;
            self.pos += 1;
            let right = self.factor()?;
            left = Expr::Bin(Box::new(left), op, Box::new(right));
        }
        Ok(left)
    }

    // factor := '-' factor | NUMBER | IDENT | '(' expr ')'
    fn factor(&mut self) -> Result<Expr, String> {
        match self.next() {
            Some(Token::Op('-')) => Ok(Expr::Neg(Box::new(self.factor()?))),
            Some(Token::Num(n)) => Ok(Expr::Num(n)),
            Some(Token::Ident(name)) => Ok(Expr::Var(name)),
            Some(Token::LParen) => {
                let e = self.expr()?;
                match self.next() {
                    Some(Token::RParen) => Ok(e),
                    _ => Err(String::from("expected ')'")),
                }
            }
            _ => Err(String::from("expected a number, variable or '('")),
        }
    }
}

fn eval(e: &Expr, env: &HashMap<String, i64>) -> Result<i64, String> {
    match e {
        Expr::Num(n) => Ok(*n),
        Expr::Var(name) => env.get(name).copied().ok_or(format!("unknown variable '{name}'")),
        Expr::Neg(inner) => eval(inner, env)?.checked_neg().ok_or(String::from("overflow")),
        Expr::Bin(l, op, r) => {
            let (a, b) = (eval(l, env)?, eval(r, env)?);
            let result = match op {
                '+' => a.checked_add(b),
                '-' => a.checked_sub(b),
                '*' => a.checked_mul(b),
                _ if b == 0 => return Err(String::from("division by zero")),
                '/' => a.checked_div(b),
                _ => a.checked_rem(b),
            };
            result.ok_or(String::from("overflow"))
        }
    }
}

fn run_line(line: &str, env: &mut HashMap<String, i64>) -> Result<String, String> {
    let tokens = tokenize(line)?;
    let (name, expr_tokens) = match tokens.as_slice() {
        [Token::Let, Token::Ident(name), Token::Assign, rest @ ..] => (Some(name.clone()), rest.to_vec()),
        [Token::Let, ..] => return Err(String::from("expected: let NAME = EXPRESSION")),
        _ => (None, tokens),
    };
    let mut parser = Parser { tokens: expr_tokens, pos: 0 };
    let expr = parser.expr()?;
    if parser.pos < parser.tokens.len() {
        return Err(String::from("unexpected input after expression"));
    }
    let value = eval(&expr, env)?;
    match name {
        Some(n) => {
            env.insert(n.clone(), value);
            Ok(format!("{n} = {value}"))
        }
        None => Ok(value.to_string()),
    }
}

fn main() {
    let mut env = HashMap::new();
    for line in io::stdin().lines() {
        let line = line.unwrap();
        if line.trim().is_empty() {
            continue;
        }
        match run_line(&line, &mut env) {
            Ok(out) => println!("{out}"),
            Err(e) => println!("error: {e}"),
        }
    }
}