Projects · about 60 min

Project: A Word-Frequency Tool with Options

Why this matters

This module puts the course together in complete programs. The first is the kind of tool you will write again and again: a command-line filter that reads text from standard input, takes options, does some processing, and prints a report. Along the way you will make the design decisions that separate a quick script from a reliable tool: how to parse options, where to handle errors, how to make the output deterministic, and how to keep the logic testable.

The specification

wordfreq reads text and prints the most frequent words. Writing the specification down first, precisely, is half the job:

What is a word? Runs of ASCII letters and apostrophes. "don't" stays one word; "rust-lang" is two. Being explicit avoids surprises later.
Options: -n N (how many to show), -i (ignore case), --min-len L (skip short words).
Output: COUNT WORD lines sorted by count, ties alphabetical, then a totals line.
Errors: unknown options and bad values produce a message on stderr and exit status 2, the conventional status for usage errors.

The full details are in the exercise. Notice how every ambiguous case has an answer (ties, empty input, apostrophes): those are exactly the cases tests will probe.

Step 1: the shape of the program

A good structure for command-line tools:

main()
  1. parse arguments into a Config        (pure: &[String] -> Result<Config, String>)
  2. read input                           (I/O)
  3. compute                               (pure: &str, &Config -> data)
  4. print results                         (I/O)
  errors from 1 -> message on stderr, exit(2)

Pure functions take data and return data; they do not print or exit. That makes them easy to test and reuse. All the messy, environment-dependent parts (args, stdin, stdout, exit codes) stay in main.

Step 2: a Config and parsing options by hand

std::env::args() gives the arguments as Strings (module 2). Real projects often use the clap crate, but parsing by hand is straightforward and shows the logic. Collect the options into a struct with defaults:

#[derive(Debug)]
struct Config {
    top: usize,
    ignore_case: bool,
    min_len: usize,
}

fn parse_args(args: &[String]) -> Result<Config, String> {
    let mut config = Config { top: 10, ignore_case: false, min_len: 1 };
    let mut it = args.iter();
    while let Some(arg) = it.next() {
        match arg.as_str() {
            "-i" => config.ignore_case = true,
            "-n" => {
                let v = it.next().ok_or("missing value for -n")?;
                config.top = v.parse().map_err(|_| format!("invalid number for -n: {v}"))?;
            }
            other => return Err(format!("unknown option: {other}")),
        }
    }
    Ok(config)
}

fn main() {
    let tests: [&[&str]; 4] = [&["-i", "-n", "3"], &[], &["-n"], &["--bogus"]];
    for t in tests {
        let args: Vec<String> = t.iter().map(|s| s.to_string()).collect();
        println!("{:?} -> {:?}", t, parse_args(&args));
    }
}

["-i", "-n", "3"] -> Ok(Config { top: 3, ignore_case: true, min_len: 1 })
[] -> Ok(Config { top: 10, ignore_case: false, min_len: 1 })
["-n"] -> Err("missing value for -n")
["--bogus"] -> Err("unknown option: --bogus")

Key techniques:

An iterator over the arguments lets an option consume its value with it.next(); a for loop could not do that.
match arg.as_str() dispatches on the option name (as_str turns &String into &str for matching against literals).
ok_or and map_err turn "missing" and "unparseable" into error messages, and ? returns them. Every way the input can be wrong produces a specific message, never a panic.
Testing parse_args needs no process at all: build a Vec<String> and call it, as main does here.

main then handles the error in one place:

let config = match parse_args(&args) {
    Ok(c) => c,
    Err(e) => {
        eprintln!("{e}");
        process::exit(2);
    }
};

Step 3: splitting text into words

str::split accepts a closure deciding which characters are separators. Everything that is not a letter or apostrophe separates words; empty pieces (between two adjacent separators) are filtered out:

fn words(text: &str) -> impl Iterator<Item = &str> {
    text.split(|c: char| !(c.is_ascii_alphabetic() || c == '\'')).filter(|w| !w.is_empty())
}

fn main() {
    let v: Vec<&str> = words("Don't panic: rust-lang, 42 times!").collect();
    println!("{:?}", v);
}

["Don't", "panic", "rust", "lang", "times"]

The function returns impl Iterator borrowing from text: no copying until we decide to store a word. (Lifetime elision ties the returned slices to text, module 10.)

Step 4: counting

A HashMap<String, usize> with the entry API (module 9). With -i, the key is the lowercased word; otherwise the word itself. to_lowercase allocates a new String, which is fine here.

for w in words(text) {
    if w.chars().count() < config.min_len { continue; }
    let key = if config.ignore_case { w.to_lowercase() } else { w.to_string() };
    *counts.entry(key).or_insert(0) += 1;
    total += 1;
}

Step 5: deterministic output

HashMap iteration order is random (module 9), so the report must sort explicitly. Collect the pairs and sort by count descending, then word ascending:

fn main() {
    let mut ranked = vec![("the".to_string(), 4), ("end".to_string(), 2), ("and".to_string(), 2), ("cat".to_string(), 1)];
    ranked.sort_by(|a, b| b.1.cmp(&a.1).then_with(|| a.0.cmp(&b.0)));
    for (w, c) in ranked.iter().take(3) {
        println!("{c:>4} {w}");
    }
}

   4 the
   2 and
   2 end

then_with breaks ties only when the first comparison says Equal. take(config.top) limits the output, and handles a top larger than the number of words gracefully.

Step 6: assemble and test

Put the pieces together in the exercise. Then test it the way a user would, including the awkward cases the checker uses: empty input, a word repeated in different cases with and without -i, --min-len filtering, and each error. In the sandbox shell you can run it as a real filter:

$ rustc --edition 2021 -O -o wordfreq main.rs
$ ./wordfreq -i -n 5 < /usr/share/common-licenses/GPL-3
$ ./wordfreq -n zero; echo "exit status $?"
invalid number for -n: zero
exit status 2

Extensions to try

--json: print the result as a JSON array (format it by hand with format!; escaping quotes is the interesting part).
Read from files named as arguments instead of stdin, reporting unreadable files with exit status 1.
Unicode words: use char::is_alphabetic instead of the ASCII version, and think about what -i should do with ß.

Recap

Write the specification first, including edge cases and error behavior.
Structure: parse args into a Config, read input, compute with pure functions, print, and handle errors in main.
Parse options with an argument iterator and match; turn problems into Err(String) with ok_or/map_err/?.
Usage errors go to stderr with exit status 2; results go to stdout.
Sort map contents explicitly for deterministic output.

Going further: look at how the clap crate describes options declaratively (its docs on docs.rs), and compare with the hand parser: what does it give you (help text, --opt=value syntax, combined short flags like -in 3)?

Exercises

Exercise 1 · write code

wordfreq

Build wordfreq: it reads text from stdin and prints the most frequent words.

Words: maximal runs of ASCII letters and apostrophes ('), so don't is one word and rust-lang is two. Everything else separates words. Words are compared case-sensitively unless -i is given, in which case they are lowercased.

Options (in any order):

-n N: print the top N words (default 10). N must be a positive integer.
-i: ignore case.
--min-len L: ignore words shorter than L characters (default 1).

Output: one line per word, COUNT WORD, with the count right-aligned in a width of 4, sorted by count descending, then alphabetically. Then a final line total: T words, D distinct (counting only words that passed the filters).

Errors (message on stderr, exit status 2, nothing on stdout): an unknown option (unknown option: X), a missing value (missing value for -n), or a bad number (invalid number for -n: X, likewise for --min-len).

Example: ./prog -i -n 3 with input The cat and the hat. The END and the end. prints

   4 the
   2 and
   2 end
total: 10 words, 5 distinct

File: main.rs

Starting point

use std::collections::HashMap;
use std::env;
use std::io::{self, Read};
use std::process;

struct Config {
    top: usize,
    ignore_case: bool,
    min_len: usize,
}

fn parse_args(args: &[String]) -> Result<Config, String> {
    // TODO: walk the arguments and fill in a Config
    Ok(Config { top: 10, ignore_case: false, min_len: 1 })
}

fn main() {
    let args: Vec<String> = env::args().skip(1).collect();
    let config = match parse_args(&args) {
        Ok(c) => c,
        Err(e) => {
            eprintln!("{e}");
            process::exit(2);
        }
    };
    let mut text = String::new();
    io::stdin().read_to_string(&mut text).unwrap();
    // TODO: count words and print the report
}

Example with arguments -i -n 3

input

The cat and the hat. The END and the end.

expected output

   4 the
   2 and
   2 end
total: 10 words, 5 distinct

Example

input

b a b A a b

expected output

   3 b
   2 a
   1 A
total: 6 words, 3 distinct

Hint 1

Parse options with an iterator: let mut it = args.iter(); while let Some(arg) = it.next() { match arg.as_str() { ... } }; for -n, take the value with it.next().

Hint 2

Split words with text.split(|c: char| !(c.is_ascii_alphabetic() || c == '\'')).filter(|w| !w.is_empty()).

Hint 3

Count into a HashMap, collect into a Vec of pairs, and sort by count descending then word: b.1.cmp(&a.1).then_with(|| a.0.cmp(&b.0)).

Reference solution

use std::collections::HashMap;
use std::env;
use std::io::{self, Read};
use std::process;

struct Config {
    top: usize,
    ignore_case: bool,
    min_len: usize,
}

fn parse_number(flag: &str, value: Option<&String>, positive: bool) -> Result<usize, String> {
    let v = value.ok_or(format!("missing value for {flag}"))?;
    match v.parse::<usize>() {
        Ok(n) if !positive || n > 0 => Ok(n),
        _ => Err(format!("invalid number for {flag}: {v}")),
    }
}

fn parse_args(args: &[String]) -> Result<Config, String> {
    let mut config = Config { top: 10, ignore_case: false, min_len: 1 };
    let mut it = args.iter();
    while let Some(arg) = it.next() {
        match arg.as_str() {
            "-i" => config.ignore_case = true,
            "-n" => config.top = parse_number("-n", it.next(), true)?,
            "--min-len" => config.min_len = parse_number("--min-len", it.next(), false)?,
            other => return Err(format!("unknown option: {other}")),
        }
    }
    Ok(config)
}

fn words(text: &str) -> impl Iterator<Item = &str> {
    text.split(|c: char| !(c.is_ascii_alphabetic() || c == '\'')).filter(|w| !w.is_empty())
}

fn count(text: &str, config: &Config) -> (HashMap<String, usize>, usize) {
    let mut counts = HashMap::new();
    let mut total = 0;
    for w in words(text) {
        if w.chars().count() < config.min_len {
            continue;
        }
        let key = if config.ignore_case { w.to_lowercase() } else { w.to_string() };
        *counts.entry(key).or_insert(0) += 1;
        total += 1;
    }
    (counts, total)
}

fn main() {
    let args: Vec<String> = env::args().skip(1).collect();
    let config = match parse_args(&args) {
        Ok(c) => c,
        Err(e) => {
            eprintln!("{e}");
            process::exit(2);
        }
    };
    let mut text = String::new();
    io::stdin().read_to_string(&mut text).unwrap();
    let (counts, total) = count(&text, &config);
    let distinct = counts.len();
    let mut ranked: Vec<(String, usize)> = counts.into_iter().collect();
    ranked.sort_by(|a, b| b.1.cmp(&a.1).then_with(|| a.0.cmp(&b.0)));
    for (w, c) in ranked.iter().take(config.top) {
        println!("{c:>4} {w}");
    }
    println!("total: {total} words, {distinct} distinct");
}

Exercise 2 · quiz

Why split parse_args, count and printing?

The solution separates argument parsing (parse_args), counting (count) and output (main). Which are real benefits? Select all that apply.

Each piece can be unit-tested without running the whole program or feeding stdin
Error handling is concentrated: parse_args returns a Result, and main alone decides on messages and exit codes
The compiled program is necessarily faster
Changing the output format does not risk breaking the counting logic

Answer

Each piece can be unit-tested without running the whole program or feeding stdin
Error handling is concentrated: parse_args returns a Result, and main alone decides on messages and exit codes
Changing the output format does not risk breaking the counting logic

Separating pure logic (strings in, data out) from I/O (reading stdin, printing, exiting) is the most useful structural habit for command-line programs. The logic becomes testable with plain #[test] functions, and policy decisions such as exit codes live in one place. Performance is unaffected either way.