utf8

import utf8 from "utf8";

utf8 works on the code points of UTF-8 text held in a []const u8 (a string or a literal works too). It accepts only well-formed UTF-8 as the Unicode standard defines it: overlong encodings, surrogates (U+D800 to U+DFFF), values above U+10FFFF and cut-off sequences are errors.

The module is pure and works in every profile, t0 included: it allocates nothing and never touches the outside world.

Types

Rune

pub struct Rune {
    cp: u32;    // the code point
    len: i64;   // the bytes it took, 1 to 4
}

What decode returns.

Utf8Error

let r: utf8.Rune = utf8.decode("\xE2\x82", 0) or {
    io.println("{err}");          // truncated UTF-8 sequence
    return;
};

CharIter

pub struct CharIter {
    s: []const u8;   // the text
    pos: i64;        // byte offset of the next code point
}

The iterator chars returns. Its next method gives one code point at a time:

var it: utf8.CharIter = utf8.chars("\xC3\xA9z");
let first: ?u32 = it.next();
io.println("{first!} then pos {it.pos}");     // 233 then pos 2

Constants

io.println("{utf8.REPLACEMENT} {utf8.MAX_CODE_POINT}");   // 65533 1114111

Functions

decode

pub fn decode(s: []const u8, off: i64) -> Rune or Utf8Error

Decodes the code point that starts at byte offset off. Step through text by adding each rune’s len:

let text: []const u8 = "h\xC3\xA9!";
var off: i64 = 0;
while (off < text.len) {
    let r: utf8.Rune = utf8.decode(text, off) or {
        io.println("bad byte at {off}: {err}");
        return 1;
    };
    io.println("code point {r.cp} takes {r.len} bytes");
    off = off + r.len;
}

Traps if off is negative or not less than s.len.

encode

pub fn encode(cp: u32, dst: []u8) -> i64 or Utf8Error

Writes the UTF-8 encoding of cp to the start of dst and returns how many bytes it wrote. A surrogate or a value above U+10FFFF is InvalidCodePoint, and nothing is written.

var buf: [4]u8;
let n: i64 = utf8.encode(0x20AC, buf[0..4]) or { return 1; };
io.println("{n} bytes: {buf[0]} {buf[1]} {buf[2]}");   // 3 bytes: 226 130 172

Traps if dst is shorter than the encoding. A 4-byte buffer always fits.

encode_len

pub fn encode_len(cp: u32) -> i64 or Utf8Error

How many bytes encode writes for cp: 1 to 4. InvalidCodePoint for a surrogate or a value above U+10FFFF.

let n: i64 = utf8.encode_len(0x1F600) or { return 1; };   // 4

validate

pub fn validate(s: []const u8) -> bool

True when all of s is well-formed UTF-8. The empty text is.

io.println("{utf8.validate("caf\xC3\xA9")}");   // true
io.println("{utf8.validate("caf\xE9")}");       // false: Latin-1, not UTF-8

count

pub fn count(s: []const u8) -> i64 or Utf8Error

The number of code points in s, or the first error in it.

let n: i64 = utf8.count("caf\xC3\xA9") or { return 1; };
io.println("{n} code points");    // 4 code points

chars

pub fn chars(s: []const u8) -> CharIter

The code points of s, for a for loop. Invalid bytes don’t stop the loop: each broken piece yields one REPLACEMENT, the way browsers decode text.

for (cp in utf8.chars("a\xC3\xA9\xFFz")) {
    io.println("{cp}");     // 97, 233, 65533, 122
}

“Each broken piece” is a maximal ill-formed subpart in the Unicode standard’s terms: the bytes that begin a sequence but can’t finish it. So "\xF0\x9F\x98" at the end of the text is one REPLACEMENT, and the surrogate "\xED\xA0\x80" is three, because no valid sequence starts ED A0.

The iterator borrows s: don’t keep it after the text is gone.

Last modified: 2026-10-03 07:10:23 UTC