unicode

import unicode from "unicode";

let g: collections.Vec_i64 = unicode.boundaries("e\u{301}!", unicode.Granularity.Grapheme);   // 0, 3, 4

Text in tape is UTF-8, and most of the time bytes are all you need. When text has to behave the way people read it, unicode has Unicode’s rules:

• what a character is: its category, its script, its properties
• case: upper and lower case for every script, and case folding for comparing without case
• normalization: é written as one character or as e plus an accent, made the same
• boundaries: where a user-perceived character (a grapheme), a word or a sentence ends
• collation: the order people expect names sorted in

The tables are Unicode 17.0.0’s, and the collation is CLDR’s root order (the same for every language; locale tailorings such as Swedish putting ä after z aren’t there yet). Bytes that aren’t valid UTF-8 are read as U+FFFD, so nothing fails on broken text. The module is pure: it only computes.

The tables are large: a program that imports unicode grows by about 900 KB.

Moving a cursor by graphemes

What looks like one character on screen can be several code points: é written as e and a combining accent, a flag (two regional indicators), a family emoji (seven code points joined by zero-width joiners). A text editor’s cursor, backspace and selection should step over those whole. boundaries with Granularity.Grapheme gives the byte offsets where one ends and the next begins:

/* The byte offset one grapheme after `at` (the end stays the end) */
fn next_grapheme(text: []const u8, at: i64) -> i64 {
    let b: collections.Vec_i64 = unicode.boundaries(text, unicode.Granularity.Grapheme);
    for (x in b.slice()) { if (x > at) { return x; } }
    return text.len;
}

/* The byte offset one grapheme before `at` */
fn prev_grapheme(text: []const u8, at: i64) -> i64 {
    let b: collections.Vec_i64 = unicode.boundaries(text, unicode.Granularity.Grapheme);
    var prev: i64 = 0;
    for (x in b.slice()) { if (x >= at) { return prev; } prev = x; }
    return prev;
}

For a long text, compute the boundaries of the line the cursor is on, not of the whole document: graphemes never cross a line break.

Comparing text: normalize, then fold

The same word can be stored as different bytes: é is U+00E9, or e followed by U+0301. Normalizing to NFC makes them equal. Case folding then removes the differences of case, for every script, including the ones ASCII lowercasing misses (ß and SS, Σ and ς):

fn same_text(a: []const u8, b: []const u8) -> bool {
    let fa: string = unicode.fold(unicode.normalize(a, unicode.Form.Nfc));
    let fb: string = unicode.fold(unicode.normalize(b, unicode.Form.Nfc));
    return fa == fb;
}

same_text("Café STRASSE", "cafe\u{301} straße");   // true

Use fold for comparing and for keys (a search index, a map of names); use to_lower and to_upper for showing text.

Sorting names

Byte order puts Zoe before apple and Émile after zebra. compare sorts the way people expect, and with numeric set, file9 comes before file10:

import sort from "sort";

fn name_less(a: *const []const u8, b: *const []const u8) -> bool {
    var st: unicode.CollateSettings = unicode.default_settings();
    st.numeric = true;
    return unicode.compare(*a, *b, &st) < 0;
}

@tape { sort.gen_sort_by([]const u8, name_less); }

var names: [5][]const u8 = ["file10", "Émile", "file9", "zebra", "emile"];
sort_by_name_less(names[0..5]);   // emile, Émile, file9, file10, zebra

Properties

general_category, script

pub fn general_category(cp: i64) -> []const u8
pub fn script(cp: i64) -> []const u8

A code point’s General_Category as its short name ("Lu" uppercase letter, "Nd" decimal digit, "Zs" space; "Cn" for an unassigned code point), and its script’s name ("Latin", "Greek", "Han"; "Common" for punctuation and digits shared by scripts, "Inherited" for combining marks, "Unknown").

unicode.general_category(0x3B1);   // "Ll"
unicode.script(0x3B1);             // "Greek"

property, has_property, Set

pub fn property(name: []const u8) -> ?Set
pub fn has_property(cp: i64, name: []const u8) -> bool
pub fn general_category_set(name: []const u8) -> ?Set
pub fn script_set(name: []const u8) -> ?Set
pub fn script_extensions_set(name: []const u8) -> ?Set
pub fn binary_set(name: []const u8) -> ?Set
pub fn Set.contains(*const self, cp: i64) -> bool
pub fn Set.len(*const self) -> i64
pub fn Set.range(*const self, i: i64) -> Range     // Range { first: i64; last: i64; }

A property’s code points, by the names a regular expression’s \p{...} takes: a General_Category value or group (Lu, Uppercase_Letter, L), a script (Greek or Grek), a binary property (Alphabetic, White_Space, Emoji, Extended_Pictographic, Any, ASCII, Assigned, …), or one spelled out (gc=Nd, Script=Cyrillic, scx=Deva). Names are spelled exactly as Unicode spells them. property is null for a name it doesn’t know; has_property traps for one.

let greek: unicode.Set = unicode.property("Greek") or { return 1; };
greek.contains(0x3A9);                            // true: Ω
unicode.has_property(0x0967, "Nd");               // true: Devanagari 1

A Set is a view of the module’s tables: free to copy and keep. Look one up once and reuse it; the lookup by name is the slow part. The _set functions look in one kind of property only; script_extensions_set also has the characters a script shares with others (the Devanagari danda is Common, but in the Script_Extensions of Devanagari, Bengali, …).

property_ranges

pub fn property_ranges(name: []const u8, out: *collections.Vec_i64) -> bool

The ranges of property(name) appended to out as first, last pairs; false for an unknown name. It’s what regex takes for \p{...}:

let re: regex.Regex = regex.compile(r"""\p{Lu}\p{Ll}+""", unicode.property_ranges) or { return 1; };

sequences

pub fn sequences(name: []const u8) -> ?[]const u8

A property of strings (the emoji sequences: Basic_Emoji, Emoji_Keycap_Sequence, RGI_Emoji_Modifier_Sequence, RGI_Emoji_Flag_Sequence, RGI_Emoji_Tag_Sequence, RGI_Emoji_ZWJ_Sequence) as text: a line per sequence, its code points in hex separated by spaces ("1F1EB 1F1EE" is the Finnish flag).

Case

to_upper, to_lower, fold

pub fn to_upper(s: []const u8) -> string
pub fn to_lower(s: []const u8) -> string
pub fn fold(s: []const u8) -> string

The text in upper case, in lower case, and case-folded. They use the full mappings, so the length can change: to_upper("straße") is "STRASSE". to_lower writes the final sigma at the end of a word: "ΟΔΟΣ" becomes "οδος", its last letter ς. No language’s special rules: to_upper("i") is "I" in Turkish text too. For ASCII-only text, str.to_upper is cheaper.

unicode.to_upper("Straße");          // "STRASSE"
unicode.to_lower("ΟΔΥΣΣΕΥΣ");        // "οδυσσευς"
unicode.fold("Straße") == unicode.fold("STRASSE");   // true

simple_upper, simple_lower, simple_title, simple_fold, simple_fold_next

pub fn simple_upper(cp: i64) -> i64
pub fn simple_lower(cp: i64) -> i64
pub fn simple_title(cp: i64) -> i64
pub fn simple_fold(cp: i64) -> i64
pub fn simple_fold_next(cp: i64) -> i64

One code point to one code point (the code point itself when it has no mapping). simple_fold_next walks every code point that folds the same way: from k to K (the Kelvin sign) to K and back to k.

unicode.simple_upper(0xDF);          // 0xDF: ß has no one-letter upper case
unicode.simple_title(0x1C6);         // 0x1C5: dž to Dž
unicode.simple_fold_next(0x6B);      // 0x212A

full_upper, full_lower, full_fold

pub fn full_upper(cp: i64, out: *collections.Vec_i64)
pub fn full_lower(cp: i64, out: *collections.Vec_i64)
pub fn full_fold(cp: i64, out: *collections.Vec_i64)

One code point’s full mapping (one to three code points) appended to out, for code that works on code points. No context: full_lower maps Σ to σ everywhere.

Normalization

normalize, normalize_cps, is_normalized, ccc

pub enum Form : u8 { Nfc = 1: "NFC"; Nfd: "NFD"; Nfkc: "NFKC"; Nfkd: "NFKD"; }
pub fn normalize(s: []const u8, form: Form) -> string
pub fn normalize_cps(cps: []const i64, form: Form, out: *collections.Vec_i64)
pub fn is_normalized(s: []const u8, form: Form) -> bool
pub fn ccc(cp: i64) -> i64

The four normalization forms: NFC composes (e + accent to é), NFD decomposes, and the K forms also replace compatibility characters (the fi ligature to fi, ² to 2, full-width letters to ASCII). Store and compare text in NFC; use NFKC for identifiers and search. ccc is a code point’s canonical combining class (0 for a base character).

unicode.normalize("A\u{30a}", unicode.Form.Nfc);       // "Å", one code point
unicode.normalize("\u{fb01}le", unicode.Form.Nfkc);    // "file"
unicode.is_normalized("\u{c5}", unicode.Form.Nfd);     // false

Boundaries

boundaries, boundaries_utf16

pub enum Granularity : u8 { Grapheme = 1: "grapheme"; Word: "word"; Sentence: "sentence"; }
pub fn boundaries(s: []const u8, g: Granularity) -> collections.Vec_i64
pub fn boundaries_utf16(units: []const i64, g: Granularity) -> collections.Vec_i64

The byte offsets where graphemes, words or sentences begin and end, in order, 0 and s.len included; each piece is s[b[i]..b[i + 1]]. Words include the spaces and punctuation between them as pieces of their own (check with word_like). The rules are Unicode’s defaults (UAX #29); Thai, Chinese and Japanese have no dictionary here, so their words break by script, not by meaning. boundaries_utf16 does the same over UTF-16 code units and gives unit offsets.

let s: []const u8 = "Hi. Bye.";
let b: collections.Vec_i64 = unicode.boundaries(s, unicode.Granularity.Sentence);   // 0, 4, 8

word_like, word_like_utf16

pub fn word_like(segment: []const u8) -> bool
pub fn word_like_utf16(units: []const i64) -> bool

Whether a word piece is a word (holds a letter, a digit, kana or an ideograph) rather than spaces or punctuation: what to count, select on a double click, or index.

let s: []const u8 = "can't stop, 3.14";
let w: collections.Vec_i64 = unicode.boundaries(s, unicode.Granularity.Word);
var i: i64 = 0;
while (i + 1 < w.len()) {
    let piece: []const u8 = s[w.get(i)..w.get(i + 1)];
    if (unicode.word_like(piece)) { io.println(piece); }   // can't, stop, 3.14
    i = i + 1;
}

Collation

compare, compare_utf16, compare_cps, default_settings

pub enum Sensitivity : u8 { Base = 1: "base"; Accent: "accent"; Case: "case"; Variant: "variant"; }
pub enum CaseFirst : u8 { Off = 1: "false"; Upper: "upper"; Lower: "lower"; }
pub struct CollateSettings { sensitivity: Sensitivity; ignore_punct: bool; numeric: bool; case_first: CaseFirst; }
pub fn default_settings() -> CollateSettings
pub fn compare(x: []const u8, y: []const u8, st: *const CollateSettings) -> i64
pub fn compare_utf16(x: []const i64, y: []const i64, st: *const CollateSettings) -> i64
pub fn compare_cps(x: []const i64, y: []const i64, st: *const CollateSettings) -> i64

-1, 0 or 1: x sorts before, the same as, or after y. The settings choose which differences count:

var st: unicode.CollateSettings = unicode.default_settings();
unicode.compare("résumé", "resume", &st);    // 1: the accent decides
st.sensitivity = unicode.Sensitivity.Base;
unicode.compare("Äpfel", "apfel", &st);       // 0

Compare costs more than byte comparison (each call normalizes and looks up both strings); for a large list, sort once and keep the order.

Constants

Not here yet

Line breaking (where text may wrap, UAX #14), bidirectional text (UAX #9), East Asian width, title-casing a string, language-specific case rules (Turkish, Lithuanian), locale collation (Swedish, Spanish, German phonebook) and sort keys.

Last modified: 2026-10-25 23:26:31 UTC