brotli

tape
import brotli from "brotli";

let page: []const u8 = "<html>...</html>";
var br: collections.Vec_u8 = brotli.compress(page, 11) or { return 1; };
// br.slice() is a Content-Encoding: br body, or a .br file's bytes

let body: string = fs.read_file("index.html.br") or { return 1; };
var html: collections.Vec_u8 = brotli.decompress(body, 16 << 20) or {
    io.println("not good brotli: {err}");
    return 1;
};

brotli reads brotli (RFC 7932), the br of Content-Encoding. Browsers prefer it to gzip, and sites often keep pre-compressed .br copies of their files. Every stream the RFC allows decodes:

  • every window size from 1 KiB to 16 MiB;
  • the context modes and their trees;
  • the 120 KiB static dictionary with its 121 word transforms;
  • metadata and uncompressed blocks.

Use compress and decompress when the data fits in memory. Use an Encoder or a Decoder when it arrives in pieces, such as an HTTP body: they hold a bounded amount of memory however long the stream is.

Compressed output is deterministic: the same input at the same quality gives the same bytes, in the VM and natively, however the input is split into pushes. Any brotli decoder reads it. It is not the reference encoder’s output: at quality 11 it comes out about the size of gzip -9, where the reference encoder’s is 9–15% smaller.

Decompression is strict: anything that isn’t a valid stream is an error, including bytes left over after the stream ends. brotli is a pure module, separate from compress so that a program which only needs gzip doesn’t carry the dictionary.

Errors

tape
pub enum BrotliError : u8 {
    BadData = 1: "not valid brotli data";
    Truncated = 2: "brotli data ends early";
    TooLarge = 3: "decompressed data over the limit";
    BadQuality = 4: "brotli quality outside 0..11";
}
  • BadData: the bytes are not a brotli stream. Causes include a broken prefix code, a distance before the start of the output that is not a dictionary word, non-zero padding, or bytes after the end.
  • Truncated: the data ends before the stream does.
  • TooLarge: the output would pass the max_out you gave.
  • BadQuality: compressing at a quality outside 0..11.

Whole buffers

compress

tape
pub fn compress(data: []const u8, quality: i64) -> collections.Vec_u8 or BrotliError

Compresses data at quality, from 0 (fastest) to 11 (smallest), as brotli numbers it. Fails only with BadQuality. Data that doesn’t compress comes out a few bytes larger than it went in.

tape
let css: string = fs.read_file("site.css") or { return 1; };
var br: collections.Vec_u8 = brotli.compress(css, 11) or { return 1; };
fs.write_file("site.css.br", br.slice()) or { return 1; };

decompress

tape
pub fn decompress(data: []const u8, max_out: i64) -> collections.Vec_u8 or BrotliError

Decompresses data, which must be exactly one brotli stream. max_out is the most output you accept: a small input that would expand to gigabytes stops with TooLarge at that size, and the buffer never grows past it. Traps when max_out is negative.

Streams

tape
pub fn decoder() -> Decoder
pub fn Decoder.push(*self, data: []const u8) -> i64 or BrotliError
pub fn Decoder.read(*self, dst: []u8) -> i64
pub fn Decoder.done(*const self) -> bool
pub fn Decoder.free(*self)

A Decoder works like compress’s Inflater:

  1. push the compressed bytes as they come. It returns how many it took: all of them, or fewer when 64 KiB of output is waiting.
  2. read the output into your buffer. It returns 0 when none is waiting.
  3. Repeat until done().

An error stays: every later push returns it.

tape
var z: brotli.Decoder = brotli.decoder();
defer z.free();
var buf: [16384]u8;
while (!z.done()) {
    let piece: []const u8 = conn.read_some() or { return 1; };
    var at: i64 = 0;
    while (at < piece.len) {
        at = at + (z.push(piece[at..piece.len]) or { return 1; });
        var n: i64 = z.read(buf[0..buf.len]);
        while (n > 0) {
            out.write(buf[0..n]);
            n = z.read(buf[0..buf.len]);
        }
    }
}

To compress a stream:

tape
pub fn encoder(quality: i64) -> Encoder or BrotliError
pub fn Encoder.push(*self, data: []const u8)
pub fn Encoder.finish(*self)
pub fn Encoder.output(*const self) -> []const u8
pub fn Encoder.advance_output(*self, n: i64)
pub fn Encoder.free(*self)

An Encoder works like compress’s Deflater. push takes all the bytes you give it. Compressed bytes collect in output(): send them, then advance_output past what you sent. finish writes the end of the stream. Calling push after finish, finish twice, or advancing past output() traps.

tape
var e: brotli.Encoder = brotli.encoder(5) or { return 1; };
defer e.free();
while (true) {
    let piece: []const u8 = file.read_some() or { return 1; };
    if (piece.len == 0) { break; }
    e.push(piece);
    conn.write(e.output()) or { return 1; };
    e.advance_output(e.output().len);
}
e.finish();
conn.write(e.output()) or { return 1; };

An encoder holds about 2 MiB: its 256 KiB window, the window’s hash chains and one meta-block’s commands.

A decoder’s memory is set by the stream’s window. It holds a ring of twice the window, up to 32 MiB for the largest; the ring grows from 128 KiB as output comes, so a small response stays small. It also holds up to 64 KiB of unread output and the codes of the current block.

Last modified: