h264

tape
import h264 from "h264";

var p: h264.Parser = h264.parser();
defer p.free();
for (unit in access_units) {
    let pic: ?h264.Picture = p.access_unit(unit.bytes, unit.pts) or {
        io.println("cannot decode: {p.last_error()}");
        return 1;
    };
    if (pic == null) { continue; }            // parameter sets only
    decode_into(pic!.slot, &pic!);            // the backend's work
    for (k in 0..pic!.events.output_count) { show(pic!.events.output[k]); }
    for (k in 0..pic!.events.free_count) { forget(pic!.events.free[k]); }
}
let last: h264.Events = p.flush();            // the pictures still waiting

h264 reads an H.264 stream and gives, picture by picture, everything a decoder needs: the active parameter sets, each slice’s header, where its data starts, its reference lists resolved to picture slots, its prediction weights, the picture’s scaling matrices, and which pictures to show and in what order, for a GPU’s video engine (VA-API, the Intel engine, DXVA) to decode from. Its Decoder decodes the pixels itself, bit-exact. Both are pure (no file, no device) and sans-I/O: you hand them one access unit at a time.

Streams: Constrained Baseline, Main and High, 8-bit 4:2:0, frame pictures. Interlaced pictures (fields, MBAFF), slice groups (FMO), arbitrary slice order, 4:2:2, 4:4:4, monochrome, bit depths above 8 and Extended-profile features are refused with Unsupported, and last_error() says which ("interlaced (MBAFF)"). A malformed stream is Corrupt, with where. Nothing crashes on damaged data.

The decoder

tape
var d: h264.Decoder = h264.decoder();
defer d.free();
for (unit in access_units) {
    d.decode(unit.bytes, unit.pts) or {
        io.println("cannot decode: {d.last_error()}");
        continue;                             // a damaged picture: go on
    };
    while (true) {
        let f: ?h264.Frame = d.next_frame();
        if (f == null) { break; }
        show(f!.y, f!.cb, f!.cr, f!.y_stride, f!.c_stride, f!.crop);
        d.release(f!);
    }
}
d.flush();                                    // then next_frame once more
FunctionDoes
decoder() -> DecoderA decoder (Annex B input until configure_avcc)
d.configure_avcc(avcc)As the parser’s
d.decode(data, pts)Decode one access unit
d.next_frame() -> ?FrameThe next frame in display order, or null
d.release(f)That frame is no longer needed
d.flush()The end of the stream: the frames still waiting come out of next_frame
d.last_error(), d.free()As the parser’s

A Frame has the 8-bit y, cb and cr planes (4:2:0) with y_stride and c_stride, the decoded width and height (whole macroblocks), the display rectangle crop, the pts its access unit was given, and its colour description. It stays valid until release.

The decoder takes everything the parser does: CAVLC and CABAC, I, P and B pictures, weighted prediction, the 8x8 transform and scaling matrices (Constrained Baseline, Main and High). Its frames are bit-exact: their frame_hash is ffmpeg’s. It runs on one thread, at about 13 ms a 1080p frame and 6 ms a 720p one on a desktop processor; a hardware decoder with the parser is lighter still.

The parser

FunctionDoes
parser() -> ParserAn empty parser (Annex B input until configure_avcc)
p.configure_avcc(avcc)An MP4’s avcC record: its parameter sets, and length-prefixed units from now on
p.access_unit(data, pts) -> ?PictureOne access unit: parameter sets and one picture’s slices. null when it holds no picture
p.flush() -> EventsThe end of the stream (or a seek): every picture still waiting output, every slot freed
p.last_error() -> []const u8What the last error was about
p.free()Give its memory back

Slots. The parser decides where each picture lives: 17 slots, the 16 of the decoded picture buffer and the one being decoded. Picture.slot is free when the picture arrives; decode into it, then apply Picture.events: show output in order, then forget the slots in free. Everything a call returns (the picture, its slices, the nal views into your data) stays valid until the next call.

What a picture gives

Field
slot, sps, ppsWhere to decode, the active parameter sets (every field as coded)
width_mbs, height_mbs, cropThe size in macroblocks and the display rectangle in pixels
idr, reference, long_term, frame_num, poc, top_poc, bottom_poc
refs, ref_countThe reference frames in the buffer (a hardware API’s reference list)
scaling4, scaling8The final scaling matrices, fall-back rules applied
colourThe colour description from the VUI (primaries, transfer, matrix, range)
slicesEach slice: nal (raw bytes), header, rbsp_bit_offset, data_bit_offset, list0/list1 (slots), weights
eventsThe slots to show and to free after this picture

Both offsets say where slice_data() starts, in bits from the NAL unit’s first byte. rbsp_bit_offset counts the bits with emulation prevention bytes removed; this is the one VA-API (slice_data_bit_offset) and Intel’s video engine take, even though they read the escaped bytes. data_bit_offset counts them in the raw bytes, emulation prevention bytes included. The two differ only when a slice header holds a 00 00 03. A CABAC slice’s alignment bits come after both.

The frame hash

frame_hash(y, cb, cr, y_stride, c_stride, crop) -> string is the SHA-256 of a frame cropped to crop as 8-bit I420 (Y, then Cb, then Cr), in hex: the line ffmpeg -f framehash -hash sha256 -pix_fmt yuv420p prints for a frame, so any decoder’s frames can be compared with another’s.

Last modified: