mixer

tape
import mixer from "mixer";

var m = mixer.new(mixer.Config { rate: 48000; channels: 2; });
defer m.free();

let sfx = m.add_bus("sfx");
let shot = m.add_sound(mixer.pcm_i16(sound.rate, sound.channels, sound.samples));

let v = m.play(shot, mixer.Play { bus: sfx; gain: 0.8; pan: -0.3; pitch: 1.2; });

var block: [1024]f32;
m.render_f32(block[0..]);                  // 512 frames, the sum of everything playing
stream.write_f32(block[0..]) or { return 1; };

mixer turns many sounds into one. You give it decoded samples, tell it what to play, and ask it for the next samples whenever your output wants some. It does not touch a device, read a file or start a thread: you own the output (audio.Stream, a file, anything), the decoding (wav, vorbis) and the clock. The same calls give the same samples in the VM and natively, on every platform, at any block size.

Everything you change takes effect without a click: gains, pans and pitches ramp over 5 ms, a voice’s start over 1 ms, its end and a stop over 5 ms.

It is a pure module. audio plays what the platform mixes; mixer is for a program that wants more than audio.play gives: dozens of voices, pitch, buses, distance, loops at sample positions.

A whole program

A WAV file for a sound effect and an Ogg file for the music, played for ten seconds through an audio.Stream:

tape
@profile(t1);
import audio from "audio";
import mixer from "mixer";
import vorbis from "vorbis";
import fs from "fs";
import io from "io";
import time from "time";

const RATE: i64 = 48000;

pub fn main() -> i32 {
    // The sound effect: a WAV file, decoded whole, 16-bit
    let shot_file: string = fs.read_file("shot.wav") or {
        io.println("no shot.wav: {err}");
        return 1;
    };
    let shot_pcm: audio.Sound = audio.decode_wav(shot_file) or {
        io.println("shot.wav: {err}");
        return 1;
    };
    defer audio.free_wav(shot_pcm);

    // The music: an Ogg file, decoded as it plays
    let song_file: string = fs.read_file("music.ogg") or {
        io.println("no music.ogg: {err}");
        return 1;
    };
    var song: vorbis.OggReader = vorbis.open_ogg(song_file) or {
        io.println("music.ogg: {err}");
        return 1;
    };
    defer song.free();
    let ch: i64 = song.info().channels;

    // The output is the program's: an audio.Stream of f32
    var out: audio.Stream = audio.open(audio.Format {
        rate: RATE; channels: 2; sample: audio.Sample.F32;
    }, 60) or {
        io.println("no sound: {err}");
        return 1;
    };
    defer out.close();

    var m = mixer.new(mixer.Config { rate: RATE; channels: 2; });
    defer m.free();
    let sfx = m.add_bus("sfx");
    let tunes = m.add_bus("music");
    let shot = m.add_sound(mixer.pcm_i16(shot_pcm.rate, shot_pcm.channels, shot_pcm.samples));
    let music = m.add_stream(song.info().rate, ch, 16384);

    var decoded: [8192]f32;
    var block: [1920]f32;
    var started: bool = false;
    var next_shot: i64 = RATE / 2;
    var duck_ends: i64 = 0;
    var pan: f32 = -1.0;
    m.play(music, mixer.Play { bus: tunes; gain: 0.7; fade_in_ms: 500.0; });

    while (m.position() < RATE * 10) {
        // Keep the ring full, looping the song
        while (m.stream_free(music) >= 4096 / ch) {
            let n: i64 = song.read(decoded[0..4096]);
            if (n == 0) {
                song.seek(0);
                continue;
            }
            m.stream_write_f32(music, decoded[0..n * ch]);
        }

        // A shot a second, on the frame, moving across; the music ducks under it
        if (m.position() + 960 >= next_shot) {
            m.play(shot, mixer.Play { bus: sfx; gain: 0.8; pan: pan; delay: next_shot - m.position(); });
            m.bus_fade(tunes, 0.5, 100.0);
            duck_ends = next_shot + RATE / 4;
            next_shot = next_shot + RATE;
            pan = pan + 0.2;
        }
        if (duck_ends != 0 && m.position() >= duck_ends) {
            m.bus_fade(tunes, 1.0, 400.0);
            duck_ends = 0;
        }

        // Make what the device can take now
        var frames: i64 = out.free_frames() or { return 1; };
        if (frames > 960) { frames = 960; }
        if (frames == 0) {
            if (!started) {
                out.start() or { return 1; };
                started = true;
            }
            time.sleep(time.millis(2));
            continue;
        }
        m.render_f32(block[0..frames * 2]);
        out.write_f32(block[0..frames * 2]) or { return 1; };
    }
    out.drain() or { return 1; };
    io.println("underruns: {m.stream_underruns(music)}");
    return 0;
}

The loop has three jobs: keep the music’s ring full from the decoder, start voices, and render what the output can take. Nothing is called back and nothing runs behind your back. A file of more than two channels traps in add_stream: mix it down first.

A mixer

tape
pub struct Config {
    rate: i64;                      // frames a second of the output
    channels: i64;                  // 1 or 2, interleaved left then right
    voices: i64 = 64;               // voices playing at once, at most
    ramp_ms: f32 = 5.0;             // every abrupt change takes this long
    attack_ms: f32 = 1.0;           // a voice's start takes this long
    limiter: bool = true;           // soft clip on the master
}

pub fn new(cfg: Config) -> Mixer
pub fn Mixer.free(*self)
pub fn Mixer.position(*const self) -> i64
pub fn Mixer.stats(*const self) -> Stats

pub struct Stats { playing: i64; tails: i64; stolen: i64; refused: i64; frames: i64; }
pub const MAX_VOICES: i64 = 4096;
pub const CONTROL_FRAMES: i64 = 32;

new allocates the voices and traps if there is no memory, as a container does. free releases everything the mixer allocated, not your samples. position counts the frames rendered so far. stats says how many voices count against the limit (playing), how many are ending in their last ramp (tails), and how many were stolen or refused.

Changes are read every CONTROL_FRAMES frames, so a set_gain starts at most 32 frames later. A voice’s start and Play.delay are exact to the frame.

tape
var m = mixer.new(mixer.Config { rate: 48000; channels: 2; voices: 128; attack_ms: 2.0; });
defer m.free();
let s: mixer.Stats = m.stats();
io.println("{s.playing} voices, {s.stolen} stolen, at frame {m.position()}");

new traps when rate isn’t positive, channels isn’t 1 or 2, voices is outside 1..MAX_VOICES, or ramp_ms or attack_ms is outside 0.5 to 100.

Rendering

tape
pub fn Mixer.render_f32(*self, out: []f32)
pub fn Mixer.render_i16(*self, out: []i16)

Each writes out.len / channels frames, interleaved, any number of them. render_f32 is full scale at ±1.0. render_i16 is each of those samples times 32767, rounded to nearest and clamped, with no dither. Neither allocates. They trap if out isn’t a whole number of frames.

tape
var out32: [2048]f32;
m.render_f32(out32[0..]);
var out16: [2048]i16;
m.render_i16(out16[0..]);

Splitting a render in two changes nothing: 1000 frames at once are the same bits as 300 and then 700.

The limiter is on unless limiter: false. Below 0.8 a sample passes untouched. Above it the sample bends towards full scale and doesn’t pass it, smoothly, so a crowd of voices gets rounder rather than crackling. With it off, render_f32 gives the plain sum, which may go past ±1.0.

Sounds

tape
pub struct Pcm { rate: i64; channels: i64; is_f32: bool; i16s: []const i16; f32s: []const f32; }
pub struct Source { id: i64 = -1; gen: u32 = 0; }

pub fn pcm_i16(rate: i64, channels: i64, samples: []const i16) -> Pcm
pub fn pcm_f32(rate: i64, channels: i64, samples: []const f32) -> Pcm
pub fn Mixer.add_sound(*self, pcm: Pcm) -> Source
pub fn Mixer.remove_source(*self, h: Source)

pcm_i16 and pcm_f32 describe decoded samples, mono or stereo, at any rate. add_sound makes a Source of them: a view, not a copy, shared by any number of voices. Your samples must stay until remove_source or free.

tape
let a: audio.Sound = audio.decode_wav(wav_bytes) or { return 1; };
let shot = m.add_sound(mixer.pcm_i16(a.rate, a.channels, a.samples));

var o: vorbis.Pcm = vorbis.decode_ogg(ogg_bytes) or { return 1; };
let song = m.add_sound(mixer.pcm_f32(o.rate, o.channels, o.samples));

mixer takes rate, channels and samples rather than an audio.Sound because audio touches the system and mixer doesn’t. More than two channels traps. So does a length that isn’t a whole number of frames, or a rate that isn’t positive.

remove_source ends the source’s voices. Each holds its last frame through the ramp without reading the samples again, so you may free them straight after.

tape
m.remove_source(shot);
audio.free_wav(a);

Streams

For music, too long to decode whole: a ring you refill as you decode. No callback; you look at how much room there is whenever it suits you.

tape
pub fn Mixer.add_stream(*self, rate: i64, channels: i64, frames: i64) -> Source
pub fn Mixer.stream_free(*const self, h: Source) -> i64
pub fn Mixer.stream_queued(*const self, h: Source) -> i64
pub fn Mixer.stream_write_f32(*self, h: Source, samples: []const f32) -> i64
pub fn Mixer.stream_write_i16(*self, h: Source, samples: []const i16) -> i64
pub fn Mixer.stream_end(*self, h: Source)
pub fn Mixer.stream_flush(*self, h: Source)
pub fn Mixer.stream_underruns(*const self, h: Source) -> i64
pub fn Mixer.stream_underrun_frames(*const self, h: Source) -> i64

add_stream allocates a ring of at least frames frames, at the rate and channels you will write. stream_write_f32 and stream_write_i16 take as many whole frames as fit and say how many, never waiting. stream_free is the room left, stream_queued what is waiting to play.

tape
let music = m.add_stream(song.info().rate, song.info().channels, 16384);
m.play(music, mixer.Play { gain: 0.7; });
while (m.stream_free(music) >= 2048) {
    let n: i64 = song.read(decoded[0..4096]);
    if (n == 0) { song.seek(0); continue; }
    m.stream_write_f32(music, decoded[0..n * 2]);
}

Running dry isn’t an error. The voice holds its last frame, ramps out, plays silence and counts one underrun; when you write again it ramps back in where it left off. It never plays old frames again. stream_underruns counts the times, stream_underrun_frames how long they lasted. Keep a little queued: a voice needs the next frame to read between two, so a ring you fill with exactly what was played runs dry.

tape
if (m.stream_underruns(music) > 0) { io.println("the decoder fell behind"); }

stream_end says no more is coming: the voice plays out what is queued and ends, and that isn’t an underrun. A write before then takes it back. stream_flush drops what is queued, for a seek in the decoder: the voice ramps out and back in on whatever you write next.

tape
song.seek(frame);
m.stream_flush(music);
m.stream_end(music);

A stream plays on one voice at a time: a second play takes it from the first, which ends on its last frame. It doesn’t loop (seek your decoder), and can’t start late or seek.

Voices

tape
pub struct Voice { slot: i64 = -1; gen: u32 = 0; }
pub struct Play {
    bus: ?Bus = null;               // null: the master
    gain: f32 = 1.0;                // linear, at least 0
    pan: f32 = 0.0;                 // -1 left, 0 centre, 1 right
    pitch: f32 = 1.0;               // 2.0 an octave up and twice as fast
    loop: bool = false;
    loop_start: i64 = 0;            // source frames
    loop_end: i64 = 0;              // 0: the source's end
    start: i64 = 0;                 // source frame to start from
    delay: i64 = 0;                 // output frames before the first
    fade_in_ms: f32 = 0.0;
    fade_in_ease: easing.Ease = easing.Ease.Linear;
    priority: i64 = 0;              // higher is stolen later
    interp: Interp = Interp.Linear;
    spatial: ?Spatial = null;
}

pub const MIN_PITCH: f32 = 0.001;
pub const MAX_PITCH: f32 = 1000.0;

pub fn Mixer.play(*self, h: Source, p: Play) -> Voice
pub fn Mixer.is_playing(*const self, h: Voice) -> bool
pub fn Mixer.voice_frame(*const self, h: Voice) -> ?i64

play starts a voice and gives a handle. A handle whose voice has ended is stale, and every call ignores it, even after the voice’s slot has been used again: it never reaches another voice. You needn’t keep a handle you don’t need, or check one before using it.

tape
m.play(shot, mixer.Play { pan: 0.4; });      // fire and forget
let engine = m.play(hum, mixer.Play { loop: true; loop_start: 4410; gain: 0.5; });

A source’s own rate doesn’t matter: voices are resampled to the output’s. delay starts the voice that many output frames from now, exact to the frame, so rhythm isn’t quantized to your block:

tape
let beat: i64 = 48000 * 60 / 120;            // 120 bpm at 48 kHz
for (i in 0..8) { m.play(kick, mixer.Play { delay: i * beat; }); }

is_playing is true while the voice lives: playing, paused, waiting for its delay, or in its last ramp. voice_frame is the source frame it has got to, null for a stale handle; for a stream, the frames read or flushed since it began.

tape
if (m.is_playing(engine)) {
    let at: i64 = m.voice_frame(engine) or { 0 };
    io.println("at frame {at}");
}

play gives a stale handle for a stale source, an empty sound, and a play that couldn’t get a voice (see below). It traps on a gain below 0, a pan outside -1..1, a pitch outside MIN_PITCH..MAX_PITCH, a negative delay or start, a start past the end, loop points outside the sound, a loop or start on a stream, a bus of another mixer, and Spatial settings out of range.

Changing a voice

tape
pub fn Mixer.set_gain(*self, h: Voice, gain: f32)
pub fn Mixer.set_pan(*self, h: Voice, pan: f32)
pub fn Mixer.set_pitch(*self, h: Voice, pitch: f32)
pub fn Mixer.set_loop(*self, h: Voice, on: bool)
pub fn Mixer.set_loop_points(*self, h: Voice, start: i64, end: i64)
pub fn Mixer.pause(*self, h: Voice)
pub fn Mixer.resume(*self, h: Voice)
pub fn Mixer.seek(*self, h: Voice, frame: i64)

Each new gain, pan or pitch is reached within the ramp. Gain is linear, not decibels, and may go above 1. Pan is equal power: a mono voice is cos and sin of (pan + 1) * pi / 4 left and right, so −3 dB each in the middle and as loud all the way across. A stereo voice is the same curve scaled to 1 in the middle: unity at 0, and the far side fades away. A mono output ignores pan. Pitch changes speed too, as a tape does.

tape
m.set_gain(engine, 0.3);
m.set_pan(engine, -0.5);
m.set_pitch(engine, 1.0 + speed * 0.5);

set_loop(v, false) lets a loop run out to the sound’s end. set_loop_points moves the loop; an end of 0 is the sound’s end. If either leaves the voice past the loop’s end, it ramps out, goes into the loop, and ramps in, as seek does.

tape
m.set_loop_points(engine, 4410, 48510);
m.set_loop(engine, false);                   // finish the tail, and end

pause ramps out and holds the voice where it is; resume ramps back in from the same frame. A paused voice’s delay waits too. seek is a ramp out, the jump, and a ramp in. Seeking past the end ends a voice.

tape
m.pause(engine);
m.resume(engine);
m.seek(engine, 0);

Stopping and fading

tape
pub fn Mixer.stop(*self, h: Voice)
pub fn Mixer.stop_fade(*self, h: Voice, ms: f32, ease: easing.Ease = easing.Ease.Linear)
pub fn Mixer.fade(*self, h: Voice, to: f32, ms: f32, ease: easing.Ease = easing.Ease.Linear)

stop ends the voice within the ramp. stop_fade takes ms and ends it, along any easing curve. fade moves the voice’s fade level to to and leaves it there, playing on: the level multiplies gain, 0 silent, 1 as gain says. A fade is never shorter than the ramp. Play.fade_in_ms starts a voice with one.

tape
let song_v = m.play(song, mixer.Play { fade_in_ms: 2000.0; fade_in_ease: easing.Ease.SineOut; });
m.fade(song_v, 0.3, 500.0);
m.stop_fade(song_v, 3000.0, easing.Ease.QuadOut);
m.stop(song_v);

A curve that overshoots (BackOut) can’t take a gain below 0.

Better interpolation

tape
pub enum Interp : u8 { Linear = 0; Hermite = 1; }

Linear reads between two samples: the cheapest, and what you want for most effects. It dulls the top a little when a sound is pitched down far, and lets some alias through when pitched up. Hermite is Catmull-Rom over four samples: crisper at about four times the cost. Choose it per voice.

tape
m.play(strings, mixer.Play { pitch: 0.5; interp: mixer.Interp.Hermite; });

When every voice is taken

voices is the most playing at once. A play beyond that takes the place of another voice: the one with the lowest priority not above the new play’s, then the quietest (its gain now, with its bus’s; a paused one counts as silent), then the oldest. The stolen voice fades over the ramp beside the new one, or goes at once if it hasn’t made a sound yet. If every voice has a higher priority the play is refused and gives a stale handle; Stats counts both. So is a burst that outruns the room for those fading voices, one more voices of them.

tape
let boss = m.play(roar, mixer.Play { priority: 10; });
m.play(step, mixer.Play { priority: 0; gain: 0.2; });

Buses

tape
pub struct Bus { id: i64 = 0; }
pub const MAX_NAME: i64 = 24;

pub fn Mixer.add_bus(*self, name: []const u8) -> Bus
pub fn Mixer.find_bus(*const self, name: []const u8) -> ?Bus
pub fn Mixer.master(*const self) -> Bus
pub fn Mixer.bus_set_gain(*self, b: Bus, gain: f32)
pub fn Mixer.bus_set_mute(*self, b: Bus, on: bool)
pub fn Mixer.bus_pause(*self, b: Bus)
pub fn Mixer.bus_resume(*self, b: Bus)
pub fn Mixer.bus_fade(*self, b: Bus, to: f32, ms: f32, ease: easing.Ease = easing.Ease.Linear)

A bus is a group of voices with one level, so the sound effects and the music have a slider each. There is one level of them: every bus feeds the master, master gives the master itself, and a voice with no bus plays straight on it. Every call also works on the master, which is called "master". A name is at most MAX_NAME bytes.

tape
let sfx = m.add_bus("sfx");
let music = m.add_bus("music");
m.bus_set_gain(music, music_volume);
let again: ?mixer.Bus = m.find_bus("sfx");

bus_set_mute silences a bus and lets its voices go on. bus_pause ramps it out and holds its voices and their delays where they are: for a pause menu, a paused master holds everything. bus_fade moves a level multiplying the bus’s gain and leaves it there: ducking.

tape
m.bus_pause(sfx);
m.bus_resume(sfx);
m.bus_set_mute(music, true);
m.bus_fade(music, 0.3, 200.0);               // duck under the dialogue
m.bus_fade(music, 1.0, 800.0, easing.Ease.SineInOut);
m.bus_set_gain(m.master(), 0.8);

Distance

tape
pub struct Spatial {
    pos: math3d.Vec3;
    falloff: Falloff = Falloff.Inverse;
    min: f32 = 1.0;                 // nearer than this: full gain
    max: f32 = 100.0;               // further than this: as at max
    rolloff: f32 = 1.0;
    width: f32 = 1.0;               // how far to the sides it pans: 0 centre, 1 hard
}
pub enum Falloff : u8 { None = 0; Inverse = 1; Linear = 2; }
pub struct Listener { pos: math3d.Vec3; forward: math3d.Vec3; up: math3d.Vec3; }

pub fn Mixer.set_listener(*self, l: Listener)
pub fn Mixer.set_position(*self, h: Voice, pos: math3d.Vec3)
pub fn Mixer.set_spatial(*self, h: Voice, sp: Spatial)

A voice with Play.spatial is placed in the world. Distance d to the listener, held between min and max, gives a gain multiplying its own:

  • Inverse: min / (min + rolloff * (d - min))
  • Linear: 1 - rolloff * (d - min) / (max - min), at least 0
  • None: 1

Its direction gives a pan, added to its own: how far to the listener’s right it is, from −1 to 1, times width. Nearer than min it slides to the middle, so a sound passing through your head is smooth. A stereo pair can’t tell front from behind, and this doesn’t try.

tape
let torch = m.play(crackle, mixer.Play { loop: true; spatial: mixer.Spatial {
    pos: math3d.Vec3 { x: 4.0; y: 0.0; z: -10.0; };
    min: 2.0; max: 50.0;
}; });
m.set_position(torch, torch_pos);
m.set_spatial(torch, mixer.Spatial { pos: torch_pos; falloff: mixer.Falloff.Linear; min: 1.0; max: 20.0; });
m.set_listener(mixer.Listener { pos: eye; forward: look; up: math3d.Vec3 { x: 0.0; y: 1.0; z: 0.0; }; });

Call set_listener and set_position as often as you like; each reaches its new gain and pan within the ramp. A new mixer’s listener stands at the origin looking down −z with y up. A forward parallel to up keeps the old right ear. Positions are math3d vectors in your world’s units; min and max are in those too. set_position ignores a voice that isn’t spatial; set_spatial makes one so.

Threads

A Mixer isn’t thread-safe. It belongs to one thread, which makes all the calls. Two ways to arrange that:

In the frame loop (the program above): render what out.free_frames() says each time round. Give the stream a buffer longer than your longest frame.

In a thread of its own, taking commands as messages. Its state is the mixer. One pool thread keeps the state in one place, and it runs one task at a time, so a pump that returns after a block can’t starve the commands. The thread opens the stream too, since a stream belongs to the thread that opened it, and decodes into memory it keeps, since a sound must stay valid while it plays: a command’s arguments are copied for the task, and a readonly slice is frozen only for that task, so neither can be what a sound views.

tape
thread sound {
    var m: mixer.Mixer = mixer.new(mixer.Config { rate: 48000; channels: 2; });
    var out: ?audio.Stream = null;
    var shot: mixer.Source = mixer.Source {};
    var block: [1920]f32;

    fn start(wav_bytes: []const u8) {
        let pcm: audio.Sound = audio.decode_wav(wav_bytes) or { return; };
        shot = m.add_sound(mixer.pcm_i16(pcm.rate, pcm.channels, pcm.samples));
        let s: audio.Stream = audio.open(audio.Format {
            rate: 48000; channels: 2; sample: audio.Sample.F32;
        }, 60) or { return; };
        s.start() or { return; };
        out = s;
    }
    fn shoot(gain: f32, pan: f32) {
        m.play(shot, mixer.Play { gain: gain; pan: pan; });
    }
    fn pump() {
        if (out == null) { return; }
        let s: audio.Stream = out!;
        var frames: i64 = s.free_frames() or { return; };
        if (frames > 960) { frames = 960; }
        m.render_f32(block[0..frames * 2]);
        s.write_f32(block[0..frames * 2]) or { return; };
    }
}

fn nothing() {}

// once, from the main thread:
sound.start(file) { on_done: nothing };

// each frame, from the main thread:
sound.shoot(1.0, 0.2) { on_done: nothing };
sound.pump() { on_done: nothing };
threads.poll();

What it costs

Measured on an Intel Core i9-14900K: 64 voices at 48 kHz stereo with pitch and pan, in blocks of 512 frames, as a share of one core of real time (stdlib/tests/mixer_bench.tape):

Voices ofNativeVM
mono 16-bit, Linear0.9 %41 %
stereo 16-bit, Linear1.2 %60 %
stereo f32, Linear1.1 %52 %
mono 16-bit, Hermite3.1 %134 %
stereo 16-bit, Hermite5.6 %246 %
stereo 16-bit, gain, pan and pitch changing every block1.4 %64 %

render_f32 and render_i16 allocate nothing. In the VM, plan on a few dozen Linear voices.

Hearing it

stdlib/tests/mixer_demo.tape is a small app for trying all of this by ear. Build it natively (the VM runs it too, at many times the cost, and shows on screen if the sound falls behind):

plaintext
tape build stdlib/tests/mixer_demo.tape -o mixer_demo.exe -I stdlib
./mixer_demo.exe

Each page says what a feature is, what you should hear step by step, and what would be wrong. The step playing now is lit. Where it can, a switch plays a clearly labelled bad version live, so you can hear what a click or a hard clip sounds like. Everything has sliders. Your answers are kept in mixer_demo_results.txt in the current folder. Left and Right change page, Space plays and stops, A and B switch the comparison.

Not here

Effects (reverb, filters): each bus renders to a buffer of its own, so one can go on it later. Doppler. Decoding: wav, vorbis, aac. Devices: audio.

Last modified: