Vectors
A vector holds 16 bytes of integer lanes and works on all of them at once.
Nothing is vectorized for you: a loop over u32 stays a loop over u32,
a loop over u32x4 handles four at a time.
| Type | Lanes |
|---|---|
u8x16, i8x16 | 16 of u8 / i8 |
u16x8, i16x8 | 8 of u16 / i16 |
u32x4, i32x4 | 4 of u32 / i32 |
u64x2, i64x2 | 2 of u64 / i64 |
Making one
let a: u32x4 = [1, 2, 3, 4]; // one element per lane, lane 0 first
let b: u32x4 = @splat(u32x4, 7); // every lane 7
var px: [8]u32;
let vp: *u32x4 = (&px[0]) as *u32x4; // a pointer to vectors
let c: u32x4 = *vp; // px[0..4]
let d: u32x4 = vp[1]; // px[4..8]Vectors live in locals and expressions and behind pointers. They are not
struct fields, array elements, parameters, results or globals: pass a
*u32x4 instead.
Operations
let s: u32x4 = a + b; // lane by lane, wrapping
let m: u32x4 = (a * b) >> 3; // shifts take one scalar count
let x: u8x16 = ~(s as u8x16); // `as` reinterprets the same 128 bits
var v: u32x4 = s;
v[2] = 9; // one lane
let lane: u32 = v[2];
*vp = v; // back to memory+ - * & | ^ ~and unary-work lane by lane and wrap like integers;*is on 16- and 32-bit lanes.<<and>>shift every lane by a scalar;>>is arithmetic on signed lanes. A count at or past the lane width gives 0 (or the sign).- Both operands have the same vector type: use
@splatfor a scalar. - A lane index outside the vector traps, like an array index.
Builtins
| Builtin | Result |
|---|---|
@splat(T, x) | every lane x |
@widen_lo(v), @widen_hi(v) | the low or high half of the lanes at twice the width (sign-extended for signed lanes) |
@narrow(a, b) | the lanes of a then b at half the width, saturated |
@narrow(T, a, b) | signed lanes (i16x8, i32x4) to unsigned T (u8x16, u16x8), clamped to 0 .. max |
@eq(a, b), @lt(a, b), @gt(a, b) | a mask: all ones in each lane where it holds, else 0 |
@min(a, b), @max(a, b) | in each lane the smaller / larger (8-, 16- and 32-bit lanes) |
@min_lanes(v), @max_lanes(v) | the smallest / largest lane, as the lane type (8-, 16- and 32-bit lanes) |
@any(v), @all(v) | bool: some lane / every lane is not 0 |
== and < are not defined on vectors; compare with @eq and @lt.
Unsigned lanes compare unsigned, in @lt, @min and @max_lanes alike.
// The largest of n i32, four at a time (n a multiple of 4, at least 4)
fn max_of(p: *i32, n: i64) -> i32 {
var m: i32x4 = *(p as *i32x4);
var i: i64 = 4;
while (i < n) {
m = @max(m, *((&p[i]) as *i32x4));
i = i + 4;
}
return @max_lanes(m); // the four running maxima down to one
}Example
// Four pixels at a time: every 8-bit channel scaled by a / 256
fn scale(dst: *u32, n: i64, a: u16) {
let av: u16x8 = @splat(u16x8, a);
var i: i64 = 0;
while (i + 4 <= n) {
let p: *u32x4 = (&dst[i]) as *u32x4;
let s: u8x16 = *p as u8x16;
let lo: u16x8 = (@widen_lo(s) * av) >> 8;
let hi: u16x8 = (@widen_hi(s) * av) >> 8;
*p = @narrow(lo, hi) as u32x4;
i = i + 4;
}
}The VM, x86-64 (SSE2) and arm64 (NEON) give the same results bit for bit.
Choosing without a branch
@select(c, a, b) is a when c is true, else b, for vectors as for
integers, bools and enums: both are evaluated and the choice is mask
arithmetic, never a branch (what constant-time code needs, see
Attributes).
let r: u32x4 = @select(swap, b, a);Processor features
SSE2 and NEON are always there. Some instructions are not: @cpu_has(f) asks
the processor running the program, with f one of aes, clmul, sha,
sse41, sse42, avx2, rdrand, rdseed (x86-64), neon (arm64) and
crc32 (rdrand and rdseed are for the #bytecode instructions). The answer is
read once at start-up (cpuid on x86-64, getauxval on Linux arm64,
sysctlbyname on macOS arm64; the VM asks the same way), so asking in a
loop is cheap.
if (@cpu_has(aes)) {
encrypt_hw(keys, block);
} else {
encrypt_portable(keys, block);
}AES and carry-less multiply
| Builtin | Result |
|---|---|
@aes_enc(s, k) | one AES round: ShiftRows, SubBytes, MixColumns, then ^ k |
@aes_enc_last(s, k) | the last round: ShiftRows, SubBytes, ^ k |
@aes_dec(s, k) | one decryption round (equivalent inverse cipher): InvShiftRows, InvSubBytes, InvMixColumns, ^ k |
@aes_dec_last(s, k) | InvShiftRows, InvSubBytes, ^ k |
@aes_imc(k) | InvMixColumns: an encryption round key made a decryption one |
@aes_keygen_assist(k, rcon) | the key-schedule helper: SubWord and RotWord of words 1 and 3, rcon (a literal 0..255) XORed into words 1 and 3 |
@clmul_lo(a, b), @clmul_hi(a, b) | the 128-bit carry-less product of lane 0 (or lane 1) of two u64x2 |
The AES builtins take and give u8x16 (byte i is row i % 4, column
i / 4, as in FIPS-197); the carry-less multiplies u64x2. They behave
exactly like x86-64’s AES-NI and PCLMULQDQ on both x86-64 and arm64 (arm64’s
aese adds the round key first; the compiler arranges the difference), and
the VM computes the same bits in portable code.
// AES-128: rk holds the 11 round keys
fn encrypt(rk: *u8x16, p: *u8x16, c: *u8x16) {
var s: u8x16 = *p ^ rk[0];
for (i in 1..10) { s = @aes_enc(s, rk[i]); }
*c = @aes_enc_last(s, rk[10]);
}On a processor without AES (or CLMUL for @clmul_*) a builtin stops the
program with trap: CPU lacks feature aes at file:line:col (exit code 101),
in tape run and in a native build alike. Inside if (@cpu_has(aes)) { ... }
the compiler leaves that check out, since it cannot fail there.
Last modified: