http

tape
import http from "http";
import io from "io";

var opts: http.Options = http.options();
opts.max_redirects = 5;
let r: http.Response = http.get("https://example.com/", opts) or {
    io.eprintln("fetch failed: {err.desc()}");
    return 1;
};
io.println("{r.status} {r.reason()}");
let ct: ?[]const u8 = r.header("Content-Type");
if (ct != null) { io.println("type {ct!}"); }
io.print(r.body);

http is an HTTP client over net. One call sends one request on a fresh HTTP/1.1 connection (Connection: close) and reads the whole response into memory: the status, the head as received and the body. It handles chunked transfer coding, Content-Length, bodies that end with the connection, interim 1xx responses, timeouts, size limits and, when you ask for them, redirects. A response with any status is a response: a 404 is not an error.

https:// URLs go through TLS 1.3 in net (net.tls_wrap): the server’s certificate is checked against tape’s bundled roots, or the trust store in Options.trust, for the URL’s host, and ALPN offers http/1.1. There is no option to skip the check: for a server with a private certificate, trust its root (below).

tape
// A server whose certificate comes from your own CA
var t: net.Trust = net.trust_from_pem(fs.read_file("corp-root.pem") or { return 1; }) or { return 1; };
defer t.free();
var opts: http.Options = http.options();
opts.trust = t.store();
let r: http.Response = http.get("https://intranet.example/status", opts) or { return 1; };

A Client keeps connections for the next requests, runs many requests at once, and speaks HTTP/2 to https servers that offer it.

Not covered (yet): streaming a body to a writer, HTTP/3, compression (Accept-Encoding: identity is sent), cookies, proxies, Expect: 100-continue, multipart forms, a server.

Requests

FunctionDoes
get(url, opts) -> ResponseA GET of url
request(req, opts) -> ResponseSend req and read the whole response
options() -> OptionsThe defaults, to change field by field
parse_url(text) -> UrlThe parts of an http or https URL
tape
pub enum Method : u8 { Get = 1; Head = 2; Post = 3; Put = 4; Delete = 5; Patch = 6; Options = 7; }   // m.desc() is "GET", ...
pub struct Field { name: []const u8; value: []const u8; }
pub struct Request {
    method: Method;
    url: []const u8;
    headers: []const Field;        // extra request headers
    body: []const u8;              // sent with Content-Length when not empty
}
tape
let body: []const u8 = "\{\"name\": \"tape\"}";
let r: http.Response = http.request(http.Request {
    method: http.Method.Post;
    url: "http://127.0.0.1:8080/items";
    headers: [http.Field { name: "Content-Type"; value: "application/json"; }];
    body: body;
}, http.options()) or { return 1; };

The request sent is the request line (POST /items HTTP/1.1), then Host (with the port when it isn’t the scheme’s default, an IPv6 address in brackets), User-Agent: tape-http, Accept-Encoding: identity, Connection: close, your headers in order, Content-Length when there is a body (and Content-Length: 0 for a POST, PUT or PATCH without one), an empty line and the body. A User-Agent or Accept-Encoding of yours replaces the client’s.

Some header mistakes are bugs in the calling program, so they trap instead of returning an error: a header named Host, Content-Length, Connection or Transfer-Encoding (the client sends those itself), a name that isn’t an HTTP token, a value containing CR, LF or NUL (header injection), and a negative limit in Options.

Clients

get and request open a connection for each request and close it after the response. A program that makes many requests (a browser loading a page’s scripts and images) keeps a Client instead. It keeps connections for the next request to the same scheme, host and port, which saves the connecting and, for https, the TLS handshake; it has many requests in flight at once; and over https it offers HTTP/2, so a server that picks it carries every request as a stream of one connection.

tape
var c: http.Client = http.client();
defer c.close();
let a: http.Response = c.get("https://example.com/a.png", http.options()) or { return 1; };
let b: http.Response = c.get("https://example.com/b.png", http.options()) or { return 1; };   // the same connection
FunctionWhat it does
client() -> ClientA client with no connections yet
c.get(url, opts) -> Responseget through the client’s connections
c.request(req, opts) -> Responserequest through the client’s connections
c.start(req, opts) -> i64Queue a request and return its id; it is sent at the next wait (or get, request, get_all)
c.wait() -> ?DoneThe next request to finish, in any order; null when none is left
c.get_all(urls, opts) -> Vec_DoneA GET of every URL at once; the results in the order of urls
c.get_all_strings(urls, opts) -> Vec_DoneThe same over []const string (a Vec_string’s slice())
c.close()Closes every connection, drops what is in flight
tape
pub struct Done {
    id: i64;                       // what start returned
    response: Response;
    error: ?HttpError;
}
// d.result() -> Response or HttpError

Each request finishes on its own: one that fails (a refused connection, a timeout) doesn’t stop the others, and a 404 is a response like any.

tape
let urls: [3][]const u8 = ["https://example.com/a.js", "https://example.com/b.js", "https://cdn.example/c.css"];
var all: http.Vec_Done = c.get_all(urls[..], http.options());
for (i in 0..all.len()) {
    let r: http.Response = all.get_ref(i).result() or {
        io.eprintln("{urls[i]}: {err.desc()}");
        continue;
    };
    io.println("{urls[i]}: {r.status}, {r.body.len} bytes");
}
tape
let first: i64 = c.start(http.Request { method: http.Method.Get; url: "https://example.com/"; headers: []; body: ""; }, opts);
let second: i64 = c.start(http.Request { method: http.Method.Post; url: "https://example.com/log"; headers: []; body: "hit"; }, opts);
while (true) {
    let d: ?http.Done = c.wait();
    if (d == null) { break; }
    if (d!.id == first) { ... }
}

How requests go out:

  • HTTP/2. For https, TLS offers h2 then http/1.1 (ALPN). When the server picks h2, one connection to it carries every request as a stream, as many at once as the server allows (its MAX_CONCURRENT_STREAMS; the rest wait for a stream to finish). Headers are compressed with HPACK; bodies flow within each side’s windows. c.http2 = false turns it off.
  • HTTP/1.1. Otherwise each connection carries one request at a time, up to c.per_host connections to one server at once (6, as browsers do); more requests wait for one of them. A connection is kept for the next request when the response is HTTP/1.1, has no Connection: close and its body ends by its length or its chunks; up to 8 idle ones are kept.
  • Sent again. A request the server did not process is sent again, once: an HTTP/2 stream refused (REFUSED_STREAM) or above a GOAWAY’s last stream, any method; an idempotent one (GET, HEAD, PUT, DELETE, OPTIONS) whose kept connection the server had closed, or whose HTTP/2 connection ended before the response. A server that answers HTTP_1_1_REQUIRED is reached over HTTP/1.1 from then on.
  • An HTTP/2 response’s head reads HTTP/2 200 and then its headers as lines, so r.header and r.headers work the same (names arrive in lowercase; r.reason() is "").

Nothing is sent until you wait: start only queues, so start a batch, then wait. read_timeout bounds each wait for the servers; when nothing comes for that long, every request in flight fails with TimedOut. Connecting (and the TLS handshake) to a new server happens inside the wait and holds it up meanwhile. Don’t copy a Client, and use it from one thread at a time.

Options

tape
pub struct Options {
    max_redirects: i64;            // 0: do not follow (the default)
    max_body: i64;                 // bytes
    max_head: i64;                 // status line and headers, bytes
    connect_timeout: ?time.Duration;
    read_timeout: ?time.Duration;  // each receive and each send
    trust: ?*const x509.TrustStore;  // https; null: tape's bundled roots
}

options() gives no redirects, a 64 MiB body limit, a 64 KiB head limit, 30 seconds for connecting and 30 seconds for each read and write. Build Options with options() and change what you need: a zero-filled Options would refuse every body. A null timeout waits forever.

connect_timeout covers resolving the host and connecting (racing its addresses, as net.connect_host does). read_timeout is the longest wait for the next bytes, not a limit on the whole response: a slow server that keeps sending is not cut off. For https, connect_timeout also bounds the TLS handshake.

trust is the set of roots an https server’s certificate must chain to: null is tape’s bundled roots (net.bundled_trust); a net.Trust you build gives its store(). To also trust the operating system’s roots, build a Trust with add_bundled and add_system.

Responses

tape
pub struct Response {
    status: i64;
    head: string;                  // status line and header lines as received
    body: string;
    url: string;                   // the URL finally fetched (after redirects)
}
MethodDoes
r.reason() -> []const u8The status line’s reason phrase ("Not Found"; "" when there is none)
r.header(name) -> ?[]const u8The first header called name, ignoring case; its value without surrounding spaces
r.headers() -> HeaderIterEvery header in the order received, repeated ones included

The headers are not copied out: header and headers look them up in r.head and return views into it, valid while r is.

tape
var it: http.HeaderIter = r.headers();
while (true) {
    let f: ?http.Field = it.next();
    if (f == null) { break; }
    io.println("{f!.name}: {f!.value}");
}

How much body there is follows RFC 9112:

  • none for a HEAD request and for 1xx, 204 and 304 responses;
  • with Transfer-Encoding: chunked, the chunks (any chunk extensions ignored, trailers read and dropped); a Content-Length next to it is ignored. Any other transfer coding is a BadHeader error;
  • with Content-Length, exactly that many bytes. Repeated values must agree;
  • otherwise everything until the server closes the connection (how HTTP/1.0 servers send).

Interim 1xx responses (100 Continue, 103 Early Hints) are skipped; r.head is the final response’s head. Lines ending in a bare LF are accepted as well as CRLF. Folded header lines (a line starting with a space or a tab) are refused.

Redirects

With max_redirects above 0, a 301, 302, 303, 307 or 308 with a Location is followed, up to that many times. The Location may be an absolute URL or a reference resolved against the current URL (/path, ../path, ?query, //host/path), as RFC 3986 resolves them.

  • 303, and 301 or 302 after a POST, continue as a GET without the body (a HEAD stays a HEAD). 307 and 308 keep the method and the body.
  • A redirect to another origin (another host, scheme or port) drops your Authorization and Cookie headers for the rest of the request.
  • https to http is refused (InsecureRedirect).
  • More than max_redirects is TooManyRedirects; a missing Location, or one that isn’t a usable http/https URL, is BadRedirect.

r.url is the URL of the response you got. With redirects off (the default), a 3xx is returned like any other response, Location and all.

URLs

tape
pub struct Url { scheme: []const u8; host: []const u8; port: u16; target: []const u8; }

parse_url reads scheme://host[:port][/path][?query][#fragment]:

  • the scheme is http or https in any case (scheme is the lowercase name); anything else is UnsupportedScheme;
  • the host is a name (letters, digits, -, ., _), an IPv4 address, or an IPv6 address in brackets (host without them; zone ids are refused);
  • the port is 1 to 65535; without one (or with an empty one, h:/), the scheme’s default, 80 or 443;
  • target is the path and query exactly as written, the fragment cut off, "/" when both are empty (http://h?q has the target ?q and is requested as /?q).

There is no decoding or normalization: the target goes on the wire as written. Bytes that can’t appear in a request target are refused rather than guessed at: spaces, control characters, bytes above 127 (percent-encode them, encoding can) and a % not followed by two hex digits. User information (user:password@) is refused too: put credentials in an Authorization header.

Textparse_url
http://example.comhttp, example.com, 80, /
HTTPS://Example.com:8443/a/b?q=1#tophttps, Example.com, 8443, /a/b?q=1
http://[2001:db8::1]:8080/http, 2001:db8::1, 8080, /
http://user:pw@example.com/BadUrl
ftp://example.com/fileUnsupportedScheme

Errors

tape
pub tagged HttpError {
    Io { e: io.IoError; }          // connecting, sending, receiving
    Tls { e: tls.TlsError; }       // https: the handshake, the certificate, the records
    Http { e: ProtocolError; }     // the URL, the response, redirects
    H2 { e: h2.H2Error; }          // a Client's HTTP/2 connection: the server's error codes, the limits
}

err.desc() gives the text of any. The transport errors are io.IoError values from net: HostNotFound, ConnectionRefused, TimedOut (a timeout passed), ConnectionReset, UnexpectedEof (the server closed before the end of the head), Unsupported (a platform without sockets). The TLS errors are tls.TlsError values: UnknownCa (the certificate doesn’t chain to a trusted root), NameMismatch, CertificateExpired, PeerAlert (the server refused), Truncated, and the others tls lists.

Over TLS, a body that ends with the connection must end with the server’s close_notify: a plain TCP close is Tls { Truncated } there (someone may have cut the connection). A body whose length is known (Content-Length, chunked) is complete without it, as many servers skip close_notify.

ProtocolErrorWhen
BadUrlthe URL is malformed
UnsupportedSchemenot http or https
BadStatusLinenot HTTP/1.x and a status from 100 to 599
BadHeadera malformed or folded header line, or a transfer coding other than chunked
HeadTooLargethe head (interim responses included), or a chunked body’s trailers, over max_head
BadContentLengthnot a decimal number, or several that differ
BadChunka chunk size that isn’t 1 to 15 hex digits, no CRLF after a chunk, a chunk line over max_head
BodyTooLargethe body over max_body (a Content-Length over it is refused before reading)
TooManyRedirects, BadRedirect, InsecureRedirectsee Redirects
TruncatedBodythe connection ended before the length or the last chunk said it would

Over HTTP/2 (a Client’s https requests), the response’s own failures are H2 { e }, an h2.H2Error: the RFC 9113 error codes the server resets a stream or ends the connection with (Protocol, Internal, FlowControl, Refused, Cancel, Compression, Calm, …), and Closed (the connection ended before the response), GoAway, HeadTooLarge (over max_head), BodyTooLarge (over max_body) and Malformed (a response that breaks HTTP/2’s rules).

tape
let r: http.Response = http.get(url, opts) or {
    match (err) {
        http.HttpError.Io { e } => {
            if (e == io.IoError.TimedOut) { io.eprintln("the server is too slow"); }
        }
        http.HttpError.Tls { e } => { io.eprintln("TLS: {e}"); }
        http.HttpError.Http { e } => { io.eprintln("bad answer: {e}"); }
        http.HttpError.H2 { e } => { io.eprintln("HTTP/2: {e}"); }
    }
    return 1;
};

Example

stdlib/examples/http_get.tape is a small wget: it fetches the URL given on its command line, following up to five redirects, and prints the status, the final URL and the body.

plaintext
tape run stdlib/examples/http_get.tape -I stdlib -- http://127.0.0.1:8080/

stdlib/examples/https_get.tape does the same over https, with the bundled roots, a root of your own (--ca root.pem, PEM or DER), or the operating system’s roots besides the bundled ones (--system):

plaintext
tape run stdlib/examples/https_get.tape -I stdlib -- https://example.com/
tape run stdlib/examples/https_get.tape -I stdlib -- https://intranet.example/ --ca corp-root.pem

Last modified: