HTTP/3 and QUIC

HTTP/3 and QUIC

ioxide serves HTTP/3 over its own QUIC transport. Each reactor receives datagrams on its own ring, ngtcp2 runs the protocol with TLS 1.3 inside it, and an HTTP/3 layer turns streams into requests - all on the reactor thread that owns the connection.

dotnet add package ioxide.ngtcp2
dotnet add package ioxide.http3

ioxide.ngtcp2 is the QUIC engine: ngtcp2 and picotls, on BoringSSL's crypto, in one bundled native library that needs only libc. The HTTP/3 layer is either ioxide.http3, in C#, or ioxide.nghttp3, which bundles nghttp3 the same way and needs nothing else. Both layers depend on ioxide core alone, so they would ride any QUIC engine; today there is one.

The pieces#

A complete server#

HTTP/3 on UDP port 8443, a reactor per core, the request answered once it has fully arrived:

using System.Text;
using ioxide;
using ioxide.http3;
using ioxide.ngtcp2;

// One engine for the whole server: the certificate, its key, and the one protocol it speaks.
using var engine = new QuicEngine("cert.pem", "key.pem", cidLength: 8, alpn: ["h3"]);

var config = new ServerConfig
{
    ReactorCount = Environment.ProcessorCount,
    Tcp = null,             // QUIC only: no TCP listener
    DualStack = true,       // one IPv6 socket that takes IPv4 too - a browser asks for localhost on ::1
    Quic = new QuicOptions
    {
        Port = 8443,                                // a UDP port, bound by every reactor
        LocalCidLength = 8,                         // the engine's cidLength
        ConnectionFactory = engine.CreateFactory(), // adopts each new connection
    },
};

// Built once and returned for every request to "/".
Http3Response hello = Http3Response.Text("Hello, World!\n");

var threads = new Thread[config.ReactorCount];

for (int id = 0; id < threads.Length; id++)
{
    var reactor = new Reactor(id, config);

    // One call per connection; the handler runs once per request, on this reactor's thread.
    reactor.QuicHandle = (r, connection) => new Http3Connection(connection).RunAsync(request =>
    {
        if (request.Path.Span.SequenceEqual("/"u8))
        {
            return hello;
        }

        string method = Encoding.ASCII.GetString(request.Method.Span);
        string path = Encoding.ASCII.GetString(request.Path.Span);
        return Http3Response.Text($"{method} {path}: {request.Body.Length} bytes in the body\n", status: 404);
    });

    threads[id] = new Thread(reactor.Run) { Name = $"reactor-{id}" };
    threads[id].Start();
}

// Keep the engine alive while the reactors use it: it is disposed when this returns.
foreach (Thread thread in threads)
{
    thread.Join();
}

It needs a certificate and its key as PEM files. For a test, a self-signed pair for localhost:

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes -keyout key.pem -out cert.pem -days 365 -subj "/CN=localhost" -addext "subjectAltName=DNS:localhost,IP:127.0.0.1,IP:::1"

Then, with a curl built with HTTP/3 - a stock one has no --http3-only:

curl --http3-only --cacert cert.pem https://localhost:8443/

• Tcp = null opens no TCP listener. Left at its default, every reactor binds TCP port 8080 as well, for a TcpHandle this program never sets.
• QuicHandle runs once per connection, from the client's first packet - before the handshake has finished. RunAsync owns the handler's reference to the connection and lets it go when the connection ends.
• The handler runs on the reactor thread once per request, with the whole body in request.Body. Everything on the request is bytes, valid until the handler returns: compare with u8 literals rather than decoding.
• The engine outlives the reactors. Disposing it frees the native state every connection reads, so the program joins the reactor threads before using lets it go.

Choosing an HTTP/3 layer#

The two layers are interchangeable at the call site - the same shape, different type names - and differ in what they implement:

ioxide.http3 is the lighter of the two and, on the repository's runs, the faster; ioxide.nghttp3 brings the reference implementation, the dynamic table and the GOAWAY drain. Pin alpn: ["h3"] on the engine either way: ioxide.http3 closes a connection that did not negotiate h3, ioxide.nghttp3 serves whatever reaches it.

On ioxide.http3 a response reused across requests keeps its cached head as long as its status, header count and body length are unchanged - so change a header's value and the old one still goes out. Build a new response instead.

Three ways to answer a request#

A streaming request is paced by QUIC flow control: a chunk is credited back to the peer only when ReadAsync hands it over, so a slow handler holds the sender back rather than buffering - one 256 KiB stream window, whatever the body. Each chunk is valid until the next call, an empty one is the end, and a body the handler leaves unread is credited back when it returns.

reactor.QuicHandle = (r, connection) => new Http3Connection(connection).RunAsync(async request =>
{
    long total = 0;
    while (true)
    {
        ReadOnlyMemory<byte> chunk = await request.BodyReader!.ReadAsync();   // empty: end of body
        if (chunk.IsEmpty) break;
        total += chunk.Length;                    // valid until the next ReadAsync
    }
    return Http3Response.Text($"{total} bytes\n");
});

A streamed response writes its headers once, before any body and with no content-length. Each flush of ioxide.http3's writer is one DATA frame; nghttp3's writer hands its chunk to nghttp3, one in flight at a time. FlushAsync waits while the connection is at its send-retention high-water and reports IsCompleted once the peer has stopped reading. The writer is completed for a handler that returns without doing so, with a 500 if it never wrote headers.

reactor.QuicHandle = (r, connection) => new Http3Connection(connection).RunStreamedResponseAsync(async (request, writer) =>
{
    var head = new Http3Response { Status = 200 };
    head.Headers.Add(("content-type"u8.ToArray(), "text/plain"u8.ToArray()));
    writer.WriteHeaders(head);                    // once, before any body byte; no content-length

    for (int i = 0; i < 1000; i++)
    {
        int written = Encoding.ASCII.GetBytes($"line {i}\n", writer.GetSpan(32));
        writer.Advance(written);

        FlushResult flushed = await writer.FlushAsync();   // one DATA frame; waits at the high-water
        if (flushed.IsCompleted) return;                   // the peer stopped reading
    }
});                                               // the writer is completed for you on return

On every model a handler that throws before its headers are out answers 500; once a streamed response's headers are out, the stream is reset with H3_INTERNAL_ERROR, since a clean end would pass a truncated body off as whole. The Playground has each one: Http3/ManagedBuffered, Http3/Nghttp3Buffered, Http3/Nghttp3Request, Http3/Nghttp3Response, Http3/Nghttp3StreamedBoth, and Http3/ManagedStreamedBoth for the C# writer.

The engine is the TLS configuration#

There is no TlsService on QUIC. TLS 1.3 runs inside the transport, so the certificate is a constructor argument of QuicEngine, and one engine serves every reactor: whichever reactor a client's first packet reaches calls the factory, which reads the engine's current certificates without a lock. The TCP side's TLS (TLS and kTLS) is a separate stack with its own configuration - a server offering HTTP/2 and HTTP/3 configures and renews its certificates on each.

The negotiated protocol is QuicConnection.NegotiatedProtocol once the handshake completes. Every connection makes a full handshake: the engine issues no session tickets, so there is no resumption and no 0-RTT. A client that offers the post-quantum X25519MLKEM768 key exchange gets it.

Several names on one port#

using var engine = new QuicEngine("default.crt", "default.key", cidLength: 8, alpn: ["h3"]);

engine.AddHost("alpha.example", "alpha.crt", "alpha.key");
engine.AddHost("beta.example", "beta.crt", "beta.key");

QuicConnectionFactory factory = engine.CreateFactory();   // the host table is closed from here

• Register every name before CreateFactory. Afterwards AddHost throws: handshakes on every reactor read the table without a lock. To change a serving engine's names, replace its certificates.
• The constructor's certificate is the default: a client that sends no name, or one that is not registered, gets it rather than a refusal.
• Names match in full, ignoring case - there are no wildcards. A name registered twice is refused, and so is one no client could ask for - with a *, an IP address, a non-ASCII name, over 255 bytes - and a handshake naming more than one host.
• The name picks the certificate and nothing else. Route requests by request.Authority. Mutual TLS applies to every name alike.

Renewing certificates#

Because the engine is shared, renewal is one call, and it covers every reactor: ReplaceCertificates builds the new set, then publishes it with a single store, so a handshake sees every old certificate or every new one, never a mixture.

void Renew()
{
    try
    {
        // The same paths: their new contents are read. The whole set, table included.
        engine.ReplaceCertificates(
            new QuicCertificate("default.crt", "default.key"),
            new Dictionary<string, QuicCertificate>
            {
                ["alpha.example"] = new("alpha.crt", "alpha.key"),
                ["beta.example"] = new("beta.crt", "beta.key"),
            });
    }
    catch (Exception e)
    {
        Console.Error.WriteLine($"renewal refused, still serving the previous set: {e.Message}");
    }
}

// What an ACME hook sends after rewriting the PEM files.
using var sighup = PosixSignalRegistration.Create(PosixSignal.SIGHUP, context =>
{
    context.Cancel = true;
    Renew();
});

• It replaces the whole set. On an engine serving names, leaving out the table throws rather than silently answering every name with the default; pass an empty dictionary to mean that. A name added to the table is served from then on.
• Nothing is published unless all of it loads: a bad path or a key that does not match throws and leaves the previous set serving.
• Established connections keep the certificate they started with. Replaced sets stay in memory until the engine is disposed, because picotls keeps reading them for as long as a connection lives.
• The client trust anchors and requireClientCertificate do not change: they belong to the engine, and changing them takes a new one. The TCP side renews each reactor's TlsService on its own - see Certificates and mutual TLS. Playground/Http3/Rotate renews on SIGHUP and on a timer.

Mutual TLS#

using var engine = new QuicEngine("server.crt", "server.key", cidLength: 8, alpn: ["h3"],
    clientCaPemPath: "clients-ca.crt",      // turns client certificates on
    requireClientCertificate: true);        // no certificate: the handshake fails

reactor.QuicHandle = (r, connection) => new Http3Connection(connection).RunAsync(request =>
{
    // Read per request: the identity exists once the handshake is done.
    string? name = ((QuicEngineConnection)connection).PeerCommonName;

    return name == "alice"
        ? Http3Response.Text("hello, alice\n")
        : Http3Response.Text("forbidden\n", status: 403);
});

• With anchors set, every client is asked for a certificate. requireClientCertificate: false lets a client with none in, unnamed; one presenting a certificate from another CA is refused either way.
• Validation covers the chain, the signatures and the validity window - not revocation: no CRL or OCSP is consulted.
• PeerCommonName is the value to authorise on; compare it whole, ordinal. PeerSubject is for logs: its escaping lets a crafted organisation name contain /CN=admin. Neither is there until the handshake completes, so read them per request, not when QuicHandle starts.
• QUIC settles client authentication during the handshake and RFC 9001 forbids asking later, so it covers the whole connection. A protected area that needs certificates while the rest does not needs a second port. Playground/Http3/MutualTls is the full program.

Ending connections gracefully#

ioxide.nghttp3's Shutdown() sends GOAWAY, refuses new requests, lets those in flight finish, then closes the connection with H3_NO_ERROR. It must run on the connection's reactor, and the deadline after which you give up is yours to keep:

reactor.QuicHandle = async (r, connection) =>
{
    var h3 = new Nghttp3Connection(connection);
    lock (live) live.Add((r, h3));
    try
    {
        await h3.RunBufferedAsync(request => Nghttp3Response.Text("ok\n"));
    }
    finally
    {
        lock (live) live.Remove((r, h3));
    }
};

// On SIGTERM: a GOAWAY on every connection, each from its own reactor's thread.
using var drain = PosixSignalRegistration.Create(PosixSignal.SIGTERM, context =>
{
    context.Cancel = true;
    lock (live)
    {
        foreach ((Reactor owner, Nghttp3Connection h3) in live)
        {
            owner.ScheduleOnReactor(static state => ((Nghttp3Connection)state!).Shutdown(), h3);
        }
    }
});

ioxide.http3 has no equivalent. On either layer, QuicConnection.Close(code), called on the connection's reactor, ends a connection at once with that application error code.

The other option nghttp3 adds is Nghttp3Options, passed as new Nghttp3Connection(connection, options): QpackDynamicTableCapacity (0) advertises a QPACK dynamic table the peer may insert its request headers into - 4096 is the conventional size - and QpackBlockedStreams (0) how many header blocks may wait for an insert; raise both together.

Browsers and localhost#

Browsers try localhost at ::1, and ioxide's sockets are IPv4-only unless ServerConfig.DualStack is set - a server bound the default way receives no packets from a browser on the same machine, which looks like a broken HTTP/3 stack. DualStack = true binds one IPv6 socket on :: that takes IPv4 clients as well, as the program above does, so it answers at 127.0.0.1 and at [::1] alike.

A browser also has to learn that an origin speaks HTTP/3 before it tries: it opens TCP first and looks for an alt-svc header, which ioxide never adds by itself. A QUIC-only server is for clients that go straight to HTTP/3, such as curl --http3-only; to bring browsers, advertise it from your HTTP/1.1 or HTTP/2 responses.

Measured#

From the repository, on one machine, at 2 reactors:

The August run predates the October changes to the send path, and used h3x, which the bench has since replaced with h2load: h3x divides by the whole run, so a few requests that never finish drag its rate down. Both runs predate the move of QUIC's crypto from the system's libcrypto to BoringSSL's. Each row is one operating point on one machine - compare runs taken together, never across machines.

Limits#

• No HTTP/3 client. QuicClientEngine speaks raw QUIC, IPv4 only, and does not check the server's certificate (the QUIC client). ioxide.httpclient is HTTP/1.1 (HTTP client).
• No Retry, no session resumption, no 0-RTT. Every new connection costs a full handshake, and an Initial that passes ngtcp2's checks gets a connection without an address check first.
• Trailers are dropped on both layers, and neither does server push.
• ioxide.nghttp3 sends at most 64 field lines in a response, :status and the content-length it adds included; lines past that are dropped without an error.
• ioxide.http3 refuses a request whose header section is over 64 KiB, closing the connection with H3_EXCESSIVE_LOAD.

Where to go next#