HTTP/3 and QUIC

QUIC below HTTP

The HTTP/3 layers are ordinary code on top of QuicConnection, the surface every QUIC connection has. Use it directly for a protocol of your own - every stream at once, one stream behind an IDuplexPipe, several protocols chosen by ALPN - and, from the other end, as a client.

Raw streams#

A server that speaks a protocol called echo and sends every stream back to its sender:

using ioxide;
using ioxide.ngtcp2;

// QUIC with no HTTP: the protocol this server speaks is "echo", and every stream is echoed back.
using var engine = new QuicEngine("cert.pem", "key.pem", cidLength: 8, alpn: ["echo"]);

var config = new ServerConfig
{
    ReactorCount = Environment.ProcessorCount,
    Tcp = null,
    Quic = new QuicOptions
    {
        Port = 8443,
        LocalCidLength = 8,
        ConnectionFactory = engine.CreateFactory(),
    },
};

var threads = new Thread[config.ReactorCount];

for (int id = 0; id < threads.Length; id++)
{
    var reactor = new Reactor(id, config);

    // Runs once per connection, from before the handshake completes until the connection ends.
    reactor.QuicHandle = async (r, connection) =>
    {
        try
        {
            while (true)
            {
                QuicRecvSnapshot snapshot = await connection.ReadAsync();

                while (connection.TryGetDelivery(in snapshot, out QuicRecvRing.Delivery delivery))
                {
                    if (delivery.Kind == QuicStreamEvent.Data)
                    {
                        // Same stream, same bytes; the peer's FIN ends our side too.
                        connection.SendStream(delivery.StreamId, delivery.AsSpan(), fin: delivery.Fin);
                    }
                    connection.ReturnBuffer(in delivery);
                }

                if (snapshot.IsClosed) return;
                connection.ResetRead();
            }
        }
        finally
        {
            connection.DecRef();   // the handler's reference
        }
    };

    threads[id] = new Thread(reactor.Run) { Name = $"reactor-{id}" };
    threads[id].Start();
}

foreach (Thread thread in threads)
{
    thread.Join();
}

The engine is the same QuicEngine HTTP/3 uses, with your protocol's name as its ALPN - a client must offer it. The handler is the TCP handler's shape: wait for a snapshot, drain the deliveries up to it, give each buffer back, ResetRead before the next wait, and let go of the connection with DecRef when it closes. It resumes inline on the reactor once ngtcp2 has finished with the datagram that brought the data. The difference from TCP is that a delivery names its stream: one connection carries many, and lifecycle events ride the same queue as the data, in order.

The rest of the surface, all of it used on the reactor thread, which is where a handler runs:

The queue holds 256 deliveries. A handler that stops reading while its peer keeps sending - awaiting something else between reads, say - has its connection closed when the queue overflows.

Sending more than the high-water#

SendStream never refuses bytes: it copies them into the connection's retention, where they stay until acknowledged. A producer of large or endless output checks CanQueueSend and carries on from OnSendCapacityAvailable, which fires on the reactor when acknowledgements bring retention below QuicOptions.SendRetentionBytes, or when the connection closes. To send body, a ReadOnlyMemory<byte>, on stream:

// Called on the reactor thread. Queues while the connection is under its high-water, and
// carries on each time acknowledgements bring it back below.
void Pump()
{
    while (!body.IsEmpty && connection.CanQueueSend)
    {
        int n = Math.Min(body.Length, 64 * 1024);
        connection.SendStream(stream, body.Span[..n], fin: n == body.Length);
        body = body[n..];
    }

    // Fired when retention drains below the high-water, or when the connection closes.
    connection.OnSendCapacityAvailable = body.IsEmpty || connection.IsClosed ? null : Pump;
}

Pump();

OnSendCapacityAvailable is one property: the HTTP/3 layers set it for themselves and the pipe writer adds to it, so set it only on a connection you drive yourself. A producer that ignores the high-water reaches SendRetentionCeilingBytes and its connection is closed.

One stream behind an IDuplexPipe#

QuicConnectionDualPipe puts one stream of a connection behind a PipeReader and a PipeWriter, so code written for System.IO.Pipelines runs over QUIC unchanged:

reactor.QuicHandle = async (r, connection) =>
{
    var pipe = new QuicConnectionDualPipe(connection);   // binds to the first stream the peer uses
    try
    {
        while (true)
        {
            ReadResult result = await pipe.Input.ReadAsync();
            foreach (ReadOnlyMemory<byte> segment in result.Buffer)
            {
                pipe.Output.Write(segment.Span);
            }
            pipe.Input.AdvanceTo(result.Buffer.End);

            FlushResult flushed = await pipe.Output.FlushAsync();   // waits at the high-water
            if (result.IsCompleted || flushed.IsCompleted) break;
        }
        pipe.Output.Complete();    // our FIN on the stream
    }
    finally
    {
        pipe.Input.Complete();     // hands back what the reader still holds
        connection.DecRef();
    }
};

• One stream. A PipeReader is one byte stream, so the pipe binds to the first stream the peer sends on, or to the id you pass. Deliveries for any other stream are dropped - the reader is the connection's only consumer. That is why HTTP/3, which needs many streams at once, takes the raw surface instead.
• Reading completes on the stream's FIN, its reset or close, or the connection's close. A StopSending concerns the writer and is ignored here. Input.Complete() hands back the buffers the reader still holds.
• Writing stages bytes, and FlushAsync hands them over in one SendStream. It waits while the connection is at its high-water, and reports IsCompleted once the connection is closed. Complete() sends the FIN; Complete(exception) discards what was staged and resets the stream with code 0. Flushing bytes before any stream is bound throws.
• Staging more than SendRetentionCeilingBytes in a single flush closes the connection, since one flush is one hand-over: flush as you write.

Playground/Quic/Pipe is the full server - QUIC's twin of Tcp/Pipe.

Two protocols on one port#

ALPN names the protocol during the handshake, so one QUIC port can serve several. The engine accepts the list - the server's order decides when a client offers more than one - and the handler branches on what was agreed:

// Engine side: offer both, in the server's order of preference.
using var engine = new QuicEngine("cert.pem", "key.pem", cidLength: 8, alpn: ["h3", "echo"]);

reactor.QuicHandle = async (r, connection) =>
{
    // The handler starts before the handshake ends. Wait for the first delivery - stream data
    // comes after the handshake - and drain nothing: whoever takes over reads the same queue.
    await connection.ReadAsync();

    if (connection.NegotiatedProtocol == "h3")
    {
        await new Http3Connection(connection).RunAsync(request => Http3Response.Text("h3\n"));
        return;                    // the HTTP/3 layer released the handler's reference
    }

    await EchoAsync(connection);   // owns the reference from here
};

QuicHandle starts with the client's first packet, before the handshake has finished, when NegotiatedProtocol is still null. Stream data only arrives once the handshake is done, so the first ReadAsync to return with data is the point at which the protocol is known. Nothing has been drained, so the layer that takes over finds every delivery still queued and its own first read returns at once. A connection that closes before its handshake completes wakes the read with IsClosed and no protocol, and falls to the second branch, which must cope with a closed connection - the pipe echo above simply completes. EchoAsync stands for that pipe echo; whichever branch runs owns the handler's reference from then on. Playground/Quic/Alpn does this with alpn: null, so a client offering any protocol, or none, lands in the echo branch.

The QUIC client#

QuicClientEngine opens connections from a reactor, and they behave as accepted ones do - the same QuicConnection surface, resumed inline on the reactor that opened them. A client needs no listener of any kind:

using System.Text;
using ioxide;
using ioxide.ngtcp2;

// A client opens no listener: TCP, UDP and QUIC all stay off.
var config = new ServerConfig { ReactorCount = 1, Tcp = null, Udp = null, Quic = null };

using var engine = new QuicClientEngine("echo");   // the one ALPN this engine offers
var reply = new TaskCompletionSource<string>(TaskCreationOptions.RunContinuationsAsynchronously);

var reactor = new Reactor(0, config)
{
    OnStart = r =>
    {
        // An IPv4 literal and a port; serverName goes out as SNI.
        QuicEngineConnection connection = engine.Connect(r, "127.0.0.1", 8443, serverName: "localhost");

        // Streams open only after the handshake.
        connection.HandshakeCompleted = () => _ = EchoAsync(connection);
    },
};

var thread = new Thread(reactor.Run) { Name = "quic-client" };
thread.Start();

Console.WriteLine(reply.Task.Wait(TimeSpan.FromSeconds(10)) ? $"echoed: {reply.Task.Result}" : "no answer");

reactor.Stop();
thread.Join();   // the reactor is done with the engine before it is disposed

async Task EchoAsync(QuicEngineConnection connection)
{
    try
    {
        long stream = connection.OpenBidiStream();             // negative: the peer allows no more
        connection.SendStream(stream, "hello over QUIC"u8, fin: true);

        var received = new List<byte>();
        bool finished = false;

        while (!finished)
        {
            QuicRecvSnapshot snapshot = await connection.ReadAsync();

            while (connection.TryGetDelivery(in snapshot, out QuicRecvRing.Delivery delivery))
            {
                if (delivery.StreamId == stream && delivery.Kind == QuicStreamEvent.Data)
                {
                    received.AddRange(delivery.AsSpan());
                    finished |= delivery.Fin;
                }
                connection.ReturnBuffer(in delivery);
            }

            if (snapshot.IsClosed) break;
            connection.ResetRead();
        }

        reply.TrySetResult(Encoding.UTF8.GetString(received.ToArray()));
        connection.Close(0);    // CONNECTION_CLOSE, application error 0
    }
    finally
    {
        connection.DecRef();    // the reference Connect gave the caller
    }
}

• new QuicClientEngine(string alpn = "h3") offers one protocol. Make one per process, or per protocol, and dispose it only after the reactors using it have stopped, as the program does.
• Connect(reactor, ipv4, port, serverName) runs on the reactor's thread - from OnStart or a handler - and takes an IPv4 literal: no names, no IPv6. The first connect on a reactor opens a UDP socket on an ephemeral port, also on a reactor that serves QUIC: the serving port is shared by every reactor, and replies to it would be hashed to any of them.
• HandshakeCompleted is raised once, on the reactor, after the handshake - the earliest a stream can be opened. OpenBidiStream() returns the new stream's id, negative when the server allows no more.
• SendBatched(Action send) runs send as one engine cycle, so the datagrams it produces leave in one batch. A server's answers get this for free, because they happen while a datagram is being handled; a client's sends usually come from its own loop, and would otherwise cost a sendmsg each.
• The connection has two owners, the transport and you: Close(code) ends it on the wire, and DecRef() lets go of yours. On a reactor with no QuicOptions a client connection that hears nothing for 30 seconds - handshake included - is dropped, and its IDs are 8 bytes; otherwise ReadTimeoutMs and LocalCidLength apply.

Playground/Clients/Quic is this client at scale - one stream per connection, one payload in flight - and is how the repository load-tests Quic/Raw, Quic/Pipe and Quic/Alpn, which no HTTP load generator can drive.