HTTP/3 and QUIC

QUIC on the ring

One UDP socket per reactor carries every QUIC connection that reactor owns. This page follows a datagram from the kernel to its connection and back: how it reaches the owning reactor when the kernel delivers it elsewhere, what wakes a connection's timers, and how much a connection may hold unacknowledged.

Sockets and buffers#

Every reactor binds QuicOptions.Port with SO_REUSEPORT, and the kernel spreads incoming datagrams over the reactors by hashing the sender's address and port. Raw UDP ports in ServerConfig.Udp live beside it; the QUIC port needs no entry there, and takes the UdpOptions defaults when Udp is null.

• Receiving. One multishot RECVMSG per socket draws from a provided-buffer ring shared by the reactor's UDP sockets: UdpOptions.RecvSlots buffers, each with room for a 64 KiB payload, returned to the ring as soon as their datagram has been dispatched. A burst that drains the ring only pauses the receive until it is re-armed.
• GRO. With Gro on, a run of equal-sized datagrams from one peer arrives as one completion. The reactor splits it before routing: connections can share a peer's address and port, so neighbours in one run can belong to different connections.
• Sending. What a connection produces in one engine cycle - handling an incoming datagram, or a timer - goes out as one batch of up to 63 KiB in a single sendmsg with UDP_SEGMENT. A send made outside a cycle, by a handler resumed by a database reply say, goes out at once, a sendmsg per datagram. Datagrams are at most 1452 bytes.

Socket buffer sizes#

UdpOptions.SocketBufferBytes asks for 8 MiB of receive and send buffer on every UDP socket. It is a request: Linux clamps it to net.core.rmem_max and wmem_max without an error, and on a stock install that is 212,992 bytes. ioxide reads the granted size back and, once per process, says so at startup:

[ioxide] udp: asked for 8192 KiB of socket buffer, the kernel granted 208 KiB (capped by net.core.rmem_max). Expect datagrams to be dropped under load; raising the cap trades those drops for queueing delay, so measure it.

Raising the cap is not automatically better. On the repository's HTTP/3 benchmark, granting the full 8 MiB cost about 45% of throughput at saturation: the drops stopped, a deep standing queue took their place, and peers timed out and retransmitted instead (the note on UdpOptions.SocketBufferBytes). Early drops are the signal congestion control reads; measure on the deployment. A value of 0 or less is refused at startup.

How a datagram finds its connection#

One socket carries every connection, so the reactor demultiplexes by the destination connection ID, which the first bytes of every QUIC packet carry in the clear (RFC 8999). A long header - the handshake - states the ID's length; a short header - everything after - does not, so the reactor reads exactly QuicOptions.LocalCidLength bytes. That is why the engine's cidLength and LocalCidLength must be equal. A long header whose ID is empty or longer than 20 bytes is dropped, since no ID the server issues is either.

1. A known ID goes to its connection, and ngtcp2 decrypts it, processes acknowledgements and flow control, and hands decrypted stream data to the connection's delivery queue. The handler is woken once, after ngtcp2 has returned, so an answer it sends straight away never re-enters ngtcp2 in the middle of a read.
2. An unknown long header is a new client, unless it is a Handshake packet (see below). QuicOptions.ConnectionFactory - the engine's - checks the Initial before building anything, since a connection costs about 75 KB, then creates it. Both the client's ID and the one the server mints route to it, and QuicHandle starts.
3. An unknown short header belongs to a connection on another reactor, or to none - see below.

Every connection ID the server mints names its reactor: the first byte is chosen so that cid[0] % ReactorCount is the owner's ShardIndex, the rest stays random. ngtcp2 issues further IDs as the connection goes on, and each is routed as it is issued and dropped when the peer retires it.

When the kernel picks the wrong reactor#

The kernel's choice follows the client's address and port. When those change - a NAT rebinding after a quiet spell is enough, no phone has to move - the hash picks another reactor, one that has never seen the connection. The connection cannot follow: its ngtcp2 state, TLS session and streams belong to one thread. So the datagram moves.

Under QuicRouting.Forward, the default, a reactor that receives a short header for an ID it does not hold reads the owner from the ID's first byte, copies the datagram - its buffer goes back to the ring when dispatch returns - and posts the copy to the owner's queue, the one reactors already use to hand each other work. Up to 1024 may wait for one reactor; past that they are dropped, and QUIC resends them. An ID that names this reactor and is not here is stale - retired after a migration, long gone, or forged - and is dropped; no stateless reset is sent.

Handshake packets are forwarded as well: their ID is the one the server minted in its Initial, so it names the owner too. It matters at startup: a reactor that binds the port after a client's first Initial changes the hash, and without the forwarding such a handshake would stall until the 10-second handshake timeout gave up on it. Initial and 0-RTT packets are never forwarded: their ID is the client's choice, and routing on it would let anyone aim connections at a reactor of their choosing.

Once the datagram is with its owner, the address is ngtcp2's decision. ioxide hands it the address every datagram came from; ngtcp2 probes a new one with PATH_CHALLENGE, and until the path is validated sends at most three times what it has received on it, so a forged address cannot turn the server into an amplifier. The engine accepts up to 8 connection IDs from the client, so a second migration soon after the first still finds a fresh one to answer on.

Claiming a migrated client's address#

Forwarding would charge every datagram of a migrated client a hop for the rest of the connection. With PinMigratedPeers on (the default, and only under Forward), the owner claims the new address: from its sweep, every 250 ms tick, it opens a socket bound to the QUIC port and connect()ed to the client. A connected socket outranks the wildcard binds in the kernel's lookup, so that client's datagrams arrive at the owner directly. connect() on UDP sends nothing, and a connected socket takes no part in the reuseport hash, so no other client moves.

• Claims are made from the sweep rather than when ngtcp2 reports the move: a path is reported many times while it validates, alternating between the old address and the new, and by the next sweep it has settled.
• Only a moved connection is claimed, only with more than one reactor, and at most 512 per reactor - past that, forwarding carries on. Each claim is a file descriptor.
• A claim is released when its connection ends or moves again. The release cancels the socket's armed receive before closing it: left armed, the receive would keep the socket bound and connected, and a client that came back from the same address would never complete a handshake again.

Letting the kernel route by ID#

QuicRouting.KernelFilter attaches a classic-BPF program to the port's reuseport group, so the kernel reads the ID's first byte itself and delivers to the owner directly: a migrated client costs nothing and nothing is claimed. An Initial is routed on the first byte of the ID the client chose - a stable hash, so the packets of one handshake stay together.

The program answers with a position in the group, and a position is bind order, so reactors open their sockets in ShardIndex order at startup, each waiting up to 10 seconds for its turn before steering is abandoned. It degrades rather than fails: if the kernel refuses the program - an old kernel, seccomp, a restricted container - or the reactors do not all bind in time, ioxide says so on stderr and Forward stays underneath. With more than 256 reactors it is not attempted. QuicKernelSteeringAttached is true on the reactor that attached it, the last to bind.

Measured on one machine at two reactors on an HTTP/3 benchmark, and documented on QuicRouting as indicative. Unless a large share of clients migrate, Forward costs less overall - hence the default.

Reactors find each other - to forward, and to take their turns - through the ServerConfig instance. Build every reactor of a server from the same instance, with ids 0 to ReactorCount - 1.

Watching it#

foreach (Reactor reactor in reactors)
{
    Console.WriteLine(
        $"r{reactor.ShardIndex}: forwarded {reactor.QuicForwardsSent}, received {reactor.QuicForwardsReceived}, " +
        $"dropped {reactor.QuicForwardsDropped}, stale {reactor.QuicStaleDatagrams}, " +
        $"claims {reactor.QuicPinsOpen} open / {reactor.QuicPinsCreated} made, " +
        $"kernel steering {reactor.QuicKernelSteeringAttached}");
}

Forwards sent and received stay at zero while no client moves. QuicForwardsDropped should stay at zero: it counts datagrams dropped because their owner had 1024 waiting or had stopped, so rising means a reactor is behind. QuicStaleDatagrams counts short headers for IDs that are gone - a migration produces a few every time.

Timers and deadlines#

QUIC lives on millisecond deadlines - loss detection, probe timeouts, delayed acknowledgements, pacing - and a retransmit that waits for a 250 ms tick turns a burst of loss into a storm. So each connection reports its next deadline, and the reactor keeps the earliest across its connections:

• At the top of every loop pass the reactor compares the clock with that deadline - one comparison unless something is due - and runs the due connections' timers. Every datagram a connection receives and every send it makes folds its new deadline back in.
• An idle reactor waits in io_uring_enter until the next ticker run or the earliest QUIC deadline, whichever is sooner, plus a millisecond, so a retransmit fires when it is due rather than at the next tick. A reactor whose last connection has gone stops waking for its deadline.
• The sweep, on the 250 ms ticker, drops a connection whose peer has sent nothing for ReadTimeoutMs - any datagram routed to it counts - silently, as an idle timeout ends a QUIC connection. The server advertises no idle timeout of its own, so this is the bound.
• While the server owes a response - a request stream the peer has finished and the server has not - and the handler still holds the connection, ngtcp2 sends keep-alive PINGs at half the tightest of the read timeout, less up to two ticks, and both sides' idle timeouts (the peer's taken as at least a second). A slow handler is not timed out by either side; once the handler lets go, the pings stop.
• A connection that has not finished its handshake within the engine's handshakeTimeoutMs, 10 seconds by default, is dropped.
• A connection whose timer throws is dropped (QuicEvictReason.TimerFault); the reactor and its other connections carry on.

Sending, and the retention high-water#

ngtcp2 does not copy stream data: it keeps pointers into it until the peer acknowledges it, in case it has to retransmit. So SendStream copies your bytes into native chunks the connection owns, ngtcp2 reads from those, and acknowledgements free them. Bytes the congestion or flow-control window cannot take yet wait in the chain and go out as the windows open; nothing queued is dropped.

What a connection retains - sent and not yet acknowledged, plus not yet sent, across all its streams - is bounded twice:

• SendRetentionBytes, the high-water (16 MiB, at least 256 KiB). At or above it, CanQueueSend is false and a well-behaved producer waits. When acknowledgements bring retention back below it, OnSendCapacityAvailable fires on the reactor - from the path that processes acknowledgements and timers, because a peer downloading a response sends nothing but acknowledgements and the read loop never sees them. Both HTTP/3 layers wait here, for buffered bodies and writers alike, and so does QuicConnectionPipeWriter.FlushAsync. So what a connection retains stays near this figure whatever the size of the response.
• SendRetentionCeilingBytes, the backstop (32 MiB, never below the high-water). A connection that crosses it is closed with INTERNAL_ERROR: its producer ignored the wait, or its peer stopped acknowledging.

Before any stream data, a connection costs about 75 KB (the note on QuicEngine.CreateFactory), most of it the 63 KiB buffer its datagrams are batched in.

How a connection ends#

Whatever the cause, the handler's next ReadAsync returns a snapshot with IsClosed set, and a writer parked at the high-water wakes to find the connection gone. The handler then lets go of its reference with DecRef - the HTTP/3 layers do it for you.

Options#

var config = new ServerConfig
{
    ReactorCount = Environment.ProcessorCount,
    DualStack = false,                       // true: one AF_INET6 socket on :: that takes IPv4 as well
    Tcp = null,                              // QUIC only
    Udp = new UdpOptions
    {
        RecvSlots = 16,                      // datagrams in flight per reactor, ~64 KiB each
        SocketBufferBytes = 8 * 1024 * 1024, // a request; the kernel may grant less
        Gro = true,                          // a burst from one peer arrives as one completion
    },
    Quic = new QuicOptions
    {
        Port = 443,
        LocalCidLength = 8,                  // equal to the engine's cidLength
        ConnectionFactory = engine.CreateFactory(),
        Routing = QuicRouting.Forward,
        PinMigratedPeers = true,
        ReadTimeoutMs = 60_000,
        SendRetentionBytes = 16L << 20,
        SendRetentionCeilingBytes = 32L << 20,
    },
};

sealed record QuicOptions - setting ServerConfig.Quic turns the transport on.

sealed record UdpOptions - the UDP sockets, QUIC's included.

ServerConfig.DualStack (false) binds every socket, TCP and UDP, as one IPv6 socket on :: that takes IPv4 clients too; off, sockets are IPv4 only. ServerConfig.Udp null opens no raw UDP sockets; QUIC still binds its port.

The engine advertises the same transport parameters on every connection: 256 KiB of initial flow-control window per stream and 1 MiB per connection, 1024 bidirectional and 100 unidirectional streams - a window that grows back as streams close, not a lifetime cap - no idle timeout, and on the server a limit of 8 active connection IDs from the client.