HTTP client / Under the hood

How ioxide.httpclient works

A pool belongs to one reactor. Its connections are opened, written and read on that reactor's ring, a sweep on the reactor's tick keeps them healthy, and a request that has to wait for one waits on the reactor too - no timer, no thread pool.

One reactor's pool, in 3D#

Drag to turn the view, scroll to zoom. Each box on the ring is a connection, and the line from it to the origin is the connection itself - bright while it carries a request, red once the origin has closed it. Each sphere is a request, and the trail behind it is where it has been: light on the way out, green on the way back. Trails fade a few seconds after their request ends; Trails: keep leaves the last 80 on screen. The pulse spreading from the middle is the sweep. Time is slowed down - a tick is a second here, 250 ms in ioxide - and the pool has 8 connections.

• Origin closes idle connections is a keep-alive timeout at the origin: the idle connections get a red halo, because their FIN sits unread - nothing reads an idle connection. The next pulse finds them and closes them, and a warm pool opens replacements one per tick. A request that arrives before that pulse takes a dead connection and fails, which is the window the sweep leaves; at a real tick of 250 ms it is a short one.
• Burst of 12 sends more requests than there are idle connections. Each one that finds none opens its own while the pool has room, and the rest queue on the left until a connection is handed to them.
• On demand is one origin's pool inside RingHttpClient: nothing until a request asks, connections that close once idle, and the pool itself dropped when nothing is open and nobody is inside.
• MaxLifetimeMs retires a connection once it is old enough - after its request, or from the sweep while idle - and its replacement resolves the name again.

Where the work runs#

Everything a request does runs on its reactor's thread. The client submits each connect, send and receive to the reactor's ring through IRingHost, and the completion resumes the waiting code inline, as the reactor reads the completion queue - no thread switch on the way out, none on the way back. Anything that completes elsewhere is posted back to the reactor by its synchronization context; the only such work in the client is the system resolver's lookup when a connection to a name opens.

That is why a pool is per reactor and needs no lock: one thread ever touches it. It is also why it must be created on its reactor, from OnStart, which the reactor runs on its own thread before it enters its loop.

A request, step by step#

1. Acquire. The pool takes the most recently used idle connection. Reusing the newest keeps a small set of connections warm and lets the oldest go idle long enough for the sweep to close them. With none idle, the request opens a connection of its own if the pool has fewer than PoolSize open or opening, and no more opens in flight than requests waiting, this one included - so a burst opens in parallel. After a failed connect, opens pause for 200 to 400 ms, so a dead origin is not hammered. Then the request waits in a FIFO queue, with a deadline AcquireTimeoutMs from when it started.
2. Send. The head is written into the connection's send buffer: the request line, the host line built when the connection opened, content-length when there is a body, and your headers. A body that fits in the buffer goes out in the same send; a larger one is sent straight from your memory after the head, without a copy.
3. Read. Bytes are received into the connection's buffer, which grows when a response needs more, up to MaxResponseBytes. Glyph11's hardened parser decides when the head is complete and what frames the body; interim 1xx responses are skipped. Headers and body are copied into the response's own pooled buffer, header names lowercased.
4. Release. Because the response owns its bytes, the connection is free the moment it is read - not when you finish with the response. A connection that broke, that the origin said to close, that ended its body with the close, or that is older than MaxLifetimeMs, is closed. Otherwise it goes straight to the first waiting request, or back to the idle list with the time it went idle.

Opening a connection#

Pick a scenario: a name whose IPv6 address never answers, one whose IPv6 address refuses, a certificate issued for another name, an address over plain http, and a name that does not resolve. Each packet leaves its path drawn - amber for the resolver, white for TCP, violet for TLS, blue and green for the request and its answer - so the finished exchange reads as a whole. Step goes one step at a time.

Why TCP_USER_TIMEOUT and not a timer: a ring operation cannot be abandoned while the kernel still holds it, and a shutdown() on a socket that is still connecting does not end an io_uring connect - io_uring issues the connect again and the kernel starts a new one. The kernel's own bound has nothing to race.

The sweep#

HttpClientPool.Start registers the pool's sweep as a reactor ticker; RingHttpClient registers one that sweeps every origin's pool. The reactor runs its tickers every 250 ms, waking from its wait for them even when idle.

1. Every tick: a request that has waited past its acquire deadline is failed. This is the pool's only timer.
2. Every SweepIntervalMs (250 ms by default, rounded up to whole ticks), each idle connection is closed if it is broken, has been idle longer than IdleTimeoutMs, is older than MaxLifetimeMs, or was closed by the origin - found with a poll() for POLLRDHUP with a zero timeout, one per idle connection.
3. Then it refills, one connection per sweep behind the same backoff: a warm pool toward PoolSize, an on-demand pool only while requests are waiting.

Waiting for a connection#

In a pool that is the bottleneck - a reverse proxy with more requests in flight than upstream connections - most requests wait. A waiting request queues a TaskCompletionSource with its deadline and awaits it. A release on the reactor completes it, and its continuation goes into the reactor's own queue: no thread, no syscall.

It used to wait on Task.WhenAny(waiter, Task.Delay(timeout)). The waiter's continuation was WhenAny's, which has no context, so every hand-over went through the thread pool and came back to the reactor by an eventfd write - on top of a timer, a cancellation source and a WhenAny per wait. On the Clients/Http sample at 8 upstream connections per reactor, removing it took the sample from 8.97 to 6.14 us of CPU per request; the cost of waiting, against a pool large enough that nobody waits, fell from 3.7 us to 0.4 us.

What each part costs#

Keeping a dead connection's object to reuse it would save the 0.2 us; a closed TCP connection cannot be revived, so the connect and the handshake are always paid again. What could be saved for TLS is the handshake's CPU, through session resumption, which the client does not do yet.

Benchmarks#

wrk against the Clients/Http sample - wrk, the sample, the client, a Tcp/Raw origin - at 2 reactors and 64 connections, with the sample's 8 upstream connections per reactor; 6 rounds, interleaved, on one machine. Before and after the rework that brought names, IPv6 and a pool that waits on the reactor, released in 0.17.339:

The repository's bench/any.sh runs this sample, and bench/ab.sh compares two git refs with Bench.Clients, a load driver for the client itself.