TCP connections

Lifetime and limits

How connections arrive and how they end - including the clocks that end them when the peer will not - and what bounds how many a reactor holds.

Listeners#

For every port - Port and each of ExtraPorts - every reactor opens a listening socket of its own with SO_REUSEADDR and SO_REUSEPORT, binds it to every interface and arms one multishot accept on it. The kernel spreads new connections across the reactors' listeners by a hash of their addresses and ports, and a connection stays on the reactor that accepted it until it closes. Nothing moves it to a quieter reactor, so a few long-lived, busy connections can load one reactor while others idle. Accepted sockets inherit TCP_NODELAY from the listener - Nagle's algorithm is off - and are opened close-on-exec.

There is no option for the address: listeners bind every interface. A bind that fails - say the port is held by a socket without SO_REUSEPORT - throws from Reactor.Run before the loop starts, to Reactor.OnFault when one is set.

One handler, several ports#

TcpConnection.ListenerPort carries the port a connection arrived on, stamped at accept. One handler branches on it:

var config = new ServerConfig
{
    ReactorCount = Environment.ProcessorCount,
    Tcp = new TcpOptions { Port = 8080, ExtraPorts = [9090] },
};

for (int i = 0; i < config.ReactorCount; i++)
{
    var reactor = new Reactor(i, config);
    reactor.TcpHandle = (r, conn) => conn.ListenerPort == 9090
        ? ServeAdminAsync(r, conn)    // each of these releases conn itself
        : ServePublicAsync(r, conn);
    new Thread(reactor.Run) { Name = $"reactor-{i}" }.Start();
}

The engine never learns what a port means: terminate TLS on one, serve an admin endpoint on another. Playground/Tls/MultiPort serves plaintext and TLS this way, through one serve loop over an IDuplexPipe. An extra port costs work only at accept; a second set of reactors for it would instead double the threads competing for the same cores whenever both ports are busy.

A connection's life#

1. Accept. The reactor takes a connection object from its pool, or makes one with its write slab and receive queue. In incremental mode it also registers the connection's buffer ring, or closes the connection at once if it cannot. It records the connection in its table by descriptor, sets two references - its own and the handler's - stamps ListenerPort, arms a multishot receive and starts TcpHandle.
2. Serving. Receives queue up for the handler; each flush is a send.
3. Ending. One of the events below.
4. Recycling, once both references are gone. The receive is cancelled; buffers still held by the connection's pipe reader or stream, and buffers still queued, go back to the ring (in incremental mode the connection's ring is unregistered instead); the descriptor is closed; the object is reset and pushed back to the pool - or freed, if the pool already holds PoolMax.

Every operation the reactor submits for a connection carries the connection's generation, and recycling bumps it. A completion from a previous life is recognized and dropped, and an await left over from one resolves as closed, so a descriptor number the kernel hands out again never receives the old connection's completions.

How a connection ends#

Teardown waits for the kernel: a connection torn down while the kernel still holds its send keeps its slab and table slot until that send completes, so the slab is never reused under it.

No half-close from the peer#

ioxide has no half-close for the peer. A client that shuts down its sending side while it still expects an answer gets the whole connection torn down: the reactor treats a zero-byte receive as the end. The handler sees the request and IsClosed in the same snapshot, and its answer goes out only if its send reached the kernel before the FIN was seen. A handler that answers inline usually wins that race; one that awaits anything first loses it silently - its flush drops the response and completes. The chaos suite pins this as a choice of design: teardown is clean and the reactor keeps serving, but the response is best effort.

The other direction exists. ShutdownWrite half-closes the server's side, for a protocol layer that is done sending - HTTP's Connection: close:

conn.Write(lastResponse);       // a response whose head says "Connection: close"
await conn.FlushAsync();
conn.ShutdownWrite();           // the FIN goes now; reads still work until the peer closes

With a flush still in flight, the FIN waits for it and goes from the first sweep after it completes, as it does after a DecRef mid-flush. Call it on the reactor thread.

Timeouts#

Two clocks reap connections whose peer has stopped taking part. A sweep on the reactor's 250 ms ticker enforces both, on a busy reactor too, so a connection closes at the first tick after its deadline. 0 turns either off.

The clocks are separate because the read clock cannot see a stalled send: a peer that keeps sending while it has stopped reading restarts it with every receive - the reported case was a websocket written from a background task.

Limits and memory#

A reactor's pool holds on to at most PoolMax objects, each with its write slab and receive queue - and in incremental mode its ring and buffers too.

Out of file descriptors#

An accept that fails for want of descriptors (EMFILE, ENFILE) or memory (ENOMEM, ENOBUFS) would fail again at once if re-armed at once, and used to spin every reactor at full CPU. Now the listener pauses: the reactor logs it once - [r0] accept failed (-24): out of file descriptors or memory, retrying every 250 ms - and the TCP sweep re-arms the accept on its next tick. Connections wait in the listen backlog meanwhile and are accepted once descriptors free up. Any other accept error is logged, and the accept re-armed at once if it ended.