TCP connections

TCP connections

Every reactor listens on the server's TCP ports and accepts its own connections. Your handler gets each one as a TcpConnection on the reactor that accepted it: its receives and sends go through that reactor's ring, and their completions resume the handler inline on that reactor's thread.

How it fits together#

ServerConfig.Tcp, a TcpOptions, configures the transport. It defaults to a listener on port 8080; null opens no TCP listener at all, which is what a server that only speaks QUIC wants. Each reactor opens its own SO_REUSEPORT listener on every port, so the kernel spreads new connections across the reactors, and a connection stays on the reactor that accepted it until it closes.

For each accepted connection the reactor calls TcpHandle(reactor, connection) once, on its own thread. The handler runs until an await has to wait; from then on, the reactor's completion loop resumes it directly when its receive or send completes - no thread pool, no queue in between. A handler is ordinary async code: one loop per connection, written top to bottom.

A complete echo server#

using ioxide;
using ioxide.utils;

var config = new ServerConfig
{
    ReactorCount = Environment.ProcessorCount,   // one reactor per core (the default is 12)
    Tcp = new TcpOptions { Port = 7000 },
};

var threads = new Thread[config.ReactorCount];
for (int i = 0; i < threads.Length; i++)
{
    var reactor = new Reactor(i, config);

    // Runs once per accepted connection, on this reactor's thread.
    reactor.TcpHandle = async (r, conn) =>
    {
        try
        {
            while (true)
            {
                // Everything received so far, or the close. Resumes inline on the reactor.
                RecvSnapshot snapshot = await conn.ReadAsync();

                while (conn.TryGetItem(snapshot, out SpscRecvRing.Item item))
                {
                    if (item.HasBuffer)
                    {
                        conn.Write(item.AsSpan());    // copy into the connection's write slab
                        conn.ReturnBuffer(in item);   // and give the receive buffer back
                    }
                }

                await conn.FlushAsync();              // one send for everything staged

                if (snapshot.IsClosed) return;        // the peer closed
                conn.ResetRead();                     // re-arm the read for the next snapshot
            }
        }
        finally
        {
            conn.DecRef();                            // the handler's reference, released once
        }
    };

    threads[i] = new Thread(reactor.Run) { Name = $"reactor-{i}" };
    threads[i].Start();
}

foreach (Thread thread in threads) thread.Join();

Run it and connect with nc 127.0.0.1 7000: what you type comes back. The loop is the whole raw API in miniature:

1. ReadAsync returns a RecvSnapshot: a position in the connection's receive queue, and whether the connection has closed. It completes at once when bytes are already queued.
2. TryGetItem takes each queued receive up to that position. Its bytes sit in a buffer the kernel filled - nothing was copied - and ReturnBuffer hands the buffer back to the reactor's ring.
3. Write copies into the connection's write slab. FlushAsync sends everything staged as one operation and completes when the kernel has sent it.
4. IsClosed on the snapshot means the connection is gone: the snapshot still carries the last bytes received, but nothing more will come. Otherwise ResetRead re-arms the read for the next turn.
5. DecRef, in a finally, lets go of the connection.

The repository's Playground/Tcp/Raw is this loop answering HTTP with a fixed response; Tcp/Pipe, Tcp/Incremental, Tcp/Big and Tcp/TaskRun each change one thing about it.

What the handler owns#

A connection has two owners, each holding a reference: the reactor, for its receive side, and your handler. It is recycled only when both have let go, so it is never reused under a handler that is still running. What belongs to the handler:

• Its reference. Release it with DecRef, once, in a finally. If the peer is still connected, DecRef sends it a FIN once the last flush is out, and the reactor keeps reading until the peer closes, for at most ReadTimeoutMs. When both references are gone the connection is recycled - its descriptor closed, its object pooled for the next accept - so after DecRef the object may already belong to another connection. Never touch it again.
• Its faults. A handler that throws is logged to standard error ([r0] connection handler faulted: ...) and its reference is released for it. The handler's release counts once per connection, whoever makes it, so a finally is safe whatever the handler throws. Before 0.17.339, a handler that released and then threw was released twice, which recycled the connection under the reactor and could later close a descriptor number already handed to another connection.
• The read side. One ReadAsync at a time - a second, while one waits, throws InvalidOperationException - and ResetRead after every snapshot.
• Every buffer it takes. Each item TryGetItem hands out goes back with ReturnBuffer, exactly once. A buffer kept is gone from its reactor's ring for good; why.
• The write slab. One writer, one flush at a time, and no writing while a flush is in progress.

Never block the reactor thread: .Result or .Wait() there waits for a completion only that thread can run. Ordinary awaits are fine - the reactor's synchronization context brings a continuation that completed elsewhere, such as a Task.Run, back to the reactor before it touches the connection again.

Raw, pipes or a stream#

The pipes cost little: in the repository's benchmark the Tcp/Pipe sample used 2.36 us of CPU per request against 2.30 us for Tcp/Raw (the numbers). On top of a connection, TLS terminates through TlsService.AcceptAsync(conn) and serves through a TlsConnectionDualPipe, and HTTP/2 runs over any IDuplexPipe. ioxide ships no HTTP/1.1 library: the samples frame it by hand, in a few lines.

Where to go next#