TCP connections

Reading and writing

Bytes arrive in buffers the kernel filled and wait in the connection's receive queue until the handler takes them. They leave from the connection's write slab, one flush at a time. Both directions, the options that shape them, and the pipe and stream adapters on top.

Reading: snapshots#

The reactor keeps one multishot receive armed per connection, and appends each completion - a buffer the kernel filled - to the connection's receive queue. ReadAsync returns a RecvSnapshot: the queue's tail at that moment, and IsClosed. It completes at once when anything is queued or the connection has closed; otherwise the next receive resumes it inline.

Complete, MarkClosed and Dispose are public as well, but belong to the reactor: a handler never calls them.

Why every buffer must go back#

By default, receive buffers come from one provided-buffer ring per reactor, registered with the kernel at start-up: ServerConfig.RecvSlots buffers of RecvBufferSize bytes - 4096 of 32 KiB, 128 MiB per reactor. Every receive takes a whole buffer however few bytes arrived, and all the reactor's connections draw from the same ring.

TryGetItem is a dequeue: the connection stops tracking the buffer, and nothing will return it for you. A buffer the handler keeps is gone from its reactor's ring for the life of the process. When the ring runs dry, that reactor's receives end with ENOBUFS; it parks them and re-arms them as buffers come back, so a leak starves a reactor rather than crashing it. A returned buffer is the kernel's to fill again: parse first, and copy what you keep. A return made off the reactor thread goes through a queue and an eventfd wake.

To parse a message that came in several receives as one sequence, dequeue the whole snapshot as memories over the same buffers:

RecvSnapshot snapshot = await conn.ReadAsync();

UnmanagedMemoryManager[] memories = conn.GetSnapshotMemories(snapshot);
ReadOnlySequence<byte> bytes = memories.ToReadOnlySequence();   // a view over the ring buffers
Parse(bytes);                                                     // parse while they are yours
conn.ReturnBuffers(memories);                                     // then hand every one back

conn.ResetRead();

This allocates - an array, and a memory manager per buffer - where the TryGetItem loop does not. A message split across two snapshots still needs a carry: keep its items unreturned until the rest arrives, or copy its start aside. The pipe reader does the first for you.

The receive queue#

TcpOptions.RecvQueueEntries (64) sizes each connection's queue, at 24 bytes a slot: 1.5 KiB per connection object, pooled ones included. A queued receive holds its buffer until the handler takes it, so one connection whose handler stops reading holds at most 64 of its reactor's shared buffers before it is closed. When the queue is full, the receive that does not fit is dropped and the connection closed with RecvOverflowed set, which the pipe reader and the stream report as an IOException. Raise it for a handler that lets its peer get further ahead; from 0.16.268 to 0.17.339 the default was 4096, 96 KiB a connection.

Shared ring or incremental buffers#

Setting ServerConfig.Incremental to an IncrementalOptions gives each connection a small buffer ring of its own (IOU_PBUF_RING_INC, kernel 6.12+) and runs the reactor's incremental loop; the shared-ring knobs go unused.

Handler code is identical: ReturnBuffer and ReturnBuffers route to the right ring. Incremental mode packs small messages densely and keeps each connection's receive memory its own, so one connection cannot drain the others' buffers. It costs a ring registration per connection and, per the README, a page of locked memory per live connection - up to 16 MB per reactor at the default MaxConnections.

Writing#

The write slab and FlushAsync#

Each connection has a write slab: TcpOptions.WriteSlabSize (16 KiB) of native memory, kept across the object's pooled lives. Write(span) copies into it, and since TcpConnection is an IBufferWriter<byte>, a formatter or serializer can write in place:

static void WriteNumber(TcpConnection conn, long value)
{
    Span<byte> span = conn.GetSpan(20);              // room for any long, inside the slab
    Utf8Formatter.TryFormat(value, span, out int written);
    conn.Advance(written);                           // commit what was used
}

FlushAsync hands everything staged to the reactor as one send and completes when the kernel has sent all of it. Sends carry MSG_WAITALL by default, so the kernel carries a short send on itself and a flush completes as one completion; a send that still comes back short is resubmitted from where it stopped. TcpOptions.SendWaitAll = false sends without it, bound to each connection at accept: a short send then completes as it is and the reactor submits the rest, a completion and a submit more per short send. TcpConnection.SendOpFlags shows what a connection uses.

• One flush at a time. A second FlushAsync while one is outstanding throws InvalidOperationException, and so do Write, GetSpan, GetMemory and Advance: await the flush before staging more.
• Nothing staged completes at once.
• It does not throw when the peer is gone. On a closed connection it drops what is staged and completes; a failed send tears the connection down and completes the flush. Check IsClosed before producing more; the pipe writer reports it as FlushResult.IsCompleted.

Responses bigger than the slab#

Zero-copy send#

TcpOptions.ZeroCopySend = true sends with IORING_OP_SEND_ZC: the kernel pins the slab's pages instead of copying them and posts a second completion when it lets go of them, and FlushAsync completes only then, so the next write never touches pages still on their way out. The pinning and the extra completion are paid per send, so it pays only for large responses; TcpOptions says to leave it off for small ones. The repository commits no measurement of it; Playground/Tcp/Big, a 100 KB response with knobs for zero-copy, the slab size and the overflow strategy, is the sample to measure with. It covers the contiguous send only: a Segmented response that spilled goes out as a plain SENDMSG, a connection handed to kernel TLS uses plain sends, and incremental mode never applies it.

Reading a file into the slab#

ReadFileAsync(fileFd, len, fileOffset) reads straight into the write slab at its tail, with an IORING_OP_READ on the connection's reactor; AdvanceWrite(n) makes the bytes part of the next flush. A head and the file behind it leave in one send, with no buffer and no copy between:

static async Task<bool> SendFileAsync(TcpConnection conn, int fileFd, int length)
{
    conn.Write(Encoding.ASCII.GetBytes($"HTTP/1.1 200 OK\r\nContent-Length: {length}\r\n\r\n"));

    for (int offset = 0; offset < length; )
    {
        int n = await conn.ReadFileAsync(fileFd, length - offset, offset);
        if (n <= 0) return false;   // an errno, or the file is shorter than promised: close instead
        conn.AdvanceWrite(n);       // the read placed the bytes; this makes them part of the send
        offset += n;
    }

    await conn.FlushAsync();        // head and body leave in one send
    return true;
}

• It returns the bytes read, 0 at end of file, or a negative errno, and may return fewer bytes than asked for - loop.
• It grows the slab to fit len, and the slab stays that size until the connection is recycled: send big files in chunks.
• One read in flight per connection. On a closed connection it returns 0 without reading.
• The bytes go on the wire as read: right for cleartext and for kernel TLS transmit, where the slab holds plaintext; wrong for BoringSSL's TLS, where it must hold ciphertext. Its own documentation says cleartext only; the Clients/File sample also uses it under kTLS transmit.

The repository's note on Clients/File measured skipping the copy at 1.22× in cleartext and 1.21× under kTLS, on one saturated reactor at 4 KiB. Files covers opening and sharing descriptors with ioxide.file.

Flushing from another thread#

Only a reactor's own thread submits to its ring. FlushAsync called from another thread puts the flush on the reactor's lock-free flush queue (4096 entries) and writes the reactor's eventfd to wake it; the reactor submits the send on its next pass. The send completes on the reactor, and an await that had to wait for it resumes inline there. Writing is not synchronized for you: one writer at a time is still yours to ensure.

Most handlers never take this path: the reactor's synchronization context posts a continuation that completed elsewhere back to the reactor, unless the await says ConfigureAwait(false). The round trip has a price: see Tcp/TaskRun below.

Pipes and streams#

TcpConnectionPipeReader and TcpConnectionPipeWriter adapt a connection to System.IO.Pipelines, and TcpConnectionDualPipe pairs them as the IDuplexPipe that ioxide.http2, ioxide.nghttp2 and the TLS pipes take. The overview's server, now answering each HTTP request:

using System.Buffers;
using System.IO.Pipelines;
using ioxide;

byte[] response = "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 2\r\n\r\nok"u8.ToArray();

var config = new ServerConfig
{
    ReactorCount = Environment.ProcessorCount,
    Tcp = new TcpOptions { Port = 8080 },
};

var threads = new Thread[config.ReactorCount];
for (int i = 0; i < threads.Length; i++)
{
    var reactor = new Reactor(i, config);

    reactor.TcpHandle = async (r, conn) =>
    {
        var pipe = new TcpConnectionDualPipe(conn);
        try
        {
            await ServeAsync(pipe, response);
        }
        finally
        {
            pipe.Input.Complete();   // hands back any receive buffers the reader still holds
            conn.DecRef();
        }
    };

    threads[i] = new Thread(reactor.Run) { Name = $"reactor-{i}" };
    threads[i].Start();
}

foreach (Thread thread in threads) thread.Join();

// Sees an IDuplexPipe and nothing else, so a TlsConnectionDualPipe serves through it unchanged.
static async Task ServeAsync(IDuplexPipe pipe, byte[] response)
{
    while (true)
    {
        ReadResult read = await pipe.Input.ReadAsync();
        ReadOnlySequence<byte> buffer = read.Buffer;

        // One response per complete request head, however the bytes were split across receives.
        var reader = new SequenceReader<byte>(buffer);
        SequencePosition consumed = buffer.Start;
        int answered = 0;
        while (reader.TryReadTo(out ReadOnlySequence<byte> _, "\r\n\r\n"u8, advancePastDelimiter: true))
        {
            consumed = reader.Position;
            answered++;
        }

        // Consumed: the whole heads. Examined: everything, so a partial head waits for more bytes.
        pipe.Input.AdvanceTo(consumed, buffer.End);

        for (int n = 0; n < answered; n++)
        {
            pipe.Output.Write(response);   // straight into the connection's write slab
        }
        if (answered > 0)
        {
            FlushResult flushed = await pipe.Output.FlushAsync();
            if (flushed.IsCompleted) return;   // the connection is gone
        }

        if (read.IsCompleted) return;
    }
}

• The reader chains the received buffers into one ReadOnlySequence<byte>, without copying. AdvanceTo(consumed, examined) returns each buffer once consumption passes it and honours examined: with everything examined, the next read waits for new bytes. A partial request stays held across reads - the carry the raw API leaves to you.
• At steady state neither side allocates: a waiting read or flush chains onto the connection's own value-task source.
• The writer writes into the slab, so everything above applies to it.
• Complete the reader when done, to return what it holds; if a handler forgets, the reactor reclaims the buffers at recycle. Completing the writer only marks it. One reader per connection.

TcpConnectionStream is a Stream over the connection, for SslStream and other Stream APIs. A read copies from the ring's buffers into your array - the copy the Stream contract forces - and returns each buffer once drained. Each WriteAsync stages and flushes, one send per write; Flush() does nothing. One reader and one writer, on the reactor thread, where the synchronous Read and Write must never be called: they block on the asynchronous ones, waiting for a completion only that thread can run. Disposing it returns its held buffer, not the connection.

What they cost#

One run of the repository's bench/any.sh, recorded in bench/results/20260809T181559Z.json: wrk with 64 connections against 2 reactors, a 2-byte body (13 bytes for the last two samples), and CPU as the server process's user and system time per request. One run on one machine: compare the rows, not the absolutes.