Core runtime

How the core works

A reactor is a loop around one io_uring ring: each turn it hands the kernel everything staged, waits for completions, and runs the code waiting on each one - inline, on its own thread.

One turn of the loop#

1. Drain the hand-off queues: receive buffers, flushes and finished connections handed back by other threads, and ring operations other threads asked for.
2. Run the posted callbacks - ScheduleOnReactor and the synchronization context share one queue - including any they post while it drains.
3. Re-arm starved receives, if buffers have come back, and fire due QUIC timers.
4. Submit and wait. One io_uring_enter submits every SQE staged since the last and waits for at least one completion - at most until the next tick or the earliest QUIC deadline.
5. Read the clock once for the batch: a read per completion cost 4-9% of throughput on the small-response samples (Reactor.Tcp.Sweep.cs).
6. Dispatch the batch: read the completion queue's tail once, route every completion, publish the head once.
7. Run the tickers, if a tick is due.

Handler code runs in step 6. A completion that ends a handler's await resumes the handler right there, and what the handler submits next - its response's send, its next receive, a query - is staged as an SQE that leaves with the next turn's io_uring_enter. A batch of completions becomes a batch of submissions, and one system call carries them.

A wait that runs out returns -ETIME, which is not an error, and the loop carries on from EINTR, EAGAIN and EBUSY. Any other error from io_uring_enter ends the reactor through OnFault.

The ring#

The ring is created on the reactor thread with two flags that need Linux 6.1. SINGLE_ISSUER promises the kernel one submitting thread, so it skips locking the submission queue. DEFER_TASKRUN holds completion work until the thread enters the kernel to collect it, so the kernel never interrupts the reactor mid-turn - and it is why the ring is created, entered and torn down on one thread, all inside Run. NO_SQARRAY (6.6) saves a store per submission; a kernel that refuses it gets a ring without it.

• Size. RingEntries (8192) is the submission queue's depth, at most 32768; the kernel makes the completion queue twice as deep. A submission queue that fills mid-turn is flushed with an enter that does not wait, and the reactor retries up to 16 times before it gives up.
• A ring that cannot be created says why: a kernel older than 6.1, too many entries, RLIMIT_MEMLOCK, or io_uring disabled by the kernel.io_uring_disabled sysctl or a container's seccomp profile. ENOMEM is retried five times over 155 ms, because the kernel frees a closed ring's memory asynchronously: 30,000 rings created and closed back to back failed 14,788 times, every failure clearing on a retry 5 ms later (Ring.cs).
• Routing. Each SQE carries its route in user_data: a kind, a 16-bit connection generation, and a descriptor or a slot. A TCP completion finds its connection in a table indexed by descriptor and is dropped if the generation has moved on - a straggler from an earlier connection on a reused descriptor. Every IRingHost operation holds a slot in a table of completions instead, OpSlots (4096) long at first and doubled when full.

Completions resume inline#

Every ring-backed await in ioxide is a reusable IValueTaskSource that the reactor completes during dispatch. RingOpSource is the general one: it is both the IRingCompletion the reactor calls and the source of the ValueTask<int> the caller awaits, so wrapping an operation takes three lines:

// One ring operation as an awaitable: arm the source, submit with it as the completion,
// return its task. RingSocket's connect, send and receive are these same three lines.
sealed class PositionalReader(IRingHost host, int fd)
{
    private readonly RingOpSource _op = new();

    public ValueTask<int> ReadAsync(nint buffer, int length, long offset)
    {
        ValueTask<int> pending = _op.Prepare();
        host.SubmitRead(fd, buffer, length, offset, _op);
        return pending;
    }
}

Two settings make the resume inline. RunContinuationsAsynchronously = false makes SetResult call the continuation directly. And OnCompleted clears the UseSchedulingContext flag that await passes: left set, the source would capture the reactor's synchronization context and post every resume to its queue, because RunContinuationsAsynchronously only governs the case with no context (TcpConnection.Read.cs). A connection's reads and flushes, the pipes, and the pg, Redis and HTTP clients do the same.

The reactor frees an operation's slot before it calls Complete, so the resumed code can submit its next operation at once, and Prepare throws while an operation is still in flight. Nothing is allocated per operation. Complete runs inside the loop's dispatch, so an exception that escapes it ends the reactor; an async method resumed from it keeps its exceptions in its own task.

The reactor's synchronization context#

Run installs a ReactorSynchronizationContext on the reactor thread before anything else. An await in reactor code captures it, so a continuation that would otherwise run on the thread pool - after Task.Run, Task.Delay, the BCL's HttpClient - is posted back and runs on the reactor, where the connection's state is safe to touch. A task completed on the reactor thread resumes its awaiter inline instead - or through the queue, if it was made with RunContinuationsAsynchronously. The ring's own awaitables bypass the context, as above: they complete on the reactor already.

A post always queues; it never runs the callback on the spot. What follows depends on the posting thread:

The same eventfd carries every other hand-off - a buffer returned, a flush, a connection released, a ring operation submitted from another thread - and Stop. Otherwise the loop wakes for a completion, or when its wait runs out. Send runs the callback at once on the reactor thread; from another thread it posts it and blocks until the reactor has run it. An exception from a posted callback, an async void method's included, is logged and the loop carries on.

What leaving the reactor costs#

Three Playground samples answer the same request three ways. The repository's bench measured them in one run - 2 reactors, 64 connections, wrk, 10 seconds a sample, on one machine (bench/results/20260809T181559Z.json, commit 1f2e52c):

CPU per request is the server process's CPU time divided by the requests it served; process CPU is that time as a share of two cores.

• Task.Yield does not leave. With a synchronization context current, .NET posts the continuation to it rather than to the thread pool - here, to the reactor's own queue, with no system call. That is why Tcp/Hop costs what Tcp/Raw costs; the trip through the thread pool its header describes does not happen.
• Task.Run leaves and comes back: a work item on the pool, then a post home with an eventfd write to wake the reactor and an eventfd read on it. The CPU per request doubled and the process's CPU rose from 101% to 187% of two cores, the serialisation itself included.
• Task.Delay comes back the same way, from the runtime's timer queue. A deadline that stays on the ring is one IORING_OP_TIMEOUT - see timers.
• Blocking hangs. A reactor blocked in .Result reads no completions and runs no posted continuations, so if what it waits for needs either - a ring operation, or an async method that resumes on the reactor - it never finishes. Whatever else it waits for, every connection on the reactor waits too.
• A loop of yields starves the reactor. The post queue is drained until it is empty, and each yield posts the next continuation while it drains, so the loop never gets to io_uring_enter: no completion is read and no submission leaves until the yielding stops. To let I/O in, await something on the ring.

Work from other threads#

Only the reactor thread submits to the ring. The entry points other threads may call check the caller's thread: on the reactor they act directly, anywhere else they queue the work and wake the reactor.

A ring operation submitted from another thread is queued, submitted on the next turn and completed on the reactor thread like any other, so the code after the caller's await usually resumes on the reactor - and on the caller's thread or the thread pool when the operation wins the race against the await. Rather than drive ring operations from outside, post the whole piece of work to the reactor and hand its result back, as ioxide.Kestrel does for an ASP.NET Core endpoint:

static Task<T> RunOnReactor<T>(Reactor reactor, Func<Reactor, Task<T>> work)
{
    if (reactor.OnReactorThread)
    {
        return work(reactor);
    }

    var result = new TaskCompletionSource<T>(TaskCreationOptions.RunContinuationsAsynchronously);
    reactor.ScheduleOnReactor(_ => _ = RunAsync(), null);
    return result.Task;

    async Task RunAsync()
    {
        try
        {
            result.SetResult(await work(reactor));
        }
        catch (Exception e)
        {
            result.SetException(e);
        }
    }
}

The work runs entirely on the reactor. RunContinuationsAsynchronously keeps the caller's continuation off the reactor thread when the result is set.

Tickers and timeouts#

No timer drives the tickers. The loop's wait is bounded by the next tick, so an idle reactor wakes every 250 ms, and a busy one checks the clock against the tick after every batch. A due tick runs the tickers in the order they were added and sets the next one TickMs later; a late tick is not made up. ioxide registers its own:

• The TCP sweep, on a reactor with TCP enabled: it closes connections past ReadTimeoutMs or SendTimeoutMs, sends FINs that waited for a flush, and re-arms accepts paused for want of descriptors (Lifetime and limits).
• The QUIC idle sweep, with QUIC enabled. QUIC's loss and retransmission timers do not wait for a tick: the earliest one bounds the loop's wait as well, and due ones fire at the top of the turn (QUIC on the ring).
• TlsService's handshake timeouts, and the client pools' sweeps.

So a timeout in seconds is checked once a tick and fires at the first tick after its deadline. A precise deadline is a ring timeout: IRingHost.SubmitTimeout puts an IORING_OP_TIMEOUT on the ring, the kernel holds it, and it completes with -ETIME on the reactor thread. ioxide.timer wraps it.

Receive buffers#

A TCP connection's receive is multishot with provided buffers: armed once, it completes each time data arrives, the kernel choosing a buffer and naming it in the completion. The handler reads the bytes where they landed and hands the buffer back. Where the buffers come from is set once, by ServerConfig.Incremental:

A receive that finds no buffer does not close the connection: the kernel ends it with ENOBUFS, the reactor parks the connection and re-arms it as soon as a buffer comes back. The handler's side is the same in both modes (Reading and writing). UDP has a ring of its own: UdpOptions.RecvSlots buffers of about 64 KiB, one datagram or GRO train each, shared by the reactor's UDP sockets.

Outbound sockets: RingSocket#

The network clients - ioxide.httpclient, ioxide.pg, and StackExchange.Redis through ioxide.redis.stackexchange - open their connections as a RingSocket: a TCP socket whose connect, sends and receives are IRingHost operations on the reactor that made it. It is also where a client of your own starts. This program fetches the quick start's response:

using System.Runtime.InteropServices;
using System.Text;
using ioxide;

// No listener: the client below is all that runs on this ring. It brings its own buffers,
// so the shared receive ring, which only accepted connections draw from, stays small.
var config = new ServerConfig { ReactorCount = 1, Tcp = null, RecvBufferSize = 4096, RecvSlots = 256 };
var done = new ManualResetEventSlim();

var reactor = new Reactor(0, config)
{
    OnStart = r => _ = FetchAsync(r),   // started on the reactor thread; completes once the loop runs
};
new Thread(reactor.Run) { IsBackground = true }.Start();
done.Wait();

async Task FetchAsync(Reactor r)
{
    try
    {
        using RingSocket socket = RingSocket.CreateTcp(r);
        int rc = await socket.ConnectAsync("127.0.0.1", 8080);   // an IPv4 literal: no DNS on a reactor
        if (rc < 0) throw new IOException($"connect: errno {-rc}");

        // The kernel reads and writes this memory while an operation is in flight, so it must
        // not move: a pinned array never does.
        byte[] buffer = GC.AllocateArray<byte>(4096, pinned: true);
        nint address = Marshal.UnsafeAddrOfPinnedArrayElement(buffer, 0);

        int length = Encoding.ASCII.GetBytes("GET / HTTP/1.1\r\nHost: localhost\r\n\r\n", buffer);
        for (int sent = 0; sent < length; )
        {
            int n = await socket.SendAsync(address + sent, length - sent);   // may send part
            if (n < 0) throw new IOException($"send: errno {-n}");
            sent += n;
        }

        int received = await socket.RecvAsync(address, buffer.Length);   // bytes, 0 at end, or -errno
        if (received < 0) throw new IOException($"recv: errno {-received}");
        Console.WriteLine(Encoding.ASCII.GetString(buffer, 0, received));
    }
    catch (Exception e)
    {
        Console.WriteLine(e.Message);
    }
    finally
    {
        done.Set();
    }
}

Each call is one ring operation and returns what the ring returns: a byte count, 0 for a connect that succeeded or a receive at end of stream, or a negative errno; a send may be partial. The memory must stay where it is until the operation completes - native memory, or a pinned array as here. The reference has the rest, including IPv6.