Files and timers

Files

ioxide.file opens every file under a directory once, shares the descriptors with every reactor, and reads them with positional io_uring reads on the reactor that asks. It speaks no HTTP and keeps no copy of any file: serving one is a dictionary lookup and a read on the ring, resumed inline.

dotnet add package ioxide.file

It depends on ioxide core alone. Sending a file without a copy is core's TcpConnection.ReadFileAsync; ioxide.file supplies the descriptors it reads.

What is in it#

All four are in namespace ioxide.file. Every read is an IORING_OP_READ at an offset through the reactor's IRingHost (reference), resumed on the reactor's thread.

A static file server#

Every file under wwwroot - or the directory given as the first argument - on port 8080, from every core:

using System.Runtime.InteropServices;
using System.Text.Unicode;
using ioxide;
using ioxide.file;
using ioxide.utils;

// Every file under the root, opened once, before any reactor starts: one snapshot for the process.
var assets = new StaticAssets(args.Length > 0 ? args[0] : "wwwroot");

var config = new ServerConfig
{
    ReactorCount = Environment.ProcessorCount,
    Tcp = new TcpOptions { Port = 8080, WriteSlabSize = 64 * 1024 },
};

var threads = new Thread[config.ReactorCount];
for (int i = 0; i < threads.Length; i++)
{
    var reactor = new Reactor(i, config);
    reactor.OnStart = r => r.AddService(assets);   // the same snapshot on every reactor

    reactor.TcpHandle = async (r, conn) =>
    {
        StaticAssets files = r.GetService<StaticAssets>();
        try
        {
            while (true)
            {
                RecvSnapshot snapshot = await conn.ReadAsync();

                // Held until the response is out: a reload cannot close these descriptors under the read.
                using (StaticAssets.Lease lease = files.Acquire())
                {
                    bool asked = false, found = false;
                    AssetCache.Asset asset = default;
                    while (conn.TryGetItem(snapshot, out SpscRecvRing.Item item))
                    {
                        if (item.HasBuffer)
                        {
                            if (!asked) { asked = true; found = Find(lease, item.AsSpan(), out asset); }
                            conn.ReturnBuffer(in item);
                        }
                    }

                    if (found)
                    {
                        if (!await SendFileAsync(conn, asset)) return;   // the read failed: close
                    }
                    else if (asked)
                    {
                        conn.Write("HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\n\r\n"u8);
                        await conn.FlushAsync();
                    }
                }

                if (snapshot.IsClosed) return;
                conn.ResetRead();
            }
        }
        finally
        {
            conn.DecRef();
        }
    };

    threads[i] = new Thread(reactor.Run) { Name = $"reactor-{i}" };
    threads[i].Start();
}

// After a deploy, kill -HUP <pid> opens the files again and swaps the new snapshot in.
using var reload = PosixSignalRegistration.Create(PosixSignal.SIGHUP, context =>
{
    context.Cancel = true;
    assets.Reload();
});

foreach (Thread thread in threads) thread.Join();

// The head, then the file read straight into the write slab behind it: no buffer of our own and
// no copy. A slab's worth per send, so a large file never grows the slab.
static async Task<bool> SendFileAsync(TcpConnection conn, AssetCache.Asset asset)
{
    const int Chunk = 60 * 1024;   // leaves room for the head in the 64 KiB slab

    WriteHead(conn, asset);
    for (long offset = 0; offset < asset.Length; )
    {
        int n = await conn.ReadFileAsync(asset.Fd, (int)Math.Min(Chunk, asset.Length - offset), offset);
        if (n <= 0) return false;   // an errno, or the file is shorter than the snapshot says
        conn.AdvanceWrite(n);       // the read placed the bytes; this makes them part of the send
        offset += n;
        if (offset < asset.Length) await conn.FlushAsync();
    }

    await conn.FlushAsync();        // a small file leaves with its head in one send
    return true;
}

// "GET /css/site.css?v=3 HTTP/1.1" -> "/css/site.css", looked up in the leased snapshot.
static bool Find(StaticAssets.Lease lease, ReadOnlySpan<byte> request, out AssetCache.Asset asset)
{
    asset = default;
    if (!request.StartsWith("GET "u8)) return false;

    ReadOnlySpan<byte> target = request[4..];
    int end = target.IndexOfAny((byte)' ', (byte)'?');
    return end > 0 && lease.TryGet(target[..end], out asset);
}

static void WriteHead(TcpConnection conn, AssetCache.Asset asset)
{
    Span<byte> head = conn.GetSpan(256);
    Utf8.TryWrite(head, $"HTTP/1.1 200 OK\r\nContent-Type: {ContentType(asset.Path)}\r\nContent-Length: {asset.Length}\r\n\r\n", out int written);
    conn.Advance(written);
}

// ioxide.file knows nothing of HTTP: the content type is yours to choose.
static string ContentType(string path) => Path.GetExtension(path.AsSpan()) switch
{
    ".html" => "text/html; charset=utf-8",
    ".css"  => "text/css",
    ".js"   => "text/javascript",
    ".json" => "application/json",
    ".svg"  => "image/svg+xml",
    ".png"  => "image/png",
    ".jpg"  => "image/jpeg",
    ".woff2" => "font/woff2",
    ".txt"  => "text/plain; charset=utf-8",
    _       => "application/octet-stream",
};

1. One snapshot for the process, opened before the reactors start, since opening blocks.
2. A lease per request pins the snapshot until the flush completes, so a reload cannot close the descriptor being read.
3. The lookup reads the request bytes where they landed, building no string.
4. ReadFileAsync puts the file behind the head in the write slab and AdvanceWrite commits it: a small file leaves with its head in one send, a larger one 60 KiB a send, and the 64 KiB slab never grows.
5. kill -HUP reloads after a deploy.

The content types are the program's own: ioxide.file has no table of them, nor any other part of HTTP. HEAD, ranges, validators, compression and index files are the server's to add. This one answers one request per read, as the Tcp samples do; Playground/Clients/File frames requests across reads, serves TLS by BoringSSL or the kernel, and has both ways of reading.

The snapshot#

new StaticAssets(root) walks the root and the directories under it and opens each file read-only, for the snapshot's life. A file's key is its path under the root with a leading slash, /css/site.css.

• Left out: symbolic links, which could point outside the root, and hidden and system entries. On Linux, .NET counts a name starting with a dot as hidden: .env is never served, nor anything under .well-known.
• Exact lookups, case-sensitive, with nothing decoded or normalised: %20, .., // and a trailing slash are matched as written, and miss; / finds no index file. The byte overload also misses on an empty path, one over 1024 bytes, or a non-ASCII byte. A request can only name a file the snapshot holds, so no path leads out of the root.
• Shared without a lock: each read carries its own offset, so every reactor reads the same descriptor at once.
• Trusted, not re-checked. Length is the size when the snapshot was built - a file edited in place can disagree with it - and the tree's additions and deletions show only after a reload. Deploy by writing the files, then reloading.
• One descriptor per file for the snapshot's life: mind RLIMIT_NOFILE on a large tree, and that a reload holds two snapshots' worth for a moment.

A missing root throws DirectoryNotFoundException, and a file that cannot be opened makes the constructor throw too.

Reloading#

Reload() builds a fresh snapshot of the same root and swaps it in atomically: a request sees the whole old tree or the whole new one. The old snapshot's descriptors close when its last lease is disposed - no timer, no grace period. Reloads are serialised by a lock and safe from any thread. If the rebuild throws, the served snapshot stays, standard error gets [ioxide] asset reload failed, keeping current snapshot: ..., and the descriptors the failed rebuild had opened are left for the garbage collector to close.

• StaticAssets.Lease has the same two TryGet overloads. Dispose it exactly once: it is a struct, each Dispose releases a reference, and disposing a copy too frees the snapshot early.
• StaticAssets.TryGet takes no lease: safe only where Reload never runs.
• Dispose StaticAssets only when nothing will call Acquire again: on a disposed instance, Acquire never returns.

Reading a file#

Into the write slab#

ReadFileAsync(fd, length, offset) reads straight into the connection's write slab at its tail, saving a buffer, the pool that would lend it, and a copy. Reading and writing has its full contract - short reads, 0 at the end of the file or on a closed connection, one read in flight per connection. The bytes go out as read, which decides where it is legal:

• Cleartext, and kernel TLS transmit (TlsOptions.KernelTx), where the slab holds plaintext and the kernel makes the records. The method's documentation says "cleartext only", but the Clients/File sample uses it under kernel TLS as well; with KernelTx on, a connection whose kernel TLS cannot be set up fails its handshake rather than falling back to BoringSSL.
• Not BoringSSL's TLS, where the slab must hold ciphertext: the file would leave in the clear, and the sample refuses the combination. For one connection, the slab path is safe when it has no TlsSession, or its session's KernelTx is true.

Through a reader's buffer#

Bytes that must change on the way - encrypted by BoringSSL, compressed - are read into a buffer first. AssetReader.CreatePool, from OnStart, builds a reactor's readers and registers them as a service, RingPool<AssetReader>:

reactor.OnStart = r =>
{
    r.AddService(assets);
    AssetReader.CreatePool(r, readers: 4, bufferBytes: 256 * 1024);   // RingPool<AssetReader>, a service
};

readers bounds the reads in flight: a request that finds every reader out waits its turn, in order, without a thread. bufferBytes is each reader's native buffer, allocated once. ReadAsync(fd, offset) always asks for a whole buffer, so a larger file takes a read per buffer at advancing offsets:

// Under BoringSSL the slab must hold ciphertext, so the file passes through a buffer and
// TlsSession.Write encrypts it into the slab. The head went out the same way, before this.
static async Task<bool> SendThroughBufferAsync(TcpConnection conn, TlsSession tls,
    RingPool<AssetReader> readers, AssetCache.Asset asset)
{
    AssetReader reader = await readers.RentAsync();   // waits in line, without a thread, if all are out
    try
    {
        for (long offset = 0; offset < asset.Length; )
        {
            int n = await reader.ReadAsync(asset.Fd, offset);   // up to reader.Capacity bytes
            if (n <= 0) return false;
            tls.Write(conn, Bytes(reader.Buffer, n));
            await conn.FlushAsync();
            offset += n;
        }
        return true;
    }
    finally
    {
        readers.Return(reader);   // before the next request rents it
    }
}

// The buffer is native memory; an async method cannot hold the pointer itself.
static unsafe ReadOnlySpan<byte> Bytes(nint buffer, int length) => new((void*)buffer, length);

One file: RingFile#

using RingFile file = RingFile.Open(reactor, "/srv/www/index.html");   // a blocking open, once
nint buffer = Marshal.AllocHGlobal((int)file.Length);                  // native memory you own
try
{
    int n = await file.ReadAsync(buffer, (int)file.Length, offset: 0);   // bytes, 0 at the end, or -errno
}
finally
{
    Marshal.FreeHGlobal(buffer);
}

Open is a blocking open(), read-only and close-on-exec, so it belongs in OnStart; it throws InvalidOperationException when it fails. Length is measured at the open. The memory ReadAsync reads into is yours, and must stay where it is until the read completes. One read at a time - a second while one is in flight throws InvalidOperationException - and Dispose closes the descriptor.

Hits, misses and memory#

ioxide.file holds no file bytes: each read leaves with the reactor's next batch of submissions, and the kernel supplies the bytes. Nothing in ioxide treats a page-cache hit differently from a miss, warms the cache or reads ahead - but the kernel does, and the reactor feels it:

• A hit is copied during the reactor's submit, on its thread - the submit "completes it", in the words of the repository's research notes (Research/Magpie/Ring.cs). A large read is a long copy, and no other connection on that reactor is served until it ends: read in chunks, as the server above does.
• A miss does not hold the reactor. The read completes when the disk has delivered; the reactor's wait returns on any completion, so it serves everything else meanwhile, and only the awaiting handler waits.

ioxide.file used to keep every small file's whole HTTP/1.1 response in memory. That went - it tied the snapshot to HTTP/1.1 and pinned file bytes - and the ring-read path measured about 13% under it (loopback, 4 reactors): the price of not caching. To serve a hot file from memory, read it once into an array of your own and Write it from there.

What it costs#

bench/file-matrix.sh measured the Clients/File sample in October 2026 - one saturated reactor, loopback, best of 2 passes - in requests per second and CPU per request:

At 4 KiB, skipping the copy is worth 1.23× in cleartext and 1.11× under kernel TLS, and kernel TLS into the slab leads BoringSSL through a buffer by 1.09×. At 64 KiB the kernel's per-byte encryption outweighs the copy saved, and BoringSSL through a buffer wins by 1.38×; the crossover was not measured. In the full suite (bench/results/20260809T181559Z.json, 2 reactors, 64 connections) the sample served a 4 KiB file through the slab at 670k requests per second and 3.01 us of CPU each, against 877k and 2.30 us for Tcp/Raw's 2-byte answer from memory.

What you choose#

The rest is fixed: what the walk leaves out, the 1024-byte limit on a byte lookup, and the absence of HTTP.

Where to go next#