TLS and kTLS

TLS and kTLS

TLS termination for ioxide's TCP connections. BoringSSL, which ships inside the package, runs the handshake over the reactor's own ring and then encrypts and decrypts in user space; the Linux kernel can take over the record encryption when you ask it to.

It is part of the ioxide package, in the ioxide.tls namespace - nothing else to install. The package carries its own BoringSSL, libioxide_boringssl.so for linux-x64, which needs only libc and is loaded the first time TLS is used; the system's OpenSSL, which versions up to 0.17.340 used, plays no part. Kernel TLS also needs the Linux tls module. QUIC carries its own TLS inside the transport and uses none of this (HTTP/3 and QUIC).

How it fits together#

• One TlsService per reactor. TlsService.Start(reactor, options), called from OnStart, builds the BoringSSL contexts - certificates, protocol settings, ALPN - and registers the service on that reactor. A handler fetches it with r.GetService<TlsService>(). Reactors share nothing, so each builds its own.
• A handshake per connection. await service.AcceptAsync(conn) runs the server handshake on an accepted connection and returns its TlsSession.
• A session per connection. It decrypts what arrives and encrypts what the handler writes - or leaves a direction to the kernel, and says so. It belongs to the reactor's thread, and is disposed when the connection ends.

A handler reads and writes through the session in one of two ways: the raw API, on the connection's receive queue and write slab, or TlsConnectionDualPipe, an IDuplexPipe of plaintext.

A complete HTTPS server#

Every reactor terminates TLS on one port and answers each request with a fixed response, through the pipe:

using System.Buffers;
using System.IO.Pipelines;
using System.Text;
using ioxide;
using ioxide.tls;

// A PEM certificate chain (leaf first) and its private key.
string certPath = args.Length > 0 ? args[0] : "server.crt";
string keyPath  = args.Length > 1 ? args[1] : "server.key";
ushort port     = args.Length > 2 ? ushort.Parse(args[2]) : (ushort)8443;

var config = new ServerConfig
{
    ReactorCount = Environment.ProcessorCount,
    Udp          = null,                          // a TCP-only server
    Tcp          = new TcpOptions { Port = port },
};

var tlsOptions = new TlsOptions
{
    CertificatePath = certPath,
    KeyPath         = keyPath,
    Alpn            = ["http/1.1"],               // the default, written out
};

byte[] response = Encoding.ASCII.GetBytes(
    "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 13\r\n\r\nhello, https\n");

var threads = new Thread[config.ReactorCount];

for (int i = 0; i < threads.Length; i++)
{
    var reactor = new Reactor(i, config);

    // One TlsService per reactor, made on the reactor's own thread before its loop starts.
    reactor.OnStart = r => TlsService.Start(r, tlsOptions);

    reactor.TcpHandle = async (r, conn) =>
    {
        try
        {
            // The handshake runs on this reactor's ring and resumes here, on its thread.
            TlsSession tls = await r.GetService<TlsService>().AcceptAsync(conn);

            // Plaintext in, plaintext out. Disposing the pipe sends close_notify and frees the session.
            await using var pipe = new TlsConnectionDualPipe(conn, tls);
            await ServeAsync(pipe, response);
        }
        catch (Exception e)
        {
            // Handlers run fire-and-forget: a failed handshake would otherwise vanish.
            Console.Error.WriteLine($"connection failed: {e.Message}");
        }
        finally
        {
            conn.DecRef();
        }
    };

    threads[i] = new Thread(reactor.Run) { Name = $"reactor-{i}" };
    threads[i].Start();
}

Console.WriteLine($"https://127.0.0.1:{port}/ on {config.ReactorCount} reactors");

foreach (Thread thread in threads)
{
    thread.Join();
}

// One response per complete request head. TLS delivers records, not requests, so a request
// split across two records arrives in two reads and must still be answered once.
static async Task ServeAsync(IDuplexPipe pipe, byte[] response)
{
    while (true)
    {
        ReadResult read = await pipe.Input.ReadAsync();

        int answered = 0;
        SequencePosition consumed = read.Buffer.Start;
        var reader = new SequenceReader<byte>(read.Buffer);
        while (reader.TryReadTo(out ReadOnlySequence<byte> _, "\r\n\r\n"u8, advancePastDelimiter: true))
        {
            consumed = reader.Position;
            answered++;
        }

        // Whole requests consumed, everything examined: a partial head waits for more bytes.
        pipe.Input.AdvanceTo(consumed, read.Buffer.End);

        for (int n = 0; n < answered; n++)
        {
            pipe.Output.Write(response);
        }

        if (answered > 0)
        {
            await pipe.Output.FlushAsync();   // encrypted on the way into the write slab, then sent
        }

        if (read.IsCompleted)
        {
            return;
        }
    }
}

Make a self-signed pair to try it, start it, and call it:

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes -days 30 \
  -keyout server.key -out server.crt -subj "/CN=localhost" -addext "subjectAltName=DNS:localhost"

dotnet run -c Release -- server.crt server.key 8443

curl --cacert server.crt https://localhost:8443/

• A handshake can fail, and AcceptAsync then throws IOException: TLS handshake failed: ... with BoringSSL's reason - NO_SHARED_CIPHER, UNSUPPORTED_PROTOCOL - for a client it cannot agree with, and connection closed during TLS handshake for one that closes or runs out the timeout. The session is already freed. Catch it anyway - handlers run fire-and-forget, so an exception nobody observes disappears.
• Answer per request, not per read. TLS hands back records: one request can arrive in two reads, and two requests in one. ioxide does not parse HTTP, so the loop frames requests itself.
• Disposing the pipe commits and flushes what is left, sends close_notify when it can - also in answer to the client's own - and frees the session. conn.DecRef() then lets the reactor recycle the connection.

Playground/Tls/OpenSslPipes is this server with every knob written out, and Playground/Tls/MultiPort serves plaintext on one port and TLS on another with one loop - a branch on conn.ListenerPort is the whole difference. The Tls/OpenSsl samples keep the name of the library they were written for, and run on BoringSSL.

The handshake, on the ring#

BoringSSL runs the server handshake against two memory buffers (BIOs) instead of a socket, and AcceptAsync carries the bytes between them and the connection:

1. SSL_accept runs until it needs input, and its error is read at once, before anything is awaited: BoringSSL keeps its error queue per thread, the thread is shared by every connection on the reactor, and the next connection's call clears it.
2. The server's flight that BoringSSL wrote is moved into the connection's write slab and sent with the ordinary flush.
3. The next receive from the connection's queue is written into BoringSSL's input buffer, the ring buffers go straight back, and the loop repeats.

Every await resumes inline on the reactor, like any other - no thread, no blocking socket call. When the handshake is done, AcceptAsync records the ALPN result and the client certificate, decrypts any application data that came with the client's last flight, hands a direction to the kernel if the options ask for it (the handoff), and returns. There is no renegotiation: a BoringSSL server never renegotiates.

The handshake timeout#

A handshake read has no deadline of its own: a peer that connects and says nothing, or sends its ClientHello a byte at a time, would hold a connection, an SSL object and its buffers before proving anything. HandshakeTimeoutMs (10 000 by default) bounds it. The service checks its handshakes on the reactor's 250 ms ticker, and one past its deadline has its socket shut down - the peer gets a FIN and the reactor's pending receive completes - so AcceptAsync throws connection closed during TLS handshake at the first tick after the deadline. 0 turns the sweep off; a negative value is refused.

ALPN#

Alpn lists the protocols the port serves, most preferred first; the default is ["http/1.1"]. The server walks its own list and takes the first protocol the client also offered, so the order is the policy, as in nginx and Kestrel. A client that offers nothing on the list gets no ALPN extension back and carries on without one instead of being refused, and NegotiatedAlpn is null. Each name is 1 to 255 ASCII bytes and the list may not be empty; Start throws ArgumentException otherwise. One port serving HTTP/2 and HTTP/1.1 branches on the result:

// In OnStart: Alpn = ["h2", "http/1.1"] - a client offering both gets h2.
reactor.TcpHandle = async (r, conn) =>
{
    TlsSession? tls = null;
    try
    {
        tls = await r.GetService<TlsService>().AcceptAsync(conn);

        if (tls.NegotiatedAlpn == "h2")
        {
            await using var pipe = new TlsConnectionDualPipe(conn, tls, ownsSession: false);
            await new Http2Connection(pipe).RunBufferedAsync(_ => Http2Response.Text("hello, h2\n"));
            return;
        }

        // "http/1.1", or null when the client offered nothing this port serves.
        // ... an HTTP/1.1 loop on the same session ...
    }
    finally
    {
        tls?.Dispose();
        conn.DecRef();
    }
};

Playground/Http2/Tls is the whole program, HTTP/1.1 loop included. HTTP/2 itself is in HTTP/2.

Two ways to use a session#

The pipe is what Http2Connection, Nghttp2Connection and a framework take. The raw loop, from Playground/Tls/OpenSsl, keeps decrypted bytes in a small buffer (Carry) until they make a whole request:

reactor.TcpHandle = async (r, conn) =>
{
    TlsSession? tls = null;
    var carry = new Carry();   // decrypted bytes not yet framed into requests

    try
    {
        tls = await r.GetService<TlsService>().AcceptAsync(conn);

        // A request that came in with the client's last handshake flight is already decrypted.
        carry.Append(tls.DrainPlaintext());
        if (Answer(conn, tls, carry, response))
        {
            await conn.FlushAsync();
        }

        while (true)
        {
            RecvSnapshot snapshot = await conn.ReadAsync();

            unsafe
            {
                while (conn.TryGetItem(snapshot, out SpscRecvRing.Item item))
                {
                    if (item.HasBuffer)
                    {
                        carry.Append(tls.Decrypt(item.Ptr, item.Len));   // records in, plaintext out
                        conn.ReturnBuffer(in item);
                    }
                }
            }

            if (Answer(conn, tls, carry, response))
            {
                await conn.FlushAsync();
            }

            if (snapshot.IsClosed || tls.Closed) return;
            conn.ResetRead();
        }
    }
    catch (Exception e)
    {
        Console.Error.WriteLine($"connection failed: {e.Message}");
    }
    finally
    {
        tls?.Dispose();   // close_notify, then the SSL object and its BIOs are freed
        conn.DecRef();
    }
};

// One response per complete request head in the carry, none for a partial one.
static bool Answer(TcpConnection conn, TlsSession tls, Carry carry, byte[] response)
{
    bool wrote = false;
    int end;
    while ((end = carry.Span.IndexOf("\r\n\r\n"u8)) >= 0)
    {
        carry.Consume(end + 4);
        tls.Write(conn, response);   // encrypted into the write slab - or plaintext, under KernelTx
        wrote = true;
    }
    return wrote;
}

Decrypt takes the ring buffer's pointer, so it sits in an unsafe block - allowed inside an async handler as long as no await is in it.

The request that comes with the handshake#

Under TLS 1.3 a client can send its first request right behind its Finished message, without waiting for an answer, and clients routinely do; for HTTP/2 that first data is the connection preface. The two arrive in one receive, so the handshake's last read hands BoringSSL the request along with the Finished, and the bytes are off the socket by the time the handshake ends. AcceptAsync decrypts them and keeps them in the session.

A raw loop takes them with DrainPlaintext() and answers before it parks in ReadAsync. A loop that reads first waits for a request the server already holds, while the client waits for its answer. The pipe's reader starts with those bytes, so a pipe handler has nothing to do.

TlsOptions#

Start reads the options as a whole. Combinations that contradict each other throw ArgumentException: KernelRx without KernelTx, KernelTx with a TLS 1.2 floor or with a suite list naming anything but TLS_AES_128_GCM_SHA256, both client CA sources, RequireClientCertificate without anchors, an undefined MinProtocolVersion, a negative timeout. A certificate, key or list BoringSSL cannot use throws IOException; an ALPN name or an SNI key no client could send throws ArgumentException. Thrown from OnStart, either ends that reactor's Run, so a bad configuration stops the server at startup rather than at its first handshake.

The key exchange is BoringSSL's, with no option for it: a client that offers X25519MLKEM768, the post-quantum hybrid, gets it, and others get X25519, P-256 or P-384.

In the default backend the server issues session tickets. Each reactor has its own ticket keys, so a returning client resumes when it reaches the reactor that issued its ticket, and does a full handshake otherwise.

More from the session#

A session also reports what its handshake settled on - NegotiatedAlpn, NegotiatedProtocolVersion as its wire value (0x0303 is TLS 1.2, 0x0304 TLS 1.3) and NegotiatedCipherSuiteId as the suite's IANA number - which directions the kernel took (KernelTx, KernelRx), and the client's certificate when one verified (PeerCommonName and the rest). DecryptInto is Decrypt writing straight into a PipeWriter, one copy fewer; the pipe's reader is built on it.

The session belongs to the reactor's thread. BoringSSL's session object is not thread-safe and the reactor decrypts with it, so with BoringSSL encrypting, Write called from another thread throws InvalidOperationException. Write from elsewhere through the pipe: its writer hands a FlushAsync or Complete from another thread to the reactor, which encrypts and sends there. Complete never throws out of a teardown, even with a flush still in flight: what it encrypted then goes out ahead of the close_notify if the send has finished by the time the session is disposed, and not at all if it has not.

Which TLS#

• This service for TLS on the TCP connections a reactor accepts - the default backend unless you have a reason for kernel TLS.
• QUIC's own TLS for HTTP/3: TLS 1.3 lives inside the transport there, configured on the engine.
• The HTTP client's TlsClientContext for connections a handler opens to an https:// origin: TLS in the HTTP client.
• SslStream over TcpConnectionStream when you want .NET's own TLS stack on an ioxide connection; it costs more, as measured here.

Where to go next#