TLS and kTLS

Certificates and mutual TLS

Where a server's certificate comes from, how one port serves several names, how to renew certificates on a running server without dropping a connection, and how to ask clients for theirs.

A certificate and its key#

The certificate chain - the leaf first, then its intermediates, as in a chain file - and its private key, each from a file or as PEM text:

Exactly one source for each; Start refuses none and both. A TlsCertificate holds the same four properties, and is what the SNI table and a rotation take. PEM text suits a host that keeps certificates as data - a secrets store, an X509Certificate2 - rather than as files:

// A certificate held as data - a store, a vault - handed over as PEM text. Leaf first, then its chain.
static TlsCertificate FromX509(X509Certificate2 leaf, X509Certificate2Collection intermediates)
{
    var chain = new StringBuilder(leaf.ExportCertificatePem()).Append('\n');
    foreach (X509Certificate2 intermediate in intermediates)
    {
        chain.Append(intermediate.ExportCertificatePem()).Append('\n');
    }

    using AsymmetricAlgorithm key = (AsymmetricAlgorithm?)leaf.GetRSAPrivateKey()
        ?? leaf.GetECDsaPrivateKey()
        ?? throw new ArgumentException("the certificate carries no RSA or EC private key");

    return new TlsCertificate { CertificatePem = chain.ToString(), KeyPem = key.ExportPkcs8PrivateKeyPem() };
}

RSA and EC keys load - PKCS#8, or the older PKCS#1 form (BEGIN RSA PRIVATE KEY) that openssl genrsa writes - and so do CRLF line endings, a UTF-8 byte-order mark in a file, and the readable dump openssl x509 -text prints above the PEM. Start refuses, with an IOException:

• An encrypted key. There is no passphrase option, so it is refused - never a prompt on the terminal that would hang the start.
• A key that does not belong to the certificate, of the same algorithm or another, such as an EC key beside an RSA certificate: the certificate and private key do not go together.
• A chain torn mid-block, as a renewal caught half written leaves it: refused, rather than served without the intermediates clients need.

A path is read when the service starts, and again by every rotation that names it; PEM text is a value and does not change.

Several names on one port#

A client names the host it wants in its ClientHello (Server Name Indication), and CertificatesByHost maps names to certificates:

var tlsOptions = new TlsOptions
{
    // The default: a client that sends no name, or one not in the table.
    CertificatePath = "/etc/ioxide/default.crt",
    KeyPath         = "/etc/ioxide/default.key",

    CertificatesByHost = new Dictionary<string, TlsCertificate>
    {
        ["alpha.example"] = new() { CertificatePath = "/etc/ioxide/alpha.crt", KeyPath = "/etc/ioxide/alpha.key" },
        ["beta.example"]  = FromX509(betaCertificate, betaIntermediates),   // PEM text
    },
};

• The certificate outside the table is the default. It answers a client that sends no name - anything connecting by address - and one asking for a name the table does not hold. Refusing those would make the server unreachable by address and turn a certificate problem into a dead connection; with the default, the client sees a certificate and decides. Only a name that matched is acknowledged in the handshake.
• Names match whole and case-insensitively. A wildcard certificate covers its names through the certificate, not through the table, so list the exact names. Keys are host names in ASCII - an international name in its xn-- form - with no port and never an IP address, which SNI cannot carry. Start refuses, naming it, a key no client could ask for - a blank one, one with a *, an IP address, a non-ASCII name, one over 255 bytes - and two keys that differ only in case.
• Each entry is one BoringSSL context, built at Start. Choosing one is a short scan of byte strings on the reactor thread, with no allocation, so ten names cost a handshake what one does.
• A name changes the certificate and nothing else. Client verification, the protocol floor and the kTLS pin are copied onto each connection from the default context, so a name is no way around mutual TLS or kernel TLS. The trust anchors and ALPN are set on every context, so both work for named hosts.
• Session tickets are shared by the names of one service: a ticket is not bound to the name it was issued under. Harmless while every name is verified alike.

TlsService.ServerNames lists the names a service answers for. The name in the handshake picks the certificate, not the site: HTTP/2 lets a client reuse one connection for every origin its certificate covers, so route and authorize on each request's :authority or Host. Playground/Http2/Sni shows the two side by side, and Playground/Tls/Sni is SNI under HTTP/1.1.

Renewing on a running server#

ReplaceCertificates(defaultCertificate, certificatesByHost) installs a new set on a service that is serving. The next handshake gets it; a connection already up keeps the certificate it authenticated, and none is dropped.

• It replaces the whole set. Pass the default and every name to keep. certificatesByHost defaults to null, and a rotation given only the default answers every name with it - a server that looks fine and serves the wrong certificate.
• It is per reactor. Each reactor has its own service and they all accept on one port through SO_REUSEPORT, so rotate every one: a reactor that is missed serves the old certificate to its share of clients. There is no instant at which they all flip together.
• It builds before it publishes. A path that is not there yet, or a PEM half written, throws, and the service keeps serving what it had. A failed rotation is a server that kept working: log it and try again later.
• It changes certificates only. ALPN, the protocol floor, the ciphers, kernel TLS and the client-verification settings stay as the service started. One thing follows the disk: anchors given as ClientCaPath are re-read at every rotation, so an edit to that file takes effect at the next renewal - a way to drop an issuer without a restart, and a way to widen who may connect without meaning to. ClientCaPem does not move.
• Session tickets do not survive it. Ticket keys belong to the contexts a rotation replaces, so a client holding an older ticket does a full handshake. That is deliberate: a resumed session would carry its old verification past new anchors.
• Call it from any thread. Rotations lock against each other; handshakes never lock, and read the current set in one reference load. The replaced contexts are kept for the life of the service, because a handshake may be between reading the set and using it; a few rotations a year cost kilobytes.

The shape Playground/Http2/Rotate uses: each reactor publishes its service as it starts, and SIGHUP - what an ACME hook sends once it has rewritten the files - rotates them all, re-reading the same paths:

// Each reactor publishes its service as it starts; a rotation has to reach all of them.
var services = new TlsService?[config.ReactorCount];

for (int i = 0; i < config.ReactorCount; i++)
{
    int id = i;
    var reactor = new Reactor(i, config);
    reactor.OnStart = r => Volatile.Write(ref services[id], TlsService.Start(r, tlsOptions));
    // ... TcpHandle, thread start ...
}

// What an ACME hook sends after it rewrites the PEM files in place.
using var sighup = PosixSignalRegistration.Create(PosixSignal.SIGHUP, context =>
{
    context.Cancel = true;   // a rotation, not a shutdown

    for (int i = 0; i < services.Length; i++)
    {
        try
        {
            // The whole set, every time: the default and every name to keep, re-read from disk.
            Volatile.Read(ref services[i])?.ReplaceCertificates(
                new TlsCertificate { CertificatePath = certPath, KeyPath = keyPath },
                byHost);
        }
        catch (Exception e)
        {
            // Nothing was published: this reactor still serves what it had.
            Console.Error.WriteLine($"reactor {i} kept its certificates: {e.Message}");
        }
    }
});

QUIC is different: one QuicEngine serves every reactor, so a single call rotates them all - see HTTP/3 and QUIC.

Mutual TLS#

Give the service trust anchors for client certificates and it asks every client for one during the handshake:

var tlsOptions = new TlsOptions
{
    CertificatePath          = "/etc/ioxide/server.crt",
    KeyPath                  = "/etc/ioxide/server.key",
    ClientCaPath             = "/etc/ioxide/client-ca.pem",   // turns client certificates on
    RequireClientCertificate = false,                         // a client without one still connects
};

"Optional" is about presenting nothing: a certificate that is offered is always verified. Requiring one with no anchors is refused at Start, and so is setting both ClientCaPath and ClientCaPem. The CertificateRequest carries the anchors' names, so a client holding several certificates can pick one this server accepts.

Verification covers the chain to an anchor, its signatures and its dates - not revocation. No CRL is fetched and no OCSP request is made, so a revoked certificate is accepted until it expires. Bound that with short-lived certificates, or check the identity against a list of your own.

Anchors as PEM text are parsed by ioxide rather than by BoringSSL's file loader, and read the way a file is: CERTIFICATE and TRUSTED CERTIFICATE blocks are both trusted, a byte-order mark is ignored, an issuer the bundle repeats is named once in the CertificateRequest, and a malformed block is refused rather than quietly ending the bundle. One difference remains: an anchor from a TRUSTED CERTIFICATE block is named in the CertificateRequest too, where the file loader leaves it out.

A client certificate larger than about 30 KB cannot be served: BoringSSL's TLS 1.3 session tickets carry the client's certificate, and past that size two of them no longer fit beside the first response record in the 64 KB buffer it is written from.

Who connected#

The session describes the client's certificate only when it verified; an invalid one never reaches a handler.

reactor.TcpHandle = async (r, conn) =>
{
    try
    {
        TlsSession tls = await r.GetService<TlsService>().AcceptAsync(conn);

        // Set only for a certificate that verified; null for a client that sent none.
        // Compare the common name whole and ordinal - never a substring of PeerSubject.
        bool admin = string.Equals(tls.PeerCommonName, "ops-admin", StringComparison.Ordinal);

        await using var pipe = new TlsConnectionDualPipe(conn, tls);
        await ServeAsync(pipe, admin ? ok : forbidden);   // the loop from the overview's program
    }
    catch (Exception e)
    {
        Console.Error.WriteLine($"connection failed: {e.Message}");   // a refused client lands here
    }
    finally
    {
        conn.DecRef();
    }
};

The identity belongs to the connection: it is settled in the handshake, and nothing asks for a certificate later, so read it once rather than per request. A client returning with a session ticket on a mutual-TLS port is resumed as the same identity. Ports that need different client policies need different services: start one per port with TlsService.Start(r, options, register: false) and keep the instances yourself, since GetService returns only the last one registered. Kernel TLS changes nothing here - Playground/Tls/MtlsKtlsPipes is Playground/Tls/MtlsOpenSslPipes with KernelTx = true.

SslStream instead#

The BCL's SslStream runs on an ioxide connection through TcpConnectionStream, a Stream that reads from the ring and writes into the write slab. None of this page applies to it - no TlsService, no TlsOptions: it is .NET's own TLS, configured the .NET way.

reactor.TcpHandle = async (r, conn) =>
{
    SslStream? ssl = null;
    try
    {
        // TcpConnectionStream reads from the ring and writes into the slab; SslStream does the rest.
        ssl = new SslStream(new TcpConnectionStream(conn), leaveInnerStreamOpen: false);
        await ssl.AuthenticateAsServerAsync(new SslServerAuthenticationOptions
        {
            ServerCertificate    = certificate,
            ApplicationProtocols = [SslApplicationProtocol.Http11],
            EnabledSslProtocols  = SslProtocols.Tls12 | SslProtocols.Tls13,
        });

        var request = new byte[8192];
        while (await ssl.ReadAsync(request) > 0)
        {
            await ssl.WriteAsync(response);   // one response per read: a sketch, not HTTP framing
        }
    }
    catch (Exception e)
    {
        Console.Error.WriteLine($"connection failed: {e.Message}");
    }
    finally
    {
        ssl?.Dispose();
        conn.DecRef();
    }
};

It costs more. In the bench suite's run of 9 August - 2 reactors, 64 connections, 8 KiB responses, bench/results/20260809T181559Z.json - Playground/Tls/SslStream served 331,646 requests a second at 9.27 us of CPU each, against 417,981 at 4.84 us for Playground/Tls/OpenSsl, on the system OpenSSL it used then: 0.79 times the throughput for nearly twice the CPU. The process also used 154% of the two reactors' cores, so part of its work ran on other threads. BoringSSL has since taken about 15% off the ring-native service's CPU per request, so the gap is wider today. Take it when you want .NET's TLS stack specifically; the ring-native service already does TLS 1.2 and 1.3, resumption, SNI, renewal and client certificates.

TLS to an origin#

Everything here terminates TLS for connections coming in. A handler that calls an https:// origin uses the HTTP client's TlsClientContext instead, with its own certificate checks, CA file and client certificates: see TLS in the HTTP client.