TLS and kTLS

Kernel TLS

After BoringSSL finishes the handshake, the Linux kernel can take over a connection's records: KernelTx hands it the encryption of what the server sends, KernelRx the decryption of what it receives. Both are off by default: on the repository's loopback measurements the kernel path costs more CPU per request than BoringSSL unless it saves a copy, and it brings constraints BoringSSL does not.

Three configurations#

Kernel transmit used to be the default. It became opt-in because it measured level with the user-space library or behind it, and its constraints - the module, TLS 1.3 only, one suite, no resumption - applied to everyone whether they used what it offers or not. BoringSSL has widened the gap since: see what it costs.

Turning it on#

sudo modprobe tls

The module does not load on demand for an unprivileged process and does not survive a reboot, so load it at boot - on systemd, a file in /etc/modules-load.d/ that says tls. /sys/module/tls exists while it is loaded; that is what the test suite and the benchmarks check. Then:

var tlsOptions = new TlsOptions
{
    CertificatePath = "/etc/ioxide/server.crt",
    KeyPath         = "/etc/ioxide/server.key",
    KernelTx        = true,    // the kernel makes the outbound records
    KernelRx        = false,   // true: the kernel decrypts too - experimental
};

KernelTx pins every context of the service to TLS 1.3, the one suite TLS_AES_128_GCM_SHA256 - the kernel is programmed with AES-128-GCM keys - and no session tickets. Start refuses options that contradict that, rather than letting one silently win: a MinProtocolVersion of Tls12, a CipherSuites list naming anything else (one naming exactly that suite is accepted), and KernelRx without KernelTx, since the receive keys ride on what the transmit handoff installs.

The handoff#

It happens inside AcceptAsync, after the handshake and before it returns:

1. During the handshake, BoringSSL's keylog callback hands the session the TLS 1.3 traffic secrets: the server's for transmit, the client's for receive. Only kTLS contexts install it - the secret passes through a managed string nothing can scrub.
2. Once every handshake flight has been flushed, the transmit key and IV are derived from the server's secret with HKDF-Expand-Label (RFC 8446), the socket gets the TLS upper-layer protocol - setsockopt(SOL_TCP, TCP_ULP, "tls") - and the keys go in with setsockopt(SOL_TLS, TLS_TX, ...) at record sequence 0. Zero holds because no session ticket went out: a ticket is a record sent under the same key after the handshake, and would have advanced the count.
3. The connection's sends change. Sends carry MSG_WAITALL unless TcpOptions.SendWaitAll is off, and kTLS rejects it, so the handoff clears SendOpFlags; the reactor's partial-send loop keeps a flush whole without it. Clearing it here also pins the connection to plain sends, so ZeroCopySend does not apply to a kTLS connection.
4. Every copy of the key material - the secret, the derived key and IV, the structure passed to the kernel - is zeroed once programmed. A connection that does not hand off scrubs its secrets at the same point.

Client certificates are unaffected: they are exchanged and verified during the handshake, which BoringSSL always runs. Playground/Tls/MtlsKtlsPipes is the mutual TLS sample with KernelTx = true and nothing else changed.

What changes for a handler#

With the kernel transmitting, the write slab holds plaintext, and the kernel makes the records as the ordinary io_uring send hands it the bytes. TlsSession.Write copies plaintext straight into the slab under kTLS and encrypts with BoringSSL otherwise, and the pipe picks its writer from the session, so a handler that writes through either needs no change. The difference shows only to code that writes to the connection directly:

// Correct ONLY because KernelTx = true: the slab holds plaintext and the kernel encrypts it on
// send. With BoringSSL encrypting, the same two lines put cleartext on the wire.
conn.Write(response);
await conn.FlushAsync();

Files from the write slab#

TcpConnection.ReadFileAsync reads a file with IORING_OP_READ straight into the write slab - no buffer, no copy. Over TLS that is legal only when the slab may hold plaintext, which means under kernel transmit; with BoringSSL encrypting, the file has to pass through TlsSession.Write from a buffer of your own, and that buffer is the copy. (Its own documentation says cleartext only, for that reason.) Playground/Clients/File is built around the pairing:

// Under KernelTx: the header, then the file read straight into the write slab behind it.
tls.Write(conn, header.Span);                                      // plaintext into the slab
int n = await conn.ReadFileAsync(fileFd, length, fileOffset: 0);   // IORING_OP_READ into the slab
conn.AdvanceWrite(n);
await conn.FlushAsync();                                           // the kernel encrypts both

bench/file-matrix.sh measures it on one saturated reactor over loopback. At 4 KiB, kTLS with the slab served 224,537 requests a second at 4.51 us of CPU each, against 205,801 at 4.92 us for BoringSSL through a buffer: 1.09 times the throughput. At 64 KiB it reverses, 38,310 against 52,925. The kernel's AES costs more per byte than BoringSSL's in user space - through the same buffer path, 28.32 us a request against 19.12 - and that cost grows with the body while the saved copy does not keep pace. Where the two cross, between 4 and 64 KiB, was not measured. More in Files.

Closing#

Disposing the session still sends close_notify, so the peer can tell the end of the stream from a truncation: under kTLS as an alert record the kernel encrypts, sent with sendmsg and a TLS_SET_RECORD_TYPE control message. As with BoringSSL, it is best effort and never waits - a peer that stopped reading must not park the reactor - and it is sent in answer to the peer's own close_notify too.

Kernel receive (experimental)#

With KernelRx as well, the kernel decrypts inbound records, an ordinary receive returns plaintext into ring memory, and Decrypt hands its input back untouched. The pipe reads ring memory directly, as on a cleartext connection. What makes it a research toggle rather than an option to deploy:

• The handoff must land on a record boundary. Records the handshake already took off the socket are invisible to the kernel, and the record sequence number is part of each record's nonce, so the kernel must start counting past them. Once the handshake is done, AcceptAsync keeps feeding BoringSSL what the reactor's current batch of completions brings, and at the end of the batch counts the complete records BoringSSL holds and programs TLS_RX at that number. A record cut at the end is completed first: with one non-blocking recv of exactly its rest when that is already in the socket, or by waiting for it and counting again. A connection stays on BoringSSL's receive path, and its session says so, past 1 MiB held or when the reactor's completion queue overflowed that turn. One that closes inside a record is dropped; one that stalls there runs into the handshake timeout.
• The handshake's plaintext still has to be served. It lives in the session, not in ring memory, so the pipe puts a TlsProloguePipeReader in front of the ring reader: it serves those bytes first, then passes every read straight through. It holds at most 1 MiB - a peer that sends half a request head and dribbles forever faults the connection instead of growing it.
• A record that is not data ends the connection. Under kTLS receive, a KeyUpdate or an alert can only be read with recvmsg and a TLS_GET_RECORD_TYPE control message. The reactor receives with IORING_OP_RECV, which carries none, so the kernel refuses the read. A client that sends a TLS 1.3 KeyUpdate loses its connection. The peer's close_notify is an alert too and ends the read side the same way, but the kernel keeps the record: once the read side has ended, TlsSession.Closed reads it, so a clean end can still be told from a truncation.

Nor is it faster. Measured when kernel receive was added, against the system OpenSSL then in use, uploads from 4 KiB to 256 KiB ran at the same throughput whichever side decrypted, within 1.5%, and requests with no body came out 9% behind in one run and 6% ahead in another. AES-GCM is hardware-accelerated on both sides of the system call, so moving it into the kernel moves the work rather than removing it.

What the session reports#

TlsSession tls = await r.GetService<TlsService>().AcceptAsync(conn);

// What this connection got, which is not always what the options asked for.
Console.WriteLine($"tx={(tls.KernelTx ? "kernel" : "boringssl")} rx={(tls.KernelRx ? "kernel" : "boringssl")}");

KernelTx is true once the transmit keys are installed, and KernelRx once the receive keys are - which, with the option on, is most connections but not all. TlsConnectionDualPipe reads these, not the options, to choose its halves, so a connection whose receive handoff was skipped keeps the decrypting reader whatever was configured. A handler that writes through the session or the pipe never needs to branch on them; they are for logs and metrics.

What it costs#

Every figure here is loopback, where the two things kTLS exists for - a NIC that encrypts, and data that never passes through user space (ioxide has no sendfile path) - are out of reach. They measure the kernel path's cost with none of its upside.

TLS against plaintext: HTTP/1.1, 4 reactors, wrk -t4 -c64, best of two passes of bench/tls-matrix.sh - requests a second, the ratio to plaintext, and the server's CPU per request:

The cost of TLS dwarfs the choice of backend: from 64 KiB on, both are under half of plaintext. Between the two, BoringSSL is ahead at every size: kernel transmit costs about 5% more CPU per request at 64 B and 256 KiB, and 12-29% more from 8 to 64 KiB. Over HTTP/2, which hands the socket 16 KiB DATA frames, the two stay within 5% of each other at 64 B and 8 KiB. The cleared MSG_WAITALL is not the reason: measured alone, on the plaintext path at 64 KiB, it moved throughput by under 1.5%.

Through the pipes, at 2 reactors, 64 connections and 8 KiB responses, bench/ab.sh measured Tls/OpenSslPipes at 4.24 us of CPU per request and Tls/KtlsPipes, both directions in the kernel, at 4.92 us: BoringSSL leads there too, although its halves add a copy each way - the writer stages plaintext before encrypting it, the reader decrypts into a buffer of its own - and the kernel halves do not. Numbers from one machine, to compare with each other, not as a promise.

When to turn it on#

• Kernel transmit when the deployment has something that needs the kernel to make the records: a NIC that offloads TLS, or file serving from the write slab at small sizes, as above. It costs the module, TLS 1.2 clients and session resumption.
• Kernel receive to experiment with, not to deploy, while a client's KeyUpdate ends its connection.
• Neither otherwise. The default has none of the constraints, and nothing measured here makes up for them.