NGINX Kernel TLS (kTLS): Setup, Verification and the tcp_notsent_lowat Trap
by Danila Vershinin, , revisited on
dnf, yum or apt upgrades. It maintains the largest RPM repository of NGINX module packages and Varnish VMODs, with APT packages for Debian and Ubuntu.
Free to install; commercial production use requires a subscription.
Modern Linux kernels can take over TLS record encryption from NGINX. With kernel TLS (kTLS) enabled, NGINX serves static files through SSL_sendfile(): file pages go from the page cache straight into encrypted packets, without a copy through user space. For TLS-heavy static workloads (downloads, package repositories, media) this saves CPU and raises throughput.
This guide covers which systems benefit, how to switch kTLS on per site, how to verify it, and two traps that can silently hurt you: a corrupting EL 9.7 kernel and the tcp_notsent_lowat sysctl that throttles HTTP/1.1 downloads to about 100 kB/s.
The canonical, maintained reference is the Kernel TLS (KTLS) documentation on the NGINX Extras site.
Requirements
Every GetPageSpeed NGINX build, the stock nginx package as well as NGINX-MOD, links against openssl35, which we compile with enable-ktls. kTLS capability is already in the binary. It stays inactive until you enable it per server block.
That applies to every distribution we build for, including CentOS 7 and other EL7 systems. What you additionally need is a kernel with the tls module:
- RHEL, Rocky Linux, AlmaLinux 8 and 9, Amazon Linux 2023: the distro kernel ships the module.
- CentOS 7 / EL7: the stock 3.10 kernel has no kTLS. Install a modern kernel: Everpatch Kernel brings the current 6.18 LTS line to EL7.
The cipher must be AES-GCM or ChaCha20-Poly1305 over TLS 1.2 or 1.3, which is what any current configuration negotiates anyway. The module auto-loads on first use.
Install NGINX from the GetPageSpeed repository if you have not already:
sudo dnf -y install https://extras.getpagespeed.com/release-latest.rpm
sudo dnf -y install nginx
Should you enable it?
kTLS is opt-in for a reason. Our measurements (NGINX 1.30, OpenSSL 3.5, TLS 1.3, AES-GCM, static files):
- EL9 stock kernel 5.14, x86_64: roughly CPU-neutral. There is little to gain.
- Kernel 6.8 and newer: about a third less NGINX worker CPU and substantially higher throughput on the same hardware.
Proxied responses do not use SSL_sendfile() and gain much less. kTLS applies to TCP only, so HTTP/3 (QUIC) is unaffected.
We run this in production: extras.getpagespeed.com, the repository these packages come from, has served its HTTPS downloads over kTLS since September 2026 on kernel-lt 6.18.
Do not enable on kernel 5.14.0-611.24.1.el9_7
This specific Rocky/RHEL 9.7 kernel corrupts data sent through kTLS sendfile(): clients receive files whose checksums do not match. Since the stock EL9 kernel gains next to nothing from kTLS anyway, the safe rule is to enable kTLS only on a modern kernel such as kernel-lt or kernel-ml 6.x, and to run the integrity check below after enabling it.
Enabling kTLS in NGINX
Enable it in the server block where you want it, not globally for the whole http context:
server {
listen 443 ssl;
server_name downloads.example.com;
...
sendfile on;
ssl_conf_command Options KTLS;
}
sendfile on; is what unlocks the zero-copy SSL_sendfile() path. It is often inherited from the http block already.
Note that ssl_conf_command directives are inherited from the http level only when the server block defines none of its own. If you already use ssl_conf_command at the http level, repeat those commands in any server block where you add Options KTLS.
Reload NGINX:
sudo nginx -t && sudo systemctl reload nginx
Verifying kTLS
The kernel counts kTLS sessions and errors:
cat /proc/net/tls_stat
TlsTxSw climbing while you make HTTPS requests confirms kTLS transmit is active. With OpenSSL 3.5 the receive side is offloaded too, so TlsRxSw climbs alongside it. TlsTxSwError must stay at 0. A slow trickle of TlsDecryptError from scanners sending garbage is normal on an Internet-facing server.
Then confirm integrity end to end. This is the check that catches a bad kernel:
curl -so /tmp/f https://downloads.example.com/some-large-file
sha256sum /tmp/f # compare against the file on the server
If the checksums mismatch, disable kTLS and update the kernel before trying again.
Gotcha: tcp_notsent_lowat throttles HTTP/1.1 downloads
Symptom: after enabling kTLS, every HTTP/1.1 download crawls at around 100 kB/s, even over loopback, while HTTP/2 transfers of the same file run at full speed. On our repository this hit every yum client on EL7, since they download over HTTP/1.1.
Cause: the net.ipv4.tcp_notsent_lowat=16384 sysctl, a widely copied tuning from guides about HTTP/2 prioritization in NGINX. A kTLS record carries 16384 bytes of payload plus 22 bytes of overhead, so a single queued record keeps the socket at or above the low-water mark. After each record, SSL_sendfile() gets EAGAIN and waits 40 to 200 ms for the socket to become writable again. It is not a kernel or OpenSSL bug, and tcp_nopush makes no difference.
Our loopback A/B test, 6 MB file over HTTP/1.1:
tcp_notsent_lowat |
Throughput |
|---|---|
| 16384 | 104 kB/s |
| 131072 | 34.7 MB/s |
| kernel default (4294967295) | 37.9 MB/s |
| kTLS off | 35 MB/s |
Check the current value:
sysctl net.ipv4.tcp_notsent_lowat
The kernel default is 4294967295. Anything at or below 16384 will starve kTLS. Congestion control is another kernel setting worth measuring before you copy a tuning guide; see TCP BBR vs Cubic on lossy links.
Fix: raise it to at least 131072, or remove the override entirely. Find where it is set, then persist the new value:
grep -r tcp_notsent_lowat /etc/sysctl.conf /etc/sysctl.d/ /usr/lib/sysctl.d/
echo 'net.ipv4.tcp_notsent_lowat = 131072' | sudo tee /etc/sysctl.d/90-ktls-notsent-lowat.conf
sudo sysctl --system
If another file in /etc/sysctl.d/ sets the old value, edit or remove that line, because the file that sorts last wins. The kernel reads the value on every write check, so the change applies at once, with no NGINX restart.
Verify with an HTTP/1.1 download of a large file:
curl --http1.1 -so /dev/null -w '%{speed_download}\n' https://downloads.example.com/some-large-file
The result is in bytes per second. It should now be in the same range as an HTTP/2 download of the same file (--http2), not around 100000.
Summary
kTLS is a real win for static file serving on modern kernels, and every GetPageSpeed NGINX build already supports it. Enable it per server, on a 6.x kernel, then check /proc/net/tls_stat, file checksums and HTTP/1.1 throughput. For the latest notes and known bad kernels, see the NGINX kTLS documentation.