Site icon GetPageSpeed

Encrypted Client Hello in NGINX: Now Shipping in Our RPMs

Encrypted Client Hello in NGINX: Now Shipping in Our RPMs

TLS 1.3 encrypts almost everything. Almost.

Every handshake still announces, in plaintext, the hostname you are asking for. That is Server Name Indication, and it is the reason your ISP, your employer’s middlebox, and whoever runs the coffee shop Wi-Fi can build a complete list of the sites you visit without decrypting a single byte of traffic. Encrypted DNS closed the DNS half of that leak years ago. SNI stayed open.

Encrypted Client Hello closes it. RFC 9849 went to Proposed Standard, and as of today it is in our packages: ssl_ech_file works in NGINX-MOD and EDGE, along with $ssl_ech_status and $ssl_ech_outer_server_name.

As far as we can tell we are the first RPM vendor shipping a server-side ECH-capable NGINX. If someone beat us to it, tell us and we will correct this.

How ECH actually works

The trick is two ClientHellos instead of one.

The outer ClientHello names a shared, public “cover” hostname. That is the only name an observer sees. Inside it, encrypted to a public key you publish in DNS, sits the real ClientHello with the real hostname. The server decrypts the inner one and routes accordingly.

The public key travels in the ech= parameter of an HTTPS resource record (RFC 9460):

secret.example.com.  300  IN  HTTPS  1 . alpn="h3,h2" ech="AED+DQA8bAAgACDltwtL..."

Which immediately tells you the two things everyone gets wrong, so let us get them out of the way early.

The two things everyone gets wrong

Your DNS zone has to be DNS-only. If that HTTPS record sits behind a proxying CDN, the CDN is the TLS endpoint. It publishes its own HTTPS record with its own ECH keys. Clients get the CDN’s ECH, and your ssl_ech_file is never touched. That is not a bad outcome, since a large CDN has an enormous anonymity set, but it is the CDN’s privacy feature, not yours. To serve your own ECH the record must be DNS-only.

Clients need encrypted DNS. An HTTPS record fetched over plaintext port 53 leaks the hostname to exactly the observer ECH is meant to defeat, so browsers will not use ECH without DoH or DoT. Nothing you configure on the server changes that.

Be honest about what it buys you

ECH hides which name you asked for among the names sharing a server. It does not hide that you connected, and it does not hide the IP address.

If you run multi-tenant hosting, a CDN, or any shared front end, that is a real and substantial gain. An observer learns that someone reached a box serving thousands of sites, and nothing else.

If you run one site on a dedicated IP, it buys you very little. The address alone identifies the site, and a reverse lookup or a certificate transparency search finishes the job in seconds. ECH still strips the SNI string, which has some value against crude keyword-matching censorship, but anyone selling you ECH as a privacy silver bullet for a single-tenant server is selling you something.

The privacy comes from the size of the crowd behind the public name. A cover name used by one site hides nothing.

Why OpenSSL 3.5 and not OpenSSL 4.0

ECH landed upstream in OpenSSL 4.0. We are not shipping OpenSSL 4.0.

Our TLS stack is openssl35, an ABI-isolated build of OpenSSL 3.5 LTS with a CVE stream tracked to April 2030. That LTS commitment is the whole point of it, and swapping the fleet onto a brand new major release to pick up one feature would throw it away. So we did it the other way round: backported the ECH implementation from the DEfO project onto 3.5 LTS, with the same API surface OpenSSL 4.0 exposes.

NGINX itself needed no patching at all. ssl_ech_file is upstream NGINX, compiled in automatically when the TLS library supports ECH. That is the nice thing about backporting to the library rather than patching the server: the feature arrives through the front door.

The same rebuild also carries EDGE over to openssl35, so our Ultra-tier server does not trail its own base on TLS.

Rotate your keys, and let something else do it

ECH keys are meant to be short-lived. Doing that by hand is miserable, because two facts fight each other:

NGINX reads ECH keys when it parses its configuration, so a new key does nothing until you reload. And a client that cached your HTTPS record an hour ago encrypted to the old key, so if you drop that key the moment you generate a new one, you break them.

NGINX handles the second half properly. ssl_ech_file can be repeated, and the order is load-bearing: the first file is what gets advertised in retry-configs, and every later one is kept for decryption only. That is exactly the shape a rotation scheme needs. So we packaged one:

dnf install nginx-mod-ech

vi /etc/sysconfig/nginx-ech-rotate      # ECH_PUBLIC_NAME="ech.example.com"
nginx-ech-keygen --init
nginx-ech-keygen --print-dns            # publish this in your HTTPS record
systemctl enable --now nginx-ech-rotate.timer

That is the whole setup. The timer rotates daily, keeps three generations loaded, shreds anything older, and regenerates the include NGINX reads. Keys live in /etc/nginx/ech as 0640 root:nginx inside a 0750 directory, generated under umask 077 and renamed into place, so a private key is never briefly world-readable and NGINX never sees a half-written file.

Nothing runs until you set a public name, and the timer is not enabled on install. Installing the package changes nothing about a working server. EDGE gets the same thing as edge-ech, with EDGE’s own paths and unit names.

One piece you still have to wire up

The timer rotates the key. It does not touch DNS, and it cannot – it has no idea who hosts your zone. Leave it there and your HTTPS record goes stale the first time the timer fires.

Nothing breaks, which is exactly why it is easy to miss: stale clients get failed+retry-configs, connect anyway, and self-heal. But no first handshake ever gets ECH, and that is most of the benefit gone. Hook your DNS provider onto the rotation unit with a drop-in, so publishing happens in the same transaction as rotating:

# /etc/systemd/system/nginx-ech-rotate.service.d/50-publish-dns.conf
[Service]
ExecStart=/usr/local/sbin/publish-ech-dns

Type=oneshot runs ExecStart lines in order, so it only fires after a rotation that actually succeeded. The script reads nginx-ech-keygen --print-dns and pushes the value at your provider’s API. Retention covers the propagation gap: the previous key is still loaded for decryption while the record catches up.

And get the certificates right

The cover name’s certificate is presented in the outer handshake, in the clear, to the observer ECH exists to defeat.

So do not put the names you are hiding in its SAN list. One certificate that covers both the cover name and the secret name hands back precisely what ECH just concealed, and it will pass every test you run because the handshake works fine. Issue them separately.

What happens if you get the retention window wrong

We measured it rather than guessed, and the answer is reassuring.

Client’s cached ECHConfigList Result
The advertised key ECH succeeds, routed to the inner name
A retained decrypt-only key ECH succeeds, routed to the inner name
Aged out of the retention window ECH: failed+retry-configs, connection still completes

In the last case NGINX cannot decrypt, falls back to the outer ClientHello, serves the public name’s server block, and hands back a retry-config with the current key. The client picks it up and recovers by itself. You pay a wasted round trip, not an outage.

Which is also the strongest argument for getting your cover name’s certificate right. Every stale client lands on it. Get that certificate wrong and a routine rotation turns a recoverable retry into a certificate error your users actually see.

Try it against a real server

We put one up: ech-test.getpagespeed.com.

It runs this exact stack on a public 443, with keys rotated daily by the same nginx-mod-ech package described above, and it tells you what your browser actually negotiated. There is a JSON endpoint at /status.json if you want to script against it. The cover name it hides behind, ech.getpagespeed.com, is public too – that is the only hostname an observer of that connection can see.

Client support, stated honestly

Verified end to end against the demo host, over real DoH with a real ech= HTTPS record, reaching $ssl_ech_status success: Chrome / Chromium, Firefox 147, and openssl s_client -ech_config_list.

Firefox is worth a paragraph, because our first lab attempt saw it send GREASE only and we nearly wrote that up as an unknown. It was not an interop problem. It was DNS every time, and the causes are worth knowing before you conclude your server is broken:

Point a browser at the demo host and you can tell a client-side DNS problem from a server-side one in about five seconds, which is the main reason it exists.

Clients with no ECH support are unaffected. They send an ordinary ClientHello and get served normally.

Limitations worth knowing

Split mode is not supported. NGINX implements shared mode, where the server that decrypts ECH also serves the inner name. The split arrangement, with a front end decrypting and forwarding to a separate backend, is not in upstream NGINX.

ECH requires TLS 1.3. There is no ECH for TLS 1.2.

And new keys need a reload. The rotation timer handles that for you, but it is worth knowing it is a reload and not a signal-free hot swap.

Getting it

ECH is in the testing channel now, in NGINX-MOD and EDGE, with rotation tooling in nginx-mod-ech and edge-ech. Full configuration reference, DNS examples and rotation details are in the ECH documentation, and ech-test.getpagespeed.com will tell you whether your own browser can use any of it.

If you run multi-tenant infrastructure, this is the most meaningful privacy improvement available to your users right now, and it costs you one directive and a DNS record.

D

Danila Vershinin

Founder & Lead Engineer

NGINX configuration and optimizationLinux system administrationWeb performance engineering

10+ years NGINX experience • Maintainer of GetPageSpeed RPM repository • Contributor to open-source NGINX modules

Exit mobile version