yum upgrades for production use, this is the repository for you.
Active subscription is required.
Technical Briefing: ngx_cache_purge 2.7.0 Speeds Up Purges of Vary’d Cache Entries
Today’s source is a GetPageSpeed article about version 2.7.0 of the ngx_cache_purge NGINX module, which is already in use. The release adds a shared-memory variant index. It makes repeat exact purges of Vary‘d cache keys up to roughly 1000× faster than the full-zone scan introduced in 2.6.2.
The Problem: One Cache Key, Many Files
When an upstream sends Vary: Accept-Encoding, or when gzip_vary is enabled, NGINX stores one cache file per distinct request header value under the same cache key.
Purging these files is hard for three reasons:
– A variant’s filename is a hash of the cache key plus the client’s header values.
– A purge request can’t know every Accept-Encoding string that clients have sent.
– NGINX keeps no on-disk index of variants.
So the only correct way to find every variant is to walk the entire cache zone and check each file’s embedded KEY: line.
The module has handled this in three stages:
| Version | Behavior | Consequence |
|---|---|---|
| Before 2.6.2 | Purged at most one file: the variant matching the purge request’s own Accept-Encoding |
Typically returned 412 Precondition Failed, and stale variants stayed servable |
| 2.6.2 | Exact purge deletes every stored variant by scanning the full zone | Correct, but O(every file in the zone), the same cost class as a wildcard purge |
| 2.7.0 | Shared-memory variant index | Repeat purges cost O(variants) instead of O(zone) |
Example Setup
proxy_cache_path /var/cache/nginx/site keys_zone=site:10m;
server {
listen 80;
location / {
proxy_pass http://backend;
proxy_cache site;
proxy_cache_key $uri$is_args$args;
proxy_cache_valid 200 10m;
# same-location purging: PURGE method on the same URL
proxy_cache_purge PURGE from 127.0.0.1;
}
}
These two requests create two variants of /style.css:
curl -H 'Accept-Encoding: gzip' http://example.com/style.css
curl http://example.com/style.css
This single request purges both:
curl -X PURGE -i http://127.0.0.1/style.css
The response is HTTP/1.1 200 OK with Cache-Purge-Result: exact; count=2.
Cost of the 2.6.2 Full Scan
The test setup was a single worker on Linux ext4, running in a container on an Apple M2. The benchmark script bench/variant_scan_bench.sh ships with the module source.
| Files in zone | Cold purge | Warm purge | Non-Vary control |
|---|---|---|---|
| 10,000 | 0.5 s | 0.03 s | 0.2 ms |
| 100,000 | 4.5 s | 0.2 s | 0.2 ms |
| 1,000,000 | 227 s | ~1 s | 0.1 ms |
- Warm values are the median of three trials.
- The article describes these as order-of-magnitude laptop/VM numbers, not lab-grade measurements.
- The real danger is that the NGINX worker is blocked for the entire scan. A cold purge on a million-file zone parks a worker for minutes.
How 2.7.0 Fixes It: A Shared-Memory Variant Index
What the index does. It maps each cache key to the variant hashes stored or served for that key. When an exact purge finds the key in the index, it unlinks exactly those files, at O(variants) cost.
When entries are created. An index entry is created only right after an exact purge has removed every file for the key. That is the one moment when “no variants exist” is provably true. From then on, a header filter records each variant NGINX stores or serves.
When entries are dropped. The module falls back to a full scan if any of these happens:
– an NGINX restart
– eviction under index memory pressure
– more than 64 variants on one key
The next scan re-seeds the index.
Additional guarantees:
– Stale entries that point to files the cache manager has already evicted are skipped harmlessly.
– The worst case is exactly the 2.6.2 behavior. Correctness is never traded for speed.
Caveat: the first exact purge of a Vary’d key in an instance’s lifetime still scans the zone. Only purges after that use the index.
Which Purges Cost What
| Purge | Cached response | Behavior | Latency |
|---|---|---|---|
PURGE /style.css |
no Vary | direct key lookup, one unlink | sub-millisecond, any zone size |
PURGE /style.css |
Vary, first purge | full scan, seeds index | seconds to minutes, grows with zone |
PURGE /style.css |
Vary, repeat purge | index lookup | milliseconds, any zone size |
PURGE /css/*, purge_all, cache tags |
any | zone walk by design | grows with zone |
To see which path ran, check the error log at info level:
– A scan logs exact purge removed N vary variant(s).
– The indexed path appends via variant index.
To avoid variants entirely, set proxy_ignore_headers Vary in the cached location. The cost is that all clients get one representation.
Results With the Index Enabled (Default)
| Files in zone | First purge (scan) | Repeat purge (indexed) |
|---|---|---|
| 100,000 | 17.7 s (cold) | 0.2–0.4 ms |
| 1,000,000 | – | 1–5 ms |
- Repeat purges land in the 0.2–2 ms range regardless of zone size.
- At 1M files, that is roughly 1000× faster than the warm scan and five orders of magnitude faster than the cold scan.
Cache-Purge-Result: exact; count=Nreports identical counts on either path.
Configuration
The index is on by default with an 8 MB shared memory zone, which tracks tens of thousands of purged keys. A single http-level directive controls it:
# default; size it up for very large purge-heavy deployments
cache_purge_variant_index 8m;
# or always use the full scan
cache_purge_variant_index off;
Non-Vary purges never scan and never touch the index. If your upstreams send no Vary header, nothing changes for you.
Installation
RHEL-likes:
dnf -y install https://extras.getpagespeed.com/release-latest.rpm
dnf -y install nginx-module-cache-purge
Debian/Ubuntu, after setting up the repository:
apt install nginx-module-cache-purge
Then load the module and reload NGINX:
load_module modules/ngx_http_cache_purge_module.so;
The full changelog, directive reference, and correctness model are on the module’s documentation page.
Practical Payoff
On sites that serve Vary‘d content from large cache zones, repeat exact purges drop from seconds or minutes of blocked-worker time to milliseconds. Correctness is unchanged: every variant is still removed. Only the first purge per key after startup, eviction, or index overflow pays the full-scan cost.
Read the full article: How We Made NGINX Vary Cache Purges 1000× Faster
