Accept-Encoding Normalization in NGINX for Better Caching

Accept-Encoding normalization is a critical optimization for any NGINX reverse proxy that uses caching. When NGINX acts as a reverse proxy with caching enabled, the Vary: Accept-Encoding response header tells the cache to store separate variants for each unique Accept-Encoding request header value. However, browsers and HTTP clients send this header in wildly different formats. One client sends gzip, br, another sends br, gzip, and a third sends br;q=1.0, gzip;q=0.8. All three accept the exact same compression algorithms. Without Accept-Encoding normalization, each creates a separate cache entry for the same content, wasting storage and reducing hit rates.

The ngx_http_compression_normalize_module solves this problem. It standardizes the Accept-Encoding header before it reaches the cache key computation. This article explains the cache variant explosion problem in detail. It then demonstrates how to install and configure the module, and shows verified test results proving the improvement.

The Cache Variant Explosion Problem

Every modern web server sends a Vary: Accept-Encoding response header to instruct caches that the response body depends on the client’s supported compression algorithms. This is correct behavior — a gzip-compressed response cannot be served to a client that only supports Brotli.

However, the HTTP specification (RFC 9110) allows clients considerable flexibility in how they format the Accept-Encoding header:

• Different ordering: gzip, br vs. br, gzip
• Quality values: br;q=1.0, gzip;q=0.8 vs. gzip, br
• Mixed case: GZIP, BR vs. gzip, br
• Extra whitespace: gzip , br vs. gzip,br

When you use $http_accept_encoding in your proxy cache key, each of these variations creates a separate cache entry. They all mean the same thing, yet the cache treats them differently. For a single URL, you could end up with dozens of cache variants instead of the three or four that are actually needed.

Why NGINX’s Built-in map Directive Falls Short

You might consider using NGINX’s native map directive for Accept-Encoding normalization. In theory, you could map known patterns to normalized values:

map $http_accept_encoding $normalized_ae {
    "gzip, br"  "gzip, br";
    "br, gzip"  "gzip, br";
    default     $http_accept_encoding;
}

While map handles case-insensitive matching, it has three critical limitations. First, with three compression algorithms (gzip, Brotli, zstd), there are already 6 permutations for just the full set, plus all subsets. Each additional algorithm multiplies the entries you must maintain. Second, map cannot parse quality values like br;q=1.0, gzip;q=0.5. A client sending this header will not match any map entry, falling through to the default. Third, map cannot handle whitespace variations — gzip,br and gzip, br are different strings and require separate entries.

How the Compression-Normalize Module Works

The ngx_http_compression_normalize_module takes a fundamentally different approach. Instead of trying to match every possible header variation, it:

1. Parses the Accept-Encoding header into individual encoding names
2. Respects quality values — encodings with q=0 are excluded
3. Normalizes case — GZIP and gzip are treated identically
4. Strips whitespace — extra spaces around encoding names are ignored
5. Matches against configured combinations in priority order
6. Replaces the header with the first matching combination’s canonical form

The module runs in the NGINX precontent phase. Therefore, the normalized header is available to all subsequent processing, including proxy_pass, proxy_cache_key, and logging.

Installation

RHEL, CentOS, AlmaLinux, Rocky Linux

sudo dnf install https://extras.getpagespeed.com/release-latest.rpm
sudo dnf install nginx-module-compression-normalize

Then load the module by adding this line at the top of /etc/nginx/nginx.conf, before any http block:

load_module modules/ngx_http_compression_normalize_module.so;

For more details, see the RPM module page.

Debian and Ubuntu

First, set up the GetPageSpeed APT repository, then install:

sudo apt-get update
sudo apt-get install nginx-module-compression-normalize

On Debian/Ubuntu, the package handles module loading automatically. No load_module directive is needed.

Configuration

The module provides a single directive and one variable.

Directive: compression_normalize_accept_encoding

Each argument defines a combination of compression algorithms that the server should recognize as a valid set. The module checks combinations in the order you specify. It uses the first match. Combinations with multiple algorithms must be quoted.

Basic Configuration

The following configuration covers the most common modern scenario. It handles three algorithms (gzip, Brotli, zstd) plus their subsets:

http {
    compression_normalize_accept_encoding
        "gzip, br, zstd"
        "gzip, br"
        "gzip, zstd"
        "br, zstd"
        zstd
        br
        gzip;

    # ...
}

With this configuration, a client sending Accept-Encoding: zstd, br, gzip (any order) has its header normalized to gzip, br, zstd. A client sending br;q=1.0, gzip;q=0.8 matches gzip, br because both encodings have non-zero quality values.

How Combination Matching Works

The module does not care about the order of encodings in the client’s header. It parses the header to extract which algorithms the client accepts. Encodings with q=0 are excluded. Then it checks each configured combination:

1. Split the combination by commas to get individual encoding names
2. Check whether all encodings in the combination appear in the client’s accepted set
3. The first combination where all encodings match wins

For example, with a client sending Accept-Encoding: br, gzip;q=0, zstd:

• gzip;q=0 means gzip is rejected — the accepted set is {br, zstd}
• "gzip, br, zstd" → needs gzip → skip
• "gzip, br" → needs gzip → skip
• "gzip, zstd" → needs gzip → skip
• "br, zstd" → needs br and zstd → match! → header becomes br, zstd

If no combination matches, the original Accept-Encoding header remains unchanged.

Disabling the Module

To disable normalization in a specific context, use:

location /no-normalize {
    compression_normalize_accept_encoding off;
    proxy_pass http://backend;
}

Variable: $compression_original_accept_encoding

This variable preserves the original Accept-Encoding header value before normalization. Use it for logging or debugging:

log_format cache_debug '$remote_addr - $request '
    'original_ae="$compression_original_accept_encoding" '
    'normalized_ae="$http_accept_encoding" '
    'cache_status=$upstream_cache_status';

Practical Use Case: Proxy Cache with Accept-Encoding Normalization

The primary use case is improving proxy cache hit rates. Here is a complete, production-tested configuration:

load_module modules/ngx_http_compression_normalize_module.so;

http {
    # Accept-Encoding normalization before cache key computation
    compression_normalize_accept_encoding
        "gzip, br, zstd"
        "gzip, br"
        "gzip, zstd"
        "br, zstd"
        zstd
        br
        gzip;

    proxy_cache_path /var/cache/nginx/proxy
        levels=1:2
        keys_zone=main_cache:10m
        max_size=1g
        inactive=24h;

    server {
        listen 80;
        server_name example.com;

        location / {
            proxy_pass http://backend;
            proxy_cache main_cache;

            # The normalized $http_accept_encoding produces
            # consistent cache keys
            proxy_cache_key "$scheme$request_method$host$request_uri$http_accept_encoding";
            proxy_cache_valid 200 10m;

            add_header X-Cache-Status $upstream_cache_status;
        }
    }
}

Verified Test Results

The following tests were conducted on Rocky Linux 10 with NGINX 1.28.2 and the module version 1.0.0. Each test sends a different Accept-Encoding header and reports the cache status:

Without the module, requests 2, 3, and 5 would have been cache MISSes. Their raw Accept-Encoding strings differ from the first request. With Accept-Encoding normalization, these become cache HITs. This effectively cuts the number of cache variants in half for this test set.

Performance Considerations

Zero Overhead for Simple Cases

The module runs only in the precontent phase. If the Accept-Encoding header is absent or the module is disabled, the handler returns immediately with no string processing. For requests with an Accept-Encoding header, the processing is lightweight. It requires one pass to parse the header and one pass to match combinations.

Cache Storage Savings

Without normalization, a single URL could generate N! cache variants. Here, N is the number of compression algorithms. Quality-value permutations multiply this further. With normalization, the maximum number of variants equals the number of configured combinations — typically 7 or fewer for a three-algorithm setup. For sites with thousands of cached URLs, this reclaims significant disk space.

Combination Order Matters

List your most common combination first. Most modern browsers support all three algorithms. Therefore, place the full combination first:

compression_normalize_accept_encoding
    "gzip, br, zstd"   # Most modern browsers — checked first
    "gzip, br"          # Older browsers without zstd
    "gzip, zstd"        # Less common
    "br, zstd"          # Less common
    zstd                # Single encoding fallbacks
    br
    gzip;

Troubleshooting

Module Not Taking Effect with return Directive

The module runs in the precontent phase. This phase executes after the rewrite phase. The return and rewrite directives finalize the response during the rewrite phase. They short-circuit before the module can normalize the header.

This is not a problem in practice. The return directive serves static responses that do not go through proxy caching. The module works correctly with proxy_pass, fastcgi_pass, uwsgi_pass, and similar content handlers.

Verifying Accept-Encoding Normalization Is Working

Add temporary debug headers to confirm the module normalizes correctly:

location / {
    add_header X-AE-Original $compression_original_accept_encoding always;
    add_header X-AE-Normalized $http_accept_encoding always;
    proxy_pass http://backend;
}

Then send a test request:

curl -s -D- -o/dev/null -H "Accept-Encoding: br, gzip" http://your-server/

You should see:

X-AE-Original: br, gzip
X-AE-Normalized: gzip, br

No Match Found — Header Unchanged

If a client sends an Accept-Encoding value that matches no configured combination (for example, deflate alone), the header remains unchanged. This is safe. The backend handles the unsupported encoding however it normally would.

Ensure your combination list covers all encodings your backend supports. If your backend only supports gzip and Brotli, you do not need zstd combinations.

Wildcard Accept-Encoding: *

The wildcard * in Accept-Encoding means “any encoding.” However, the module treats * as a literal encoding name. It will not match combinations like gzip, br. If your clients send Accept-Encoding: *, add it as a standalone combination:

compression_normalize_accept_encoding
    "gzip, br, zstd"
    "gzip, br"
    zstd br gzip
    "*";

Comparison with CDN-Level Normalization

Some CDN providers (such as Cloudflare and Fastly) perform Accept-Encoding normalization at the edge. However, this happens upstream of your origin server. If you use NGINX as a reverse proxy cache in front of your origin, the CDN’s normalization does not help. The requests reaching your NGINX proxy cache still need their own Accept-Encoding normalization.

Moreover, if you do not use a CDN at all, this module fills the gap entirely.

Conclusion

Accept-Encoding normalization is a small but impactful optimization for any NGINX proxy cache deployment. By standardizing how compression preferences are represented in cache keys, the ngx_http_compression_normalize_module eliminates redundant cache variants, improves hit rates, and reduces storage waste. Best of all, it requires zero changes to your backend applications.

The module is available as a prebuilt package for RHEL-based distributions from the GetPageSpeed repository and on GitHub.