NGINX Module Version Mismatch: When APT Lets It Through

Every NGINX administrator who has run dynamic modules for long enough has hit an NGINX module version mismatch and seen this line:

nginx: [emerg] module "/usr/share/nginx/modules/ngx_http_brotli_filter_module.so" version 1029008 instead of 1031005

It means a module built for one NGINX release was asked to load into another. NGINX refuses, and it is right to. A dynamic module has to be built against the exact NGINX version that loads it. That part is common knowledge.

The less obvious part is that NGINX can only enforce this when the module is loaded. Your package manager has to enforce it earlier, at install time, and it can only do that if the packages describe the dependency honestly. When they don’t, an NGINX module version mismatch gets past APT, and NGINX finds it later, at the worst possible moment.

This post shows exactly how that happens, using the published metadata of one third-party Debian and Ubuntu repository. We reproduced the failure end to end, and we explain why our own packages are built so it cannot happen.

What NGINX actually checks

Every dynamic module carries two pieces of identity, both written into it at compile time by the NGX_MODULE_V1 macro: the numeric NGINX version it was built against, and a signature string. The loader in src/core/ngx_module.c checks them in that order:

if (module->version != nginx_version) {
    ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                       "module \"%V\" version %ui instead of %ui",
                       file, module->version, (ngx_uint_t) nginx_version);
    return NGX_ERROR;
}

if (ngx_strcmp(module->signature, NGX_MODULE_SIGNATURE) != 0) {
    ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                       "module \"%V\" is not binary compatible",
                       file);
    return NGX_ERROR;
}

The version check is an exact match: 1029008 is 1.29.8, 1031005 is 1.31.5, and no other value is accepted. The signature is something else. It records a handful of pointer and type sizes, then a string of feature bits for compile-time options such as epoll, threads, file AIO, QUIC and SSL. It does not encode the release.

You can see this on real binaries. We extracted the signature from a brotli module built for 1.29.8 and from an NGINX 1.31.5 binary built by the same packager:

/usr/sbin/nginx (1.31.5):                       8,4,8,0011111111010111011111111111111111
ngx_http_brotli_filter_module.so (1.29.8):      8,4,8,0011111111010111011111111111111111

They are identical. NGINX makes no promise that its internal structures stay the same from one release to the next, and the signature does not try to describe them, so the only thing that keeps a 1.29.8 module out of a 1.31.5 process is the version check. That is why there is no “sometimes it works” with mismatched NGINX versions. NGINX refuses every time. The damage comes from when it refuses.

How Debian encodes the rule for APT

Debian solved the install-time half of this years ago with a virtual package. The nginx package provides nginx-abi-<version>-<n>, every module depends on it, and the label changes whenever the ABI does:

In trixie, stream modules additionally pin libnginx-mod-stream between 1.26.3 and 1.26.3.1~. A Debian stable release never changes the upstream NGINX version, so the label stays put for the life of the release. A repository that follows NGINX mainline changes the upstream version every few weeks. For the same mechanism to work there, the label has to move with every release, or be replaced by something that does.

The metadata

Checked against the published package indexes of a third-party APT repository on 1 October 2026. We are not naming the repository; the point is the pattern, and the same check works against any repository you use. In the quotes below, the publisher’s suffix in the version strings is replaced with vendor. Everything else is verbatim. Its nginx package for Debian 12:

Package: nginx
Version: 3:1.31.5-1vendor1~bookworm
Provides: httpd, httpd-cgi, nginx-abi-1.25.0-1

And one of its modules:

Package: libnginx-mod-http-brotli
Source: nginx
Version: 3:1.31.5-1vendor1~bookworm
Depends: libbrotli1 (>= 0.6.0), libc6 (>= 2.4), nginx-abi-1.25.0-1

NGINX 1.31.5 claims the ABI of 1.25.0. The same label appears in every suite:

The repository keeps only the current version of each package, so we went looking for an older build. An older tag of the same publisher’s Docker image, last built on 6 May 2026, still contains NGINX 3:1.29.8-1vendor16~noble and 54 installed module packages. Its nginx says Provides: httpd, httpd-cgi, nginx-abi-1.25.0-1, and its modules depend on nginx-abi-1.25.0-1.

So three NGINX releases (1.29.2, 1.29.8 and 1.31.5) all declare one ABI label, and the modules each was built with are interchangeable as far as APT is concerned. NGINX itself accepts none of them across versions.

The repository’s own documentation recommends the install pattern where this matters most. It says the nginx-light, nginx-core, nginx-extras and nginx-full meta-packages are deprecated, and that you should install nginx and then add each dynamic module you need.

With a meta-package, an exact (=) pin on every module would hold the set together. Without one, the nginx-abi label is the only thing tying a module to its NGINX.

Reproducing the failure

We started from that May image: Ubuntu 24.04 with NGINX 1.29.8, the publisher’s APT source and its pin already configured. Following the documented pattern, we removed the meta-package and kept nginx plus two enabled modules, brotli and headers-more, with a single test server block. NGINX was started through its own init script and served traffic.

Then we held one module, which is what you would do after a module upgrade caused a regression, and upgraded NGINX. That is all it takes to create an NGINX module version mismatch on disk:

# dpkg-query -W nginx libnginx-mod-http-brotli libnginx-mod-http-headers-more-filter
libnginx-mod-http-brotli               3:1.29.8-1vendor16~noble
libnginx-mod-http-headers-more-filter  3:1.29.8-1vendor16~noble
nginx                                  3:1.29.8-1vendor16~noble
# curl -s http://127.0.0.1/
ok 1.29.8

# apt-mark hold libnginx-mod-http-brotli
libnginx-mod-http-brotli set on hold.
# apt-get install --only-upgrade nginx
...
60 upgraded, 9 newly installed, 0 to remove and 138 not upgraded.

APT planned the upgrade without complaint. NGINX 1.31.5 provides nginx-abi-1.25.0-1, and the held 1.29.8 brotli module depends on nginx-abi-1.25.0-1, so as far as the resolver could tell the system would stay consistent. It wouldn’t, and the package’s own maintainer script is where that came out:

Setting up nginx (3:1.31.5-1vendor1~noble) ...
 * Upgrading binary nginx
invoke-rc.d: initscript nginx, action "upgrade" failed.
 * Restarting nginx nginx
invoke-rc.d: initscript nginx, action "restart" failed.
dpkg: error processing package nginx (--configure):
 installed nginx package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of libnginx-mod-http-headers-more-filter:
 libnginx-mod-http-headers-more-filter depends on nginx-abi-1.25.0-1; however:
...
Processing was halted because there were too many errors.
E: Sub-process /usr/bin/dpkg returned an error code (1)

The post-installation script sends USR2 to the running master to start the new binary. The new binary reads the configuration, hits the 1.29.8 module and exits. The script then falls back to a restart, which tests the configuration first and refuses. dpkg marks nginx as half-configured and leaves every module package unconfigured behind it. The reason is a single line:

# nginx -t
nginx: [emerg] module "/usr/share/nginx/modules/ngx_http_brotli_filter_module.so" version 1029008 instead of 1031005 in /etc/nginx/modules-enabled/50-mod-http-brotli.conf:1
nginx: configuration file /etc/nginx/nginx.conf test failed

Here is the state the machine is left in:

• The old server keeps running. The 1.29.8 master is still in memory and still answers ok 1.29.8, which makes everything look healthy from outside.
• APT is wedged. dpkg --configure -a re-runs the same failing script. So does every later apt-get install of anything at all, even an unrelated package, because dpkg tries to finish configuring nginx first. Security updates for the rest of the system stop installing until somebody looks.
• The next restart is an outage. Starting the 1.31.5 binary against these module files fails with the same emerg. In our test a direct start exited with status 1 and the port stopped answering.

On a systemd host the fallback restart goes through systemctl restart nginx. Systemd stops the running server first and only then runs the unit’s ExecStartPre=/usr/sbin/nginx -t, which fails. That turns the wedged-but-serving state into a down server during the apt-get run itself. We could not run systemd under x86 emulation on our Apple-silicon test machine, so that last step comes from the shipped unit file and maintainer script, not from a transcript.

Recovery is simple once you know what to look for. Unhold the module and let it upgrade; dpkg finishes configuring NGINX and the 1.31.5 binary starts. The problem is that nothing points you at the held module. APT reported no dependency problem, because by the package metadata there was none.

Why APT usually hides it, and when it can’t

If you don’t hold anything, a plain upgrade on a current distribution moves NGINX and its modules together. That isn’t the nginx-abi label working. APT 2.6 and later (Debian 12 and 13, Ubuntu 24.04) upgrade the other binaries from the same source package along with the one you asked for. Every module here is built from the nginx source, so they travel as a group. With debug output on, APT says so directly:

Upgrading libnginx-mod-http-brotli:amd64 < 3:1.29.8-1vendor16~noble | 3:1.31.5-1vendor1~noble @ii ugH > due to nginx:amd64

That safety net has holes:

• Held packages. A hold beats source grouping on every APT version, as the transcript above shows.
• Older APT. Ubuntu 22.04 ships APT 2.4 and Debian 11 ships APT 2.2. Neither has source-package grouping, and the repository publishes current builds for both. We simulated this by turning grouping off (-o APT::Get::Upgrade-By-Source-Package=false): apt-get install nginx then upgrades only nginx and nginx-common and leaves every module at the old version.
• Modules from another source package. Anything you build yourself or pull from a different repository is outside the group. A truthful ABI label is the only thing that can protect it.

A dependency label that stopped changing at 1.25.0 is not a contract. All it tells APT is that it has no reason to stop you.

The same test against our packages

We repeated the experiment on Ubuntu 24.04 against our own repository: NGINX 1.31.5 with nginx-module-brotli 1.31.5 installed, brotli held, then apt-get install --only-upgrade nginx:

The following packages have unmet dependencies:
 nginx-module-brotli : Depends: nginx-r1.31.5
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.

APT refuses, with grouping on and with grouping off. Nothing is unpacked, nothing is half-configured, and nginx -t still passes. You get the error at the command prompt, on the line that caused it, with the package name in it. An NGINX module version mismatch never reaches the disk.

That is because our packages state the rule exactly as NGINX enforces it. Each NGINX build provides a virtual named after its precise upstream version, and every module requires that virtual:

Package: nginx
Version: 1:1.31.6-20~gps1+ubuntu2404+mainline
Provides: httpd, nginx, nginx-abi-1.31.6-1, nginx-r1.31.6

Package: nginx-module-brotli
Version: 1.31.6+0.1.4-9~gps1+ubuntu2404+mainline
Depends: libbrotli1 (>= 0.6.0), libc6 (>= 2.4), nginx-r1.31.6

Our RPMs use the same scheme: Provides: nginx-r%{main_version} on NGINX, Requires: nginx-r%{main_version} on every module.

Check your own repository

Three commands tell you whether your packaging makes this promise and keeps it. First, the ABI your installed NGINX claims, next to the version it actually is:

nginx -v
apt-cache show nginx | grep -E '^(Version|Provides):'

If the nginx-abi number does not match the version, the label is not tracking the ABI. Next, what your installed modules are pinned to:

dpkg-query -W -f='${Package} ${Version} ${Depends}\n' 'libnginx-mod-*' 'nginx-module-*' 2>/dev/null \
  | grep -oE '(nginx-abi|nginx-r)[^ ,]*' | sort | uniq -c

One line, naming your exact running version, is what you want. And before any restart after a package operation, ask the binary on disk, not the process in memory:

nginx -t

A failing nginx -t while the site still answers means the process in memory is not what the next start will run. It is the only warning you will get before the next restart or reboot.

Why we rebuild the whole cohort for every release

Pinning modules to the exact version is the easy half. The hard half is honouring it: every NGINX release means rebuilding every module, on every distribution and architecture, and publishing the set without ever exposing a moment where one half exists without the other. That is what our build system is for:

• Cohort rebuilds. Each NGINX release, stable or mainline, triggers a rebuild of every module against it. Our Ubuntu 24.04 mainline index currently lists 272 module dependencies on nginx-r1.31.6 and 260 on nginx-r1.31.5.
• A publish gate. A module that requires nginx-r1.31.6 cannot be published before the NGINX that provides it, and an NGINX upgrade cannot be published if it would strand an already-published module without the NGINX it requires.
• The previous release stays installable. The suite keeps the prior NGINX and its module set, so a version pin or a rollback resolves to a matching pair instead of a broken one.

Exact pins are also what make verification mean something. Our verified modules observatory records, per module, the NGINX version and platform it was built and tested against, the binary hash, and which levels passed: config inventory, multi-worker runtime, sanitizers and the lifecycle torture suite. That evidence only transfers to your server if the module you install is bound to the same NGINX version it was tested with.

This is the same position as our OpenSSL 4.0 analysis. Packaging is a set of promises to your package manager, and they are only worth what the packager does to keep them.

Getting it

On RHEL, Rocky, AlmaLinux and CentOS:

sudo dnf install https://extras.getpagespeed.com/release-latest.rpm
sudo dnf install nginx nginx-module-brotli

On Debian and Ubuntu, set up the APT repository, then:

sudo apt-get update
sudo apt-get install nginx nginx-module-brotli

Coming from the ondrej PPA? The migration guide walks through the switch, and every NGINX build we publish links OpenSSL 3.5 LTS.

Repository metadata is public; downloading packages needs an active subscription. Subscribe to the GetPageSpeed repository for 100+ NGINX modules, each rebuilt for every NGINX release and pinned to it exactly, so an upgrade either installs a matching set or stops at the prompt.