Parallel ESI for Varnish 6.0 LTS: vmod-pesi Now Packaged

If you use Edge Side Includes with Varnish, here is a fact that quietly costs you response time on every cache miss: open-source Varnish processes ESI includes sequentially. One fragment after another, in document order. A page with two uncached fragments that each take 2 seconds at the backend takes over 4 seconds to deliver, even though nothing stops those fragments from being fetched at the same time.

Parallel ESI fixes exactly that, and until now it was hard to get. Varnish Enterprise ships it as a paid feature. For open-source Varnish, Uplex published libvdp-pesi, a parallel ESI delivery processor, but it targets Varnish 6.2 and newer, so users of the 6.0 LTS line (which Magento 2, among others, standardizes on) had to either self-compile a whole newer Varnish just for it, or go without.

GetPageSpeed has now backported libvdp-pesi v0.2.7 to Varnish 6.0 LTS and packaged it as vmod-pesi in our getpagespeed-extras-varnish60 repository channel. Install one package, add two lines of VCL, and your ESI fragments are fetched and assembled concurrently.

Sequential vs parallel: the arithmetic

With stock ESI, total delivery time on a miss approximates the sum of the fragment times. With parallel ESI, it approximates the slowest fragment. The more includes your pages carry, the wider the gap.

We verified it on a clean Rocky Linux 10 machine, installed exactly with the commands below: one page containing two ESI includes, each backed by a backend that responds after 2 seconds, cold cache, identical VCL apart from activation, byte-identical response bodies confirmed with cmp:

That is the expected shape: roughly 2 s + 2 s versus max(2 s, 2 s), plus overhead. With five slow fragments the sequential penalty would be five-fold; parallel delivery would still hover near the slowest single fragment. Fully cached pages are sub-millisecond either way; parallel ESI earns its keep whenever one or more includes actually hit the backend, which is precisely the moment your ESI page is at its slowest.

There is a second, less obvious benefit: the level-0 thread can start sending already-completed parts of the page to the client while other fragments are still being assembled, so time to first byte improves too, not just total time.

Installation

The package is available for Enterprise Linux 7 through 10 (RHEL, Rocky, Alma, CentOS Stream, Oracle), Amazon Linux 2 and 2023, Fedora 43 and 44, and SLES 16, on both x86_64 and aarch64. It installs alongside our Varnish 6.0 LTS build in the same channel and is built against varnish-6.0.18 exactly, so the module and the daemon can never drift apart.

On RHEL 8 and later derivatives:

sudo dnf -y install https://extras.getpagespeed.com/release-latest.rpm dnf-plugins-core
sudo dnf config-manager --enable getpagespeed-extras-varnish60
sudo dnf -y install varnish vmod-pesi

On EL7:

sudo yum -y install https://extras.getpagespeed.com/release-latest.rpm yum-utils
sudo yum-config-manager --enable getpagespeed-extras-varnish60
sudo yum -y install varnish vmod-pesi

Using the repository in production requires a GetPageSpeed subscription, which also covers 100+ NGINX modules and the rest of the extras collection.

Minimal VCL

Two things make ESI parallel. First, mark ESI pages for processing exactly as you always have. Second, call pesi.activate() in vcl_deliver:

vcl 4.0;

import pesi;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_backend_response {
    if (bereq.url ~ "\.html$" || bereq.url == "/") {
        set beresp.do_esi = true;
    }
}

sub vcl_deliver {
    pesi.activate();
}

Reload Varnish and parallel delivery is on. Your ESI markup, your backends, and your purging logic need no changes; vmod-pesi replaces the delivery side only, and it handles gzip stitching of compressed fragments and nested includes the same way stock ESI does, just concurrently.

The unconditional pesi.activate() is safe: for any response whose object carries no ESI data, activation is a no-op and stock processing applies untouched. We verified this on the same test machine: with the VCL above, a Range: bytes=0-9 request for a regular cached file still returned 206 Partial Content with exactly the requested bytes, and this exact scenario is pinned by a regression test in our build. (ESI responses themselves are delivered whole with a 200 regardless of Range, same as stock Varnish: their final length is unknown until assembly.)

Two more rules from upstream’s documentation:

1. Activate on every ESI level. If pesi.activate() runs for the top page, it must also run for its includes; the unconditional call above covers that. Mixing parallel and stock delivery across levels of one ESI tree is unsupported and can crash the worker process.
2. Order matters within vcl_deliver. Anything that affects ESI, gzip, or Range processing (for example touching req.http.Accept-Encoding) must happen before the pesi.activate() call.

Optional tuning

The module exposes a few knobs via pesi.set(), callable from vcl_deliver:

• pesi.set(serial, true) processes the next level of includes in the current thread. Sensible for deeper ESI levels where all includes are expected to be cache hits; not recommended at level 0.
• pesi.set(thread, false) uses a new thread only when one is immediately available instead of queueing for one. The default (true) is the right choice for almost everyone.
• pesi.pool() and pesi.workspace_prealloc() tune internal memory pooling; the defaults are fine in practice.

Since parallel ESI schedules fragment delivery on additional worker threads, fragment-heavy sites should sanity-check thread_pool_min/thread_pool_max the same way they would for any concurrency increase.

Magento 2 and other ESI-heavy stacks

Magento 2 is the textbook beneficiary. Its Varnish integration targets the 6.0 LTS line and renders private or short-lived blocks as ESI includes. On a cold or partially warm cache, those includes are fetched one by one today. Adding import pesi; and pesi.activate() to the Magento-generated VCL parallelizes them without touching the store itself.

The same applies to any CMS or custom stack that assembles pages from fragments with different TTLs: WordPress behind Varnish, portals with personalized sidebars, dashboards mixing cached and live widgets.

If you would rather run one cache tier with no Varnish at all, our Pro-tier nginx-module-esi brings native ESI with parallel fragment fetching to NGINX. Same parallelism principle, different layer. Keep Varnish and add vmod-pesi, or replace the sandwich with NGINX ESI; both paths are covered by the same subscription.

About the backport, honestly

vmod-pesi is a GetPageSpeed backport of Uplex’s open-source libvdp-pesi (BSD-2-Clause). The upstream project targets Varnish 6.2 and newer and never supported 6.0; this build is not supported or endorsed by Uplex, and issues with it belong with us, not them. Provenance is preserved: the package is based on the pristine upstream v0.2.7 release, adapted to the 6.0 LTS delivery API.

The adaptation was not a find-and-replace. Varnish 6.0 lacks the resp.filters mechanism that upstream uses for activation, so activation is re-implemented at the transport layer, keeping upstream’s semantics (including activation being conditional on the object actually containing ESI data). Along the way our full-matrix CI surfaced a fun one: on Fedora 43+ and SLES 16, where gcc 15 defaults to the C23 standard, bool became a reserved keyword, and an optional VCL argument named bool in the original vmod broke runtime VCL compilation on those distros only. The argument is renamed in our build (all callers are positional, so nothing changes for you), and it is a good reminder that VMOD code is compiled twice: once at build time, and again on your server every time a VCL loads.

The complete upstream varnishtest suite passes against Varnish 6.0.18 on every distro and architecture we ship, plus GetPageSpeed-added regression tests: one for a delivery-ordering issue in nested gzipped ESI found and fixed during porting, and one pinning stock Range behavior on non-ESI responses.

Try it

sudo dnf config-manager --enable getpagespeed-extras-varnish60
sudo dnf -y install vmod-pesi

Add import pesi; and pesi.activate();, reload your VCL, and watch a fragment-heavy miss render in the time of its slowest include instead of the sum of all of them. Subscribe here if you are not on the repository yet.