Server Setup

Varnish static files stream

by , , revisited on


We have by far the largest RPM repository with dynamic stable NGINX modules and VMODs for Varnish 4.1 and 6.0 LTS. If you want to install NGINX, Varnish and lots of useful modules for them, this is your one stop repository to get all performance related software.
You have to maintain an active subscription in order to be able to use the repository!

As mentioned in earlier post, one of the best performance stacks out there is Varnish coupled with Nginx. To make Varnish deliver static files efficiently, we looked at an approach to cache static files in Varnish.

But what about large static files? If we have downloadable .zip or .rar files, or we want users of site to listen to MP3 songs, it is not wise to store those in Varnish cache, because we would waste too much disk space.

Luckily, Varnish 4 comes with the streaming feature. When streaming is used, Varnish will deliver requested file from the backend (Nginx), byte by byte. It will start the transfer without waiting to receive complete file from Nginx. So our goal is to reduce caching of static files only to smaller static files, and deliver larger files via streaming. Using the earlier snippet for caching static files, we need to add a couple of additional statements inside .vcl file used.

First, tell Varnish not to cache the large files:

sub vcl_recv {
…
        if (req.url ~ "^[^?]*\.(mp[34]|rar|rpm|tar|tgz|gz|wav|zip|bz2|xz|7z|avi|mov|ogm|mpe?g|mk[av]|webm)(\?.*)?$")            {
            unset req.http.Cookie;
            return (hash);
        }
}
…

When fetching the file from Nginx, open static files stream:

sub vcl_backend_response {
…
       if (bereq.url ~ "^[^?]*\.(mp[34]|rar|rpm|tar|tgz|gz|wav|zip|bz2|xz|7z|avi|mov|ogm|mpe?g|mk[av]|webm)(\?.*)?$") {
          unset beresp.http.set-cookie;
          set beresp.do_stream = true;  
       }

With this simple technique, we’ve enabled streaming and saved disk space by caching only HTML, CSS, Javascript and other smaller site assets.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.