Server Setup

Varnish: do we have to cache static files?

by , , revisited on


We have by far the largest RPM repository with dynamic stable NGINX modules and VMODs for Varnish 4.1 and 6.0 LTS. If you want to install NGINX, Varnish and lots of useful modules for them, this is your one stop repository to get all performance related software.
You have to maintain an active subscription in order to be able to use the repository!

One of the best performance stacks out there is Varnish + Nginx coupled with PHP-FPM.

If we run a site using this setup, all the requests would go through Varnish.

Varnish is a great caching solution to speed up any website.
But does it need to cache static files which are already fast to deliver?

Let’s talk about best approaches to deliver static files. How can we make static files delivery fastest possible?

Don’t cache static files with Varnish. Use separate Nginx for it

In theory, Nginx outperforms Varnish when it comes to serving static files. As such, we should not serve static files via Varnish in the first place
So we have to split our configuration in two parts:

  1. additional Nginx on a separate IP (or VPS) solely for serving static files: static.example.com
  2. Varnish + Nginx + PHP-FPM solely for the time consuming PHP: www.example.com

This kind of setup requires to link static files on a separate domain, i.e. static.domain.com (where we host Nginx only) and has the following pros and cons:

  • we solve cookie-free domain optimisation practice (requests to static.domain will not contain cookies)
  • requires purchasing additional IP or VPS (because we can’t run Nginx #2 and Varnish on the same port)

This setup is beneficial for large scale projects. And as long as you have the money for more servers or IP, it should be considered.

Cache static files with Varnish. And do it smart, even on a budget

Now let’s assume we’re on a budget, and we can’t buy an additional IP and / or we want wait for our project to grow.
To make Varnish deliver static files nearly as fast as Nginx, we have to cache static files in Varnish.
Many people recommend against it since caching static files in Varnish would waste RAM, a resource that is quite precious on a small VPS.
Varnish static files cache appears not needed since those files are “fast” already. But as long as we can cache those, there is a performance improvement.

We can make a smart move and use multiple storage backends by partitioning Varnish cache:

  • cache static files onto HDD
  • cache everything else onto RAM

This way we don’t waste RAM for storing static files. The cache is split into two storages: RAM, for HTML and HDD, for static files.

To make this happen we need to update Varnish configuration. Assuming that we run Varnish 4 on CentOS 6, we have to update two files.

Varnish configuration: /etc/sysconfig/varnish

This file contains storage definition, and we have to replace it and specify 2 storages for our cache: “static” and “default”.

Find -s ${VARNISH_STORAGE}"

and replace with:

-s default=malloc,256m \
-s static=file,/var/lib/varnish/varnish_storage.bin,1G"

So here we have a 1GB of HDD dedicated to static files cache and 256MB for the rest in RAM.

VCL configuration file

We have to tell Varnish which responses should go into which cache, so in our .vcl file we have to update vcl_backend_response routine, like this:


sub vcl_backend_response {

        # For static content strip all backend cookies and push to static storage
        if (bereq.url ~ "\.(css|js|png|gif|jp(e?)g)|swf|ico") {
                unset beresp.http.cookie;
                set beresp.storage_hint = "static";
                set beresp.http.x-storage = "static";
        } else {
                set beresp.storage_hint = "default";
                set beresp.http.x-storage = "default";
        }

}

As simple as that: if files match static resources, we cache into static storage (which is HDD), whereas everything else (most notably html pages) is cached into RAM.

Now we can observe the two storages filling up by running sudo varnishstat. You will find SMA.default and SMF.static groups:

Varnish storages split
Varnish storages split
  1. Tunisie annonces

    Thank you very much !
    It was very helpful for me.

    Tip: I had put all static files in static storage using this condition:

    
    if (bereq.url ~ "^[^?]*\.(bmp|bz2|css|doc|eot|flv|gif|gz|ico|jpeg|jpg|js|less|mp[34]|pdf|png|rar|rtf|swf|tar|tgz|txt|wav|woff|xml|zip|webm)(\?.*)?$") {
                set beresp.storage_hint = "static";
                set beresp.http.x-storage = "static";
        } else {
                set beresp.storage_hint = "default";
                set beresp.http.x-storage = "default";
        }
    
    Reply
  2. Danila Vershinin

    Hi Tunisie,

    You may wish to move out some of the file types from your “if” statement so that they are not cached. Use streaming for those instead. Caching large mp3, rar, wav, etc. files is not efficient. See a new post about static files stream with Varnish.

    Reply
  3. benya

    My virtual server with 512 MB of RAM. Installation can I do? There is no problem? !!

    Reply
    • Danila Vershinin

      There is no problem installing Varnish on a 512 MB RAM VPS.

      Reply
  4. mi2test

    Hi, can you please tell me why my Age is showing 0, its look like varnish is not caching anything, my vcl file is

    
    vcl 4.0;
    
    
    backend default {
        .host = "127.0.0.1";
        .port = "8080";
    }
    
    sub vcl_recv {
         if (req.method == "PURGE") {
                return (purge);
         }
    
    }
    
    
    sub vcl_backend_response {
    
            # For static content strip all backend cookies and push to static storage
            if (bereq.url ~ "\.(css|js|png|gif|jp(e?)g)|swf|ico") {
                    unset beresp.http.cookie;
                    set beresp.storage_hint = "static";
                    set beresp.http.x-storage = "static";
            } else {
                    set beresp.storage_hint = "default";
                    set beresp.http.x-storage = "default";
            }
    
    }
    
    sub vcl_deliver {
    }
    
    Reply
  5. Danila Vershinin

    The caching depends on the HTTP headers that your website outputs. Varnish will cache things based on presence of Expires, Cache-Control, Set-Cookie headers. Your VCL is quite “empty”.

    You need at least some rules to filter out cookies – the common practice is white-listing cookies that your application needs.

    Reply
  6. Gabriel

    Hello,

    It is working for me. Thanks so much! Great post.

    One question: Do you think there will be a little difference when serving static content from HDD as opposed to serving from RAM? Will memory in RAM still be technically faster?

    Best regards,
    Gabriel

    Reply
    • Danila Vershinin

      RAM is always technically faster. However, provided that you have SSD for your drive, the performance difference will not be that important.

      Reply
  7. iarijitbiswas

    If I use WP-Rocket caching plugin, do I even need Varnish Caching? I’m using ServerPilot. I have 95k posts, handling with a 8gb ram vps just fine. If I install Varnish Cache, how much improvement I can expect? 10%? or 50%?

    Reply
    • Danila Vershinin

      Those are different things. And you can actually use both. Varnish is a full page cache, it can do wonders for high traffic websites.
      A full page cache is essentially as fast as serving static HTML. So you can serve thousands and thousands of visitors from even very modest hardware.

      Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.