Varnish

Varnish – cache on cookies

by , , revisited on


We have by far the largest RPM repository with dynamic stable NGINX modules and VMODs for Varnish 4.1 and 6.0 LTS. If you want to install nginx, Varnish and lots of useful modules for them, this is your one stop repository to get all performance related software.
You have to maintain an active subscription in order to be able to use the repository!

In ongoing process of learning Varnish, I’ve stumbled upon this topic now and then. Default behavior of Varnish is to not deliver cached pages for requests with cookies and not cache pages that have Set-Cookie in backend response.

The standard approach to leverage Varnish with a PHP app is to strip all cookies but the ones which are absolutely necessary. When a client sends request for a page with an essential app cookie (e.g. logged in user) – the page is delivered uncached. Other times (e.g. guest user) the page is delivered from cache.

With this approach, we are surely missing out on cache for logged in users (or other cases where users should be presented with different content, for example language or timezone).

If we want Varnish to cache those pages as well, we need a few bits of VCL to make things right 🙂

First thing to account for, is that the default builtin.vcl does not allow a request with Cookie to be delivered from cache:

if (req.http.Authorization || req.http.Cookie) {
    /* Not cacheable by default */
    return (pass);
}

It goes straight to backend. We want to change that. In your own VCL you should have a return statement. Its presence will ensure that the builtin.vcl logic will not be run:

In your own vcl_recv, put:

# ....
if (req.http.Authorization) {
    /* Not cacheable by default */
    return (pass);
}

return(hash);

Now second thing we should do is adjust or add vcl_hash procedure to tell Varnish that cache for a page should be different based on the value of the Cookie that we want to cache with.

sub vcl_hash {
  if (req.http.cookie ~ "mycookie=") {
    set req.http.X-TMP = regsub(req.http.cookie, ".*mycookie=([^;]+);.*", "\1");
    hash_data(req.http.X-TMP);
    remove req.http.X-TMP;
  }
  # the builtin.vcl will take care of also varying cache on Host/IP and URL 
}

The suggested approach from the mailing list (useful Varnish resource) is to use cookie vmod :

I highly recommend using vmod cookie to avoid the regex madness. I’d also extract the cookies into their own headers and hash them inconditionally, giving something like:

sub vcl_recv {
    cookie.parse(req.http.cookie);
    set req.http.cookie1 = cookie.get("COOKIE1");
    set req.http.cookie2 = cookie.get("COOKIE2");
    unset req.http.cookie;
}

sub vcl_hash {
    hash_data(req.http.cookie1);
    hash_data(req.http.cookie2);
}

Suggested read:

  1. sina

    Hi, I have the same problem. I want to cache the all pages of my web site except the cookies. I want to have a fresh PHPSESSID and other user defined cookies while the request is responded from cache. for example the fisrt PHPSESSID=ev4vfmf0iukl9j0sn509bvuv7 and if I clean the cookies in my browser I get the fresh value for PHPSESSID. I did as you said in this article:

       if (req.http.cookie ~ "PHPSESSID=") {
        set req.http.X-SESSION = regsuball(req.http.Cookie, ";(PHPSESSID)=", "; \1=");
        hash_data(req.http.X-SESSION);
        unset req.http.X-SESSION;
      }
    

    but this has not resolve my problem. I still cannot see the PHPSESSID in response header in Chrome browser.

    Reply
    • Danila Vershinin

      If you cannot see the PHPSESSID in HTTP response headers, this only means that you have extra VCL code which unsets the cookie when your server sends it.
      Obviously, that code has to be removed.

      Reply
  2. sina

    I changed my VCL config. now I can see the PHPSESSID in Request headers in chrome. it seems it is working like I expect. When I remove cookies I can get the new value for PHPSESSID. But I still have two problems the first is that when I remove the cookies from browser, I have to refresh the page at least 3 times to get the page from cache! Another issue is that I want to have the values of four more cookies but with my VCL code I can only see the PHPSESSID! Here’s my VCL code:

    sub vcl_recv {
     set req.backend_hint = apache.backend();
    
    if (req.http.Cookie)
       {
        set req.http.Cookie = ";" + req.http.Cookie;
        set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
        set req.http.Cookie = regsuball(req.http.Cookie, ";(used_ads|used_doctors|used_natives|PHPSESSID|csrf)=", "; \1=");
        set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");
    
        if (req.http.Cookie == "") {
            unset req.http.Cookie;
        }
    
    }
    
    if ((req.url ~ "admin-ajax.php") && !req.http.cookie ~ "wordpress_logged_in" ) {
          return (hash);
      }
    
    if (req.http.X-Requested-With == "XMLHttpRequest") {
            return(pass);
        }
    
    if (req.method != "GET" && req.method != "HEAD") {
          return (pass);
      }
    
    if (req.url ~ "(wp-admin|post.php|edit.php|wp-login|forms)") {
            return(pass);
      }
      if (req.url ~ "/wp-cron.php" || req.url ~ "preview=true") {
            return (pass);
      }
    
    if (req.http.Authorization) {
            return(pass);
      }
    
    if (req.http.Accept-Encoding)
      {
        if (req.url ~ ".(png|jpg|jpeg|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$")
            {
                    unset req.http.Accept-Encoding;
            }
            elsif (req.http.Accept-Encoding ~ "gzip")
            {
                    set req.http.Accept-Encoding = "gzip";
            }
            elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE")
            {
                    set req.http.Accept-Encoding = "deflate";
            }
            else
            {
                    unset req.http.Accept-Encoding;
            }
      }
    
        set req.http.X-Forwarded-For = req.http.X-Forwarded-For;
    
    
        unset req.http.Accept-Language;
        unset req.http.User-Agent;
    
        set req.http.cookie = regsuball(req.http.cookie, "wp-settings-\d+=[^;]+(; )?", "");
        set req.http.cookie = regsuball(req.http.cookie, "wp-settings-time-\d+=[^;]+(; )?", "");
        set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");
    
        #cache everything left behind
        return(hash);
    
    }
    
    sub vcl_hash
    {
    
    if (req.http.cookie ~ "(used_ads|used_doctors|used_natives|PHPSESSID|csrf)") {
        set req.http.X-TMP = regsuball(req.http.Cookie, ";(used_ads|used_doctors|used_natives|PHPSESSID|csrf)=", "; \1=");
        hash_data(req.http.X-TMP);
        unset req.http.X-TMP;
      }
    
    }
    
    sub vcl_backend_response {
            if ((bereq.url ~ "admin-ajax.php") && !bereq.http.cookie ~ "wordpress_logged_in" ) {
                unset beresp.http.set-cookie;
                set beresp.ttl = 12h;
             }
    
        if (!(bereq.url ~ "wp-(login|admin)|login|admin-ajax.php|forms"))
    
        {
                set beresp.ttl = 6h;
        }
    
    
        set beresp.grace = 2h;
    
    
    if (beresp.http.Location == "https://" + bereq.http.host + bereq.url)
    {
       if (bereq.retries > 1)
       {
          unset beresp.http.Location;
       }
       else
       {
          return (retry);
       }
    }
    
    }
    
    sub vcl_deliver {
    
    if (obj.hits > 0)
    {
           set resp.http.X-Status = "1";
    }
    else
    {
           set resp.http.X-Status = "0";
    }
    
    unset resp.http.X-Varnish;
     unset resp.http.Via;
     unset resp.http.X-Powered-By;
     unset resp.http.Server;
    
    }
    
    Reply
  3. Danila Vershinin

    If your app is WordPress, you should rather not cache at all in presence of WordPress specific cookies.
    You can cache user session though, but that means you should also develop the code to talk to Varnish and invalidate user-cache in Varnish when something changes for particular user, or just use very short TTL.

    Also, PHPSESSID is a regular PHP cookie name, so that means one of the plugins is not following on WordPress conventions. It is best to get rid of those.

    Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.