Troubleshooting / Varnish

Varnish “Backend fetch failed”

by , , revisited on


We have by far the largest RPM repository with dynamic stable NGINX modules and VMODs for Varnish 4.1 and 6.0 LTS. If you want to install nginx, Varnish and lots of useful modules for them, this is your one stop repository to get all performance related software.
You have to maintain an active subscription in order to be able to use the repository!

Here are some generic tips on finding the actual cause of “Backend Fetch Failed” error shown by Varnish Cache.

Why backend fetch fails

Common reasons of getting “Backend fetch failed” error:

  • Varnish timeout was hit
  • A backend health probe failed – your backend is marked as sick
  • Your PHP has generated fatal error (HTTP code > 500)
  • Varnish can’t communicate to the backend
  • Varnish cannot find an object to evict from cache while serving large file (ExpKill LRU_Exhausted and FetchError Could not get storage in varnishlog), similiar to this issue. Solution is often to return (pipe) for large files

Use varnishlog to see why backend fetch failed for particular request

It much better to hunt down specific request that generated the error. When you get “Backend fetch failed” in browser, take note of “XID: 12345” at the bottom of the page. Use that number to get relevant requests quickly.

varnishlog -q "vxid == 12345"

In versions prior to 5.1 you can use the X-Varnish header:

First, you will get client side request using the following command:

varnishlog -d -q 'RespHeader:X-Varnish[1] == 12345'

This will display the client side request to Varnish. Investigate it for ID of the backend request. Find a line that looks like this:

-   Link bereq 123456 pass

Now you can investigate actual errors. Review request details of Varnish connection to the backend:

varnishlog -d -q 'BereqHeader:X-Varnish == 123456'

Use varnishlog to see all failed requests

varnishlog -d -q "BerespStatus == 503"

Will show latest in-memory Varnish log entries for “Backend fetch failed”.

It may produce too many entries and quickly fill your screen. A better approach is to save in-memory Varnish log entries into a file.

varnishlog -A -d -w /var/log/varnish/varnish50x.log -q "RespStatus >= 500 or BerespStatus >= 500"

Now we can investigate the file and find out details why we have failed requests.

Enable varnishlog service for collecting longer history of failed requests

One issue with the above approaches, is that they allow us to lookup only recent requests. Varnish memory log obviously can’t hold for long period of logged data.

Let’s enable varnishlog service

systemctl start varnishlog

Now we can read all the failed requests from the binary log that is being recorded by varnishlog daemon.

varnishlog -r /var/log/varnish/varnish.log -q "RespStatus >= 500 or BerespStatus >= 500"

You can adjust how many days of varnish logs are being kept on server by looking into /etc/logrotate.d/varnish.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.