Troubleshooting

Varnish and JetPack: “Error 503 Backend fetch failed” in WordPress

by , , revisited on

We have by far the largest RPM repository with NGINX module packages and VMODs for Varnish. If you want to install NGINX, Varnish, and lots of useful performance/security software with smooth yum upgrades for production use, this is the repository for you.
Active subscription is required.

Jetpack is quite a popular WordPress plugin and has more than a million active installs. It allows you to use WordPress.com interface to remotely manage a stack of your WordPress blogs. Varnish and Jetpack combination might be tricky.

If you are using Varnish to power up your WordPress blog, you may get “Error 503 Backend fetch failed” error when activating Jetpack.

Let’s figure out what is the reason and how can we fix this (scroll down to solution below if you’re not interested in the issue’s background). We have mentioned generic steps for troubleshooting “Backend fetch failed” error in a separate post.

Troubleshooting Jetpack and Varnish combo

First, open the SSH session for your server with Varnish. Then run varnishlog in order to make recordings of backend requests to a text file:

varnishlog -q "BereqHeader ~ '^Host: .*\.domain.com'" > bereq.log

Deactivate your Jetpack and activate it back again. You will see the error in the browser:

Error 503 Backend fetch failed

Backend fetch failed

Guru Meditation:

XID: 2230744

Take note of XID value in the response. It is Varnish transaction ID.

Cancel out from varnishlog command by typing Ctrl + C in SSH. Open the bereq.log file and search for the XID number in error response.

You will find the backend request information includes something like this:

 -   BogoHeader     Too many headers: Set-Cookie: jetpackS
-   HttpGarbage    "HTTP/1.1%00"
-   BerespStatus   503
-   BerespReason   Service Unavailable
-   FetchError     http format error

This is pretty much self-explanatory. There are too many headers being sent to backend server while activating JetPack.

According to Varnish 4 documentation, there is http_max_hdr parameter, which is:

Maximum number of HTTP header lines we allow in {req|resp|bereq|beresp}.http (obj.http is autosized to the exact number of headers). Cheap, ~20 bytes, in terms of workspace memory.

The default value for the parameter is 64.

Solution

The solution to the issue is to increase the number of headers that Varnish will accept. This is done by tuning the http_max_hdr parameter from the default 64 to 96.

We have to adjust Varnish configuration for this. It’s located in either /etc/varnish/varnish.params, /etc/sysconfig/default or /etc/default/varnish depending on your Linux OS.
You need to find DAEMON_OPTS setting which controls Varnish startup parameters and add parameter for the maximum number of HTTP headers.

Example, before:

DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
             -f ${VARNISH_VCL_CONF} \
             -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
             -p thread_pool_min=${VARNISH_MIN_THREADS} \
             -p thread_pool_max=${VARNISH_MAX_THREADS} \
             - ...

After: 

DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
             -f ${VARNISH_VCL_CONF} \
             -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
             -p    http_max_hdr=96    \ #  this is the tuned parameter we added
             -p thread_pool_min=${VARNISH_MIN_THREADS} \
             -p thread_pool_max=${VARNISH_MAX_THREADS} \
             - ...

Other Varnish and JetPack problems

The other common issue with Varnish and JetPack is that you may find that you can’t manage your blog in WordPress.com.
This is due to PHP not able to see the actual server port behind Varnish. Thus XMLRPC (the communication layer between your WordPress and remote service) breaks.

Make sure to checkout more general tips on troubleshooting Backend Fetch Failed error.

To fix for this, you have to create port and protocol mapping to translate proper information over to PHP engine. We provide this configuration as part of our WordPress Setup product (on new servers). Simply purchase a new CentOS 7 VPS with Vultr or similar hosting provider, and we’ll take care of installing perfect server setup and migrate your blog over to it.

2 thoughts on “Varnish and JetPack: “Error 503 Backend fetch failed” in WordPress”

  1. steveburi

    Hey ya, I tested everthing including file cache, memory increasing, I even switched the varnish version. But when I changed the number of headers like written in this article, it is working. Absolutly nice work. Thanks!

    Reply

  2. Anthony Blears

    This is great, thank you. Using this approach I fixed an issue where creating a new order in Woo would trigger a 503 error from Varnish. It turned out that Query Monitor was the culprit, in particular its PHP error handling for AJAX requests that would send the errors as headers (giving “Too many headers: X-QM-php_errors-erro”) in Varnish.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

This site uses Akismet to reduce spam. Learn how your comment data is processed.