Nginx / Security / Server Setup

NGINX honeypot – the easiest and fastest way to block bots!

by , , revisited on


The Internet is not a safe place these days. Hosting a public website means exposing it to multiple attacks from evil bots, which, at best will cause extra CPU and I/O load to your server.

If your web server is NGINX, you may be rightfully tempted to make use of some 3rd party WAF modules to counter the bad guys. One such module is nginx-module-security, other is NAXSI.

But what if I told you that there’s a trick that would allow your NGINX to easily filter out 99% of the bots out there, without third-party modules? Read on to find out how.

Know your average enemy (bot)

; tldr #1 – Evil bots try to upload

Suppose that you have a WordPress blog, and sure enough, the bad guys are trying to check if they are able to find a weak spot. They do this by trying different upload endpoints of various plugins. As an example, one of the bots was trying to access:

  • /wp-content/plugins/ungallery/source_vuln.php
  • /wp-content/plugins/barclaycart/uploadify/uploadify.php
  • /wp-content/plugins/barclaycart/uploadify/settings_auto.php
  • /wp-content/plugins/hd-webplayer/playlist.php
  • /wp-content/plugins/cherry-plugin/admin/import-export/upload.php
  • /wp-content/plugins/viral-optins/api/uploader/file-uploader.php

Those plugins most likely do not even exist on your website!

So what we can obviously do, is ban any IP who attempts accessing a resource that doesn’t exist on our site. Honeypot resources are either:

  • locations that known to not exist
  • hostnames that are known to be invalid (see below)

We will now add NGINX honeypot that will work in a simple and effective way: when a malicious bot requests a known, yet non-existent upload location, NGINX will immediately ban their IP.

Let’s dive into implementation details for this kind of ban.

Pre-requisites

RHEL 7 based system, e.g. CentOS 7, and EPEL repository:

yum -y install epel-release

Setup FirewallD

We are going to create two FirewallD ipsets.
The two ipsets’ names are honeypot4 and honeypot6, for IPv4 and IPv6 addresses, respectively.

Any IP addresses placed to either of the two sets should be banned in the server firewall. To achieve this, we configure these ipsets to be in FirewallD’s drop zone.

firewall-cmd --permanent --new-ipset=honeypot4 --type=hash:ip --option=maxelem=1000000 --option=family=inet --option=hashsize=4096
firewall-cmd --permanent --new-ipset=honeypot6 --type=hash:ip --option=maxelem=1000000 --option=family=inet6 --option=hashsize=4096
firewall-cmd --permanent --zone=drop --add-source=ipset:honeypot4 
firewall-cmd --permanent --zone=drop --add-source=ipset:honeypot6 
firewall-cmd --reload

Next, we’ll teach our NGINX on when and how to add IP addresses to those ipsets.

Setup NGINX honeypot locations

Our “trap” locations in NGINX will forward requests to a special FastCGI daemon powered by fcgiwrap.

Each of those honeypot locations will include exactly the same directives to ensure passing requests to block-ip.cgi CGI script, that we’ll create later.

To make things clean, we will include those directives from a separate file:

/etc/nginx/includes/honeypot.conf

    fastcgi_intercept_errors off;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME /usr/local/libexec/block-ip.cgi;
    fastcgi_pass unix:/run/fcgiwrap/fcgiwrap-nginx.sock;

Add locations for some known plugin endpoints that do not really exist for your website:

/etc/nginx/sites-available/example.conf

location = /wp-content/plugins/ungallery/source_vuln.php {
    include includes/honeypot.conf;
}

location = /wp-content/plugins/barclaycart/uploadify/uploadify.php {
    include includes/honeypot.conf;
}

location = /wp-content/plugins/barclaycart/uploadify/settings_auto.php {
    include includes/honeypot.conf;
}

location = /wp-content/plugins/hd-webplayer/playlist.php {
   include includes/honeypot.conf;
}

location = /wp-content/plugins/cherry-plugin/admin/import-export/upload.php {
    include includes/honeypot.conf;
}

location = /wp-content/plugins/viral-optins/api/uploader/file-uploader.php {
    include includes/honeypot.conf;
}

Using exact location matching via = ensures that matching is fast and has priority over your existing \.php location.

Another approach would be regex matching. It may look somewhat cleaner as you can simply list all the plugins you don’t have in a single spot:

location ~ ^/wp-content/plugins/(ungallery|barclaycart|viral-optins|hd-webplayer)/ {
    include includes/honeypot.conf;
}

location ~* ^/phpmyadmin/ {
    include global/honeypot.conf;
}

The regex matching is prone to configuration errors because the position of these directives relative to other locations is important. Be sure to place your honeytrap regexes before your existing \.php location.

Install and configure fcgiwrap

The fcgiwrap is needed to empower our NGINX with ability to launch the bash script to ban IPs in firewall. Technically, you could use whatever existing scripting engine currently used for your website (PHP with WordPress). But for efficiency reasons, let’s avoid launching PHP interpreter for banning.

yum -y install fcgiwrap

The fcgiwrap conveniently ships with service file that allows us to launch multiple instances using different users. E.g. fcgiwrap@nginx.socket unit will launch fcgiwrap service with nginx user, and listen on a UNIX socket.

To find the UNIX socket’s path easily, you can run:

systemctl status fcgiwrap@nginx.socket

This would reveal: /run/fcgiwrap/fcgiwrap-nginx.sock.

Ensure that the service is enabled and running:

systemctl start fcgiwrap@nginx.socket
systemctl enable fcgiwrap@nginx.socket

Create the CGI script

Let’s create a CGI script at /usr/local/libexec/block-ip.cgi.
The script will call actual bash script for banning an IP, using sudo:

#!/bin/bash

echo "Status: 410 Gone"
echo "Content-type: text/plain"
echo

echo "Bye bye, $REMOTE_ADDR!"
sudo /usr/local/bin/block-ip.sh

exit 0

Any time NGINX will call this script, this will launch sudo /usr/local/bin/block-ip.sh and pass along REMOTE_ADDR.

Make the CGI script executable:

chmod 0755 /usr/libexec/block-ip.cgi

Create the bash script to block an IP

Let’s create the bash script that will be launched by nginx via sudo, that will block an IP address.

#!/bin/bash

if [ -z $REMOTE_ADDR ]; then
    if [ -z "$1" ]; then
        echo "REMOTE_ADDR not set!"
        exit 1
    else
        REMOTE_ADDR=$1
    fi
fi

# Put space separate list of trusted IP addresses, not to lock yourself out if you like to test the honeypot! : )
TRUSTED_IPS=(1.2.3.4 2.3.4.5)
if printf '%s\n' ${TRUSTED_IPS[@]} | grep -q -P "^$REMOTE_ADDR\$"; then
    echo "Trusted IP"
    exit 0
fi

if [ "$REMOTE_ADDR" != "${1#*[0-9].[0-9]}" ]; then
  /sbin/ipset add honeypot4 $REMOTE_ADDR
elif [ "$REMOTE_ADDR" != "${1#*:[0-9a-fA-F]}" ]; then
  /sbin/ipset add honeypot6 $REMOTE_ADDR
else
  echo "Unrecognized IP format '$1'"
fi

We could also use firewall-cmd to add to ipsets, but of course, we want to avoid the heavy lifting of the Python interpreter. On a 1 CPU VPS, firewall-cmd --ipset=honeypot4 --add-entry=... runs 0m0.494s while pure binary ipset add honeypot4 ... took only 0m0.002s to run!

sudo!!

What else is missing? Surely enough nginx runs under non-privileged user and can’t sudo /usr/local/bin/block-ip.sh, yet. We want to allow nginx to gain privileges to run the script as superuser. Moreover, for security reasons, we will allow only REMOTE_ADDR environment variable to be passed while launching the script.

So create /etc/sudoers.d/nginx-block-ip with contents:

Defaults!/usr/local/bin/block-ip.sh env_keep=REMOTE_ADDR
nginx ALL=(ALL) NOPASSWD: /usr/local/bin/block-ip.sh

Secure permissions of that file:

chmod 0440 /etc/sudoers.d/nginx-block-ip

That’s it! Restart NGINX and you’re good to go, NGINX will start banning bots for you. But there’s even more you can do… 🙂

Huge Improvement with hostname honeypot

; tldr #2 – Evil bots don’t even know your domain name!

Most of “current wave” bots will only know your IP, because they are scanning public IPv4 ranges and iterating one IP after another as their victim.

Those bots will share these common characteristics:

  • Since all they know is your IP, they request resources with either an empty or fake Host header
  • They will only make HTTP requests (not HTTPS)

So you can greatly reduce the load from those bots by blocking any client that does not provide valid hostnames. Obviously, valid hostnames are simply all domains that you host on your server, and any other domain would be an invalid hostname.

In /etc/nginx/nginx.conf, setup a map listing all your website domain names:

map $http_host $default_host_match {
    getpagespeed.com 1;
    www.getpagespeed.com 1;
    default 0;
}

In a server location of your websites, add honeypot for bad hostname:

error_page 410 = @honeypot;
if ($default_host_match = 0) {
    return 410;
}
location @honeypot {
    include includes/honeypot.conf;
}

With the configuration above, NGINX will check if the requested domain is in the list specified in nginx.conf. If there is no match, then 410 status code is returned, which is handled by the @honeypot named location, which, of course, launches our bash script for banning.

How this compares to anything

Surely enough, you should not use this approach alone. There is never “enough security”, and you should use Fail2ban, Malware Detect, and ModSecurity.

However, we can see how the honeypot approach can complement the mentioned tools, and soften their disadvantages.

For example:

  • Fail2ban continuously scans the log files in an attempt for early blocking of offenders; the honeypot banning matches immediately and reduces both the load and the log noise tremendously
  • Malware Detect finds uploaded malware, whereas honeypot banning (and ModSecurity) prevent their upload in the first place

So other tools may be either slow or “too late”. They are surely enough useful. But now we can do better with additional security layer thanks to the NGINX honeypot approach.


Also published on Medium.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.