NGINX / Security / Server Setup

NGINX honeypot – the easiest and fastest way to block bots!

by , , revisited on


We have by far the largest RPM repository with NGINX module packages and VMODs for Varnish. If you want to install NGINX, Varnish, and lots of useful performance/security software with smooth yum upgrades for production use, this is the repository for you.
Active subscription is required.

The Internet is not a safe place these days. Hosting a public website means exposing it to multiple attacks from evil bots, which, at best will cause extra CPU and I/O load to your server.

If your web server is NGINX, you may be rightfully tempted to make use of some 3rd party WAF modules to counter the bad guys. One such module is nginx-module-security, other is NAXSI.

But what if I told you that there’s a trick that would allow your NGINX to easily filter out 99% of the bots out there, without third-party modules? Read on to find out how.

Know your average enemy (bot)

; tldr #1 – Evil bots try to upload

Suppose that you have a WordPress blog, and sure enough, the bad guys are trying to check if they are able to find a weak spot. They do this by trying different upload endpoints of various plugins. As an example, one of the bots was trying to access:

  • /wp-content/plugins/ungallery/source_vuln.php
  • /wp-content/plugins/barclaycart/uploadify/uploadify.php
  • /wp-content/plugins/barclaycart/uploadify/settings_auto.php
  • /wp-content/plugins/hd-webplayer/playlist.php
  • /wp-content/plugins/cherry-plugin/admin/import-export/upload.php
  • /wp-content/plugins/viral-optins/api/uploader/file-uploader.php

Those plugins most likely do not even exist on your website!

So what we can obviously do, is ban any IP who attempts accessing a resource that doesn’t exist on our site. Honeypot resources are either:

  • locations that known to not exist
  • hostnames that are known to be invalid (see below)

We will now add an NGINX honeypot that will work in a simple and effective way: when a malicious bot requests a known, yet non-existent upload location, NGINX will immediately ban their IP.

Let’s dive into implementation details for this kind of ban.

Pre-requisites

RHEL 7 based system, e.g. CentOS 7, and EPEL repository:

yum -y install epel-release

Setup FirewallD

We are going to create two FirewallD IP sets.
The two IP sets’ names are honeypot4 and honeypot6, for IPv4 and IPv6 addresses, respectively.

Any IP addresses placed to either of the two sets should be banned in the server firewall. To achieve this, we configure these IP sets to be in FirewallD’s drop zone.

firewall-cmd --permanent --new-ipset=honeypot4 --type=hash:ip --option=maxelem=1000000 --option=family=inet --option=hashsize=4096
firewall-cmd --permanent --new-ipset=honeypot6 --type=hash:ip --option=maxelem=1000000 --option=family=inet6 --option=hashsize=4096
firewall-cmd --permanent --zone=drop --add-source=ipset:honeypot4 
firewall-cmd --permanent --zone=drop --add-source=ipset:honeypot6 
firewall-cmd --reload

Next, we’ll teach our NGINX when and how to add IP addresses to those IP sets.

Setup NGINX honeypot locations

Our “trap” locations in NGINX will forward requests to a special FastCGI daemon powered by fcgiwrap.

Each of those honeypot locations will include exactly the same directives to ensure passing requests to block-ip.cgi CGI script, that we’ll create later.

To make things clean, we will include those directives from a separate file:

/etc/nginx/includes/honeypot.conf

    fastcgi_intercept_errors off;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME /usr/local/libexec/block-ip.cgi;
    fastcgi_pass unix:/run/fcgiwrap/fcgiwrap-nginx.sock;

Add locations for some known plugin endpoints that do not really exist for your website:

/etc/nginx/sites-available/example.conf

location = /wp-content/plugins/ungallery/source_vuln.php {
    include includes/honeypot.conf;
}

location = /wp-content/plugins/barclaycart/uploadify/uploadify.php {
    include includes/honeypot.conf;
}

location = /wp-content/plugins/barclaycart/uploadify/settings_auto.php {
    include includes/honeypot.conf;
}

location = /wp-content/plugins/hd-webplayer/playlist.php {
   include includes/honeypot.conf;
}

location = /wp-content/plugins/cherry-plugin/admin/import-export/upload.php {
    include includes/honeypot.conf;
}

location = /wp-content/plugins/viral-optins/api/uploader/file-uploader.php {
    include includes/honeypot.conf;
}

Using exact location matching via = ensures that matching is fast and has priority over your existing \.php location.

Another approach would be regex matching. It may look somewhat cleaner as you can simply list all the plugins you don’t have in a single spot:

location ~ ^/wp-content/plugins/(ungallery|barclaycart|viral-optins|hd-webplayer)/ {
    include includes/honeypot.conf;
}

location ~* ^/phpmyadmin/ {
    include global/honeypot.conf;
}

TIP: talking about PhpMyAdmin, don’t be someone who installs it, eh?

The regex matching is prone to configuration errors because the position of these directives relative to other locations is important. Be sure to place your honeytrap regexes before your existing \.php location.

Install and configure fcgiwrap

The fcgiwrap is needed to empower our NGINX with the ability to launch the bash script to ban IPs in the firewall. Technically, you could use whatever existing scripting engine currently used for your website (PHP with WordPress). But for efficiency reasons, let’s avoid launching the PHP interpreter for banning.

yum -y install fcgiwrap

The fcgiwrap conveniently ships with a service file that allows us to launch multiple instances using different users. E.g. fcgiwrap@nginx.socket unit will launch fcgiwrap service with nginx user, and listen on a UNIX socket.

To find the UNIX socket’s path easily, you can run:

systemctl status fcgiwrap@nginx.socket

This would reveal: /run/fcgiwrap/fcgiwrap-nginx.sock.

Ensure that the service is enabled and running:

systemctl start fcgiwrap@nginx.socket
systemctl enable fcgiwrap@nginx.socket

Install conntrack-tools

NGINX, like any other web server, supports keepalive connections.
Simply blocking an IP in the firewall is not sufficient, because it affects only future connections.
If bots are smart enough to use keepalive (which is easy to implement), they can still make malicious requests over the first connection.

We need conntrack program, that will be invoked in the honeypot, to destroy established connections with an intruder’s IP address.

sudo yum -y install conntrack-tools

Create the CGI script

Let’s create a CGI script at /usr/local/libexec/block-ip.cgi.
The script will call actual bash script for banning an IP, using sudo:

#!/bin/bash

echo "Status: 410 Gone"
echo "Content-type: text/plain"
echo

echo "Bye bye, $REMOTE_ADDR!"
sudo /usr/local/bin/block-ip.sh

exit 0

Any time NGINX will call this script, this will launch sudo /usr/local/bin/block-ip.sh and pass along REMOTE_ADDR.

Make the CGI script executable:

chmod 0755 /usr/libexec/block-ip.cgi

Create the bash script to block an IP

Let’s create the bash script that will be launched by NGINX via sudo, that will block an IP address.

#!/bin/bash

if [[ -z ${REMOTE_ADDR} ]]; then
    if [[ -z "$1" ]]; then
        echo "REMOTE_ADDR not set!"
        exit 1
    else
        REMOTE_ADDR=$1
    fi
fi

# Put space separate list of trusted IP addresses, not to lock yourself out if you like to test the honeypot! : )
TRUSTED_IPS=(1.2.3.4 2.3.4.5 127.0.0.1)
if printf '%s\n' ${TRUSTED_IPS[@]} | grep -q -P "^$REMOTE_ADDR\$"; then
    echo "Trusted IP"
    exit 0
fi

if [[ "$REMOTE_ADDR" != "${1#*[0-9].[0-9]}" ]]; then
  /sbin/ipset add honeypot4 ${REMOTE_ADDR}
  /sbin/conntrack -D -s ${REMOTE_ADDR}
elif [[ "$REMOTE_ADDR" != "${1#*:[0-9a-fA-F]}" ]]; then
  /sbin/ipset add honeypot6 ${REMOTE_ADDR}
  /sbin/conntrack -D -s ${REMOTE_ADDR}
else
  echo "Unrecognized IP format '$1'"
fi

We could also use firewall-cmd to add to IP sets, but of course, we want to avoid the heavy lifting of the Python interpreter. On a 1 CPU VPS, firewall-cmd --ipset=honeypot4 --add-entry=... runs 0m0.494s while pure binary ipset add honeypot4 ... took only 0m0.002s to run!

That said, if you don’t care about the millisecond’s differences between different ways of blocking, using fds program will yield the cleanest approach.

sudo yum -y install https://extras.getpagespeed.com/release-latest.rpm
sudo yum -y install fds

With fds, our script becomes:

#!/bin/bash

if [[ -z ${REMOTE_ADDR} ]]; then
    if [[ -z "$1" ]]; then
        echo "REMOTE_ADDR not set!"
        exit 1
    else
        REMOTE_ADDR=$1
    fi
fi

# Put space separate list of trusted IP addresses, not to lock yourself out if you like to test the honeypot! : )
TRUSTED_IPS=(1.2.3.4 2.3.4.5 127.0.0.1)
if printf '%s\n' ${TRUSTED_IPS[@]} | grep -q -P "^$REMOTE_ADDR\$"; then
    echo "Trusted IP"
    exit 0
fi

fds block ${REMOTE_ADDR} --ipset honeypot

sudo!!

What else is missing? Surely enough NGINX runs under non-privileged user and can’t sudo /usr/local/bin/block-ip.sh, yet. We want to allow nginx to gain privileges to run the script as the superuser. Moreover, for security reasons, we will allow only REMOTE_ADDR environment variable to be passed while launching the script.

So create sudoers configuration by running sudo visudo -f /etc/sudoers.d/nginx-block-ip.
This command will open up the system editor (likely vim), in edit mode. Simply paste in:

Defaults!/usr/local/bin/block-ip.sh env_keep=REMOTE_ADDR
nginx ALL=(ALL) NOPASSWD: /usr/local/bin/block-ip.sh

Then close the editor normally, e.g. by typing :wq for vim, or Ctrl+X for nano.

That’s it! Restart NGINX and you’re good to go, NGINX will start banning bots for you. But there’s even more you can do… 🙂

Huge Improvement with hostname honeypot

; tldr #2 – Evil bots don’t even know your domain name!

Most of “current wave” bots will only know your IP, because they are scanning public IPv4 ranges and iterating one IP after another as their victim.

Those bots will share these common characteristics:

  • Since all they know is your IP, they request resources with either an empty or fake Host header
  • They will only make HTTP requests (not HTTPS)

So you can greatly reduce the load from those bots by blocking any client that does not provide valid hostnames. Obviously, valid hostnames are simply all domains that you host on your server, and any other domain would be an invalid hostname.

In /etc/nginx/nginx.conf, setup a map listing all your website domain names:

map $http_host $default_host_match {
    getpagespeed.com 1;
    www.getpagespeed.com 1;
    default 0;
}

In a server location of your websites, add honeypot for bad hostname:

error_page 410 = @honeypot;
if ($default_host_match = 0) {
    return 410;
}
location @honeypot {
    include includes/honeypot.conf;
}

With the configuration above, NGINX will check if the requested domain is in the list specified in nginx.conf. If there is no match, then 410 status code is returned, which is handled by the @honeypot named location, which, of course, launches our bash script for banning.

How this compares to anything

Surely enough, you should not use this approach alone. There is never “enough security”, and you should use Fail2ban, Malware Detect, and ModSecurity.

However, we can see how the honeypot approach can complement the mentioned tools, and soften their disadvantages.

For example:

  • Fail2ban continuously scans the log files in an attempt for early blocking of offenders; the honeypot banning matches immediately and reduces both the load and the log noise tremendously
  • Malware Detect finds uploaded malware, whereas honeypot banning (and ModSecurity) prevent their upload in the first place

So other tools may be either slow or “too late”. They are surely enough useful. But now we can do better with the additional security layer thanks to the NGINX honeypot approach.

  1. Raphael

    FOR UBUNTU AND IPSET AT STARTUP AND VERIFICATION EMAIL

    apt-get install ipset

    ipset create honeypot4 hash:ip maxelem 1000000 family inet hashsize 4096
    ipset create honeypot6 hash:ip maxelem 1000000 family inet6 hashsize 4096

    iptables -I INPUT -m set –match-set honeypot4 src -j DROP
    iptables -I FORWARD -m set –match-set honeypot4 src -j DROP
    ip6tables -I INPUT -m set –match-set honeypot6 src -j DROP
    ip6tables -I FORWARD -m set –match-set honeypot6 src -j DROP

    sudo -s iptables-save -c
    sudo -s ip6tables-save -c

    //load the sets at startup
    apt-get install netfilter-persistent
    sudo systemctl enable netfilter-persistent

    in /etc/init.d/netfilter-persistent add “ipset save>/etc/ipset.rules” just after “save)”
    in /usr/sbin/netfilter-persistent add after “start|save|flush)”:
    #if you use Failtoban
    perl -pi -e “s/^create f2b-sshd /#create f2b-sshd /” /etc/ipset.rules;
    perl -pi -e “s/^add f2b-ssh /#add f2b-ssh /” /etc/ipset.rules;
    perl -pi -e “s/^add f2b-sshd /#add f2b-sshd /” /etc/ipset.rules;
    #
    ipset restore</etc/ipset.rules;
    //load the sets at startup

    service netfilter-persistent save
    //if error check sudo nano /etc/ipset.rules

    //test with command
    sudo ipset list honeypot4

    if return
    Name: honeypot4
    Type: hash:ip
    Revision: 4
    Header: family inet hashsize 4096 maxelem 1000000
    is ok !

    You can test by restarting the server (sudo reboot) the ipset is ok !

    You can create verification in php and send email to put in a cron

    $output = shell_exec(‘sudo ipset list honeypot4 > /yourfolder/outputs.txt 2>&1’);

    $ressource = fopen(‘/root/save/outputs.txt’, ‘rb’);
    $output= fread($ressource, filesize(‘/yourfolder/outputs.txt’));

    $findme = ‘The set with the given name does not exist’;
    $pos = strpos($output, $findme);
    if ($pos === false) {
    //not find
    } else {
    exec(“echo ‘Subject: !!!!ERROR: Open /etc/ipset.rules and test ipset restore</etc/ipset.rules for verification!!!!’ | sendmail -v example@example.com“);

    }
    fclose($ressource);
    exec(‘rm /yourfolder/outputs.txt’);

    🙂

    Reply
  2. Gwyneth Llewelyn

    I know I’m being paranoid, but I don’t really like the idea of having a ‘backdoor’ sudo, no matter how well it’s protected; hackers also read these pages, and they might come up with an idea to subvert this system (e.g. somehow replacing the call to block-ip.sh with something completely different).

    However, in general, all honeypot traps are extraordinarily effective (nice touch using conntrack — I thought that project was long abandoned, but it seems that it gets a release every other year or so, the latest, at the time of writing, being from April 2020)! I only wonder if there isn’t an easier way to, say, pass information directly to fail2ban (instead of waiting for fail2ban to read the logs). I’m just thinking out loud here, mind you, perhaps your approach is far better. My only issue is that I have a reasonably complex setup for dozens of virtual hosts being maintained by an even more complex set of scripts to automate the process; I have nothing against changing configurations directly, but I’m aware that some of your suggestions will require quite a lot of tweaking, because I have several different requirements — some vhosts use PHP, some have specific configurations for WordPress, some are purely static HTML-only sites, and a few are basically front-end reverse proxies to the ‘real’ applications (which are natively compiled, not interpreted or JIT-compiled). That’s a lot of use cases, and changing each configuration manually — and making sure that it’s future-proof! — is a bit beyond my abilities…

    For now, I’m sticking with a fail2ban-only configuration. My biggest nightmare are those pestering, constant GET /.env requests. My overall nginx configuration already weeds these out (sure, I understand your issue that a connection can be kept alive, thus the hacker will easily continue to use it…) and fail2ban then picks it up for blocking the IP address completely (both at the local firewall as well as on CloudFlare). I’m even considering adding such addresses to a ‘global’ blacklist — I’m still working that out in my mind before attempting to do so 🙂

    Anyway, thanks for your very interesting suggestions!

    Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: