NGINX Bot Protection Without CAPTCHA: Testcookie Guide

Bots account for nearly half of all internet traffic today. While some bots are helpful, like search engine crawlers, many are malicious. These automated scripts scrape content, stuff credentials, and scan for vulnerabilities. Traditional CAPTCHAs frustrate legitimate users and hurt conversion rates. The testcookie NGINX module offers a better approach to bot protection. It uses invisible JavaScript challenges that block bots while remaining transparent to real visitors.

This guide shows you how to install and configure the testcookie module on RHEL, Rocky Linux, and AlmaLinux. You will learn how the challenge mechanism works internally. Additionally, you will discover advanced configurations for production environments.

Why NGINX Bot Protection Matters

Malicious bots pose several threats to your web infrastructure. Scrapers steal your content and intellectual property. Credential stuffers attempt to breach user accounts. Vulnerability scanners probe for weaknesses in your applications. All these activities consume server resources and bandwidth.

Consider the impact on your servers:

• CPU load increases from processing bot requests
• Database stress grows from automated queries
• Bandwidth costs rise from content scraping
• Security risks emerge from vulnerability scanning
• Analytics pollution skews your traffic data

Traditional CAPTCHA solutions create friction for legitimate users. They interrupt the user experience and reduce conversion rates. Accessibility concerns affect users with disabilities. Moreover, sophisticated bots can now solve many CAPTCHA types automatically.

JavaScript-based challenges offer a better solution. They verify browser capabilities without user interaction. Legitimate browsers execute the challenge automatically. Simple bots fail because they cannot process JavaScript. This approach provides NGINX bot protection that remains invisible to real visitors.

How the Testcookie Challenge Works

The testcookie module implements a cookie-based challenge-response mechanism. It operates during NGINX’s access phase, which runs before content processing. This early interception prevents bots from consuming backend resources.

The MD5 Challenge Mechanism

When a client first connects, the module generates a unique challenge value. This value combines a secret key with session data using MD5 hashing:

Challenge = MD5(secret + session_identifier)

The session identifier typically uses the client’s IP address. However, you can combine it with the User-Agent header for stricter validation. The secret key must be at least 32 characters long for security.

The Redirect Loop

The challenge process follows these steps:

1. Client requests a protected page without a valid cookie
2. Server responds with a 307 redirect and sets a challenge cookie
3. Browser automatically follows the redirect with the cookie
4. Server validates the cookie against the expected value
5. If valid, access is granted; if not, the process repeats

This loop continues until the maximum attempt limit is reached. At that point, the client redirects to a fallback URL. Legitimate browsers pass the challenge on the first redirect because they handle cookies automatically.

Why Bots Fail

Simple bots fail this challenge for several reasons. Many HTTP clients do not follow redirects automatically. Others do not store or return cookies. The testcookie module leverages these limitations to filter automated traffic. Additionally, the optional JavaScript template mode requires actual JS execution, which most bots cannot perform.

Install Testcookie on RHEL, Rocky Linux, and AlmaLinux

Installing the testcookie module requires the GetPageSpeed repository. This repository provides pre-built NGINX modules for Enterprise Linux distributions.

Prerequisites

Before installation, ensure you have:

• Root or sudo access to your server
• RHEL, CentOS, Rocky Linux, or AlmaLinux (versions 7, 8, 9, or 10)
• A working internet connection

Installation Steps

First, install the GetPageSpeed repository:

sudo dnf -y install https://extras.getpagespeed.com/release-latest.rpm

For CentOS 7 or RHEL 7, use yum instead:

sudo yum -y install https://extras.getpagespeed.com/release-latest.rpm

Next, install NGINX and the testcookie module:

sudo dnf -y install nginx nginx-module-testcookie

Finally, enable the module by adding this line at the top of /etc/nginx/nginx.conf:

load_module modules/ngx_http_testcookie_access_module.so;

Verify the configuration and restart NGINX:

sudo nginx -t
sudo systemctl restart nginx

The module is now ready for configuration. For complete directive reference, see the testcookie documentation.

Basic Testcookie Configuration

A minimal configuration requires three essential directives. The following example shows a production-ready setup that passes gixy security checks:

load_module modules/ngx_http_testcookie_access_module.so;

user nginx;
worker_processes auto;
worker_rlimit_nofile 4096;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    server_tokens off;

    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" testcookie_ok=$testcookie_ok';

    access_log /var/log/nginx/access.log main;

    server {
        listen 80;
        server_name example.com;
        root /var/www/html;

        # Testcookie bot protection
        testcookie on;
        testcookie_name BPC;
        testcookie_secret "YourVeryLongSecretKeyAtLeast32CharactersForSecurity";
        testcookie_session $remote_addr;
        testcookie_arg attempt;
        testcookie_max_attempts 3;
        testcookie_fallback /blocked.html;

        location / {
            index index.html;
        }

        location = /blocked.html {
            internal;
            return 403 "Access denied - please enable JavaScript and cookies";
        }
    }
}

Key Directives Explained

The testcookie_session variable determines how sessions are identified. Using $remote_addr alone works for most cases. For stricter validation, combine it with the User-Agent: $remote_addr$http_user_agent. This prevents cookie sharing between different browsers on the same IP.

Advanced Testcookie Configuration

Production environments often require additional features. This section covers encryption, whitelisting, and custom templates.

AES Encryption for Enhanced Security

By default, the challenge value is predictable if an attacker knows your secret. AES encryption adds an additional layer of security. The encrypted value must be decrypted client-side using JavaScript:

testcookie_refresh_encrypt_cookie on;
testcookie_refresh_encrypt_cookie_key random;
testcookie_refresh_encrypt_cookie_iv random;
testcookie_redirect_via_refresh on;
testcookie_refresh_template /etc/nginx/templates/challenge.html;

Create the challenge template at /etc/nginx/templates/challenge.html:

<!DOCTYPE html>
<html>
<head>
    <title>Verifying your browser...</title>
    <script src="/aes.min.js"></script>
</head>
<body>
    <p>Please wait while we verify your browser...</p>
    <script>
        function toNumbers(d) {
            var e = [];
            d.replace(/(..)/g, function(d) {
                e.push(parseInt(d, 16));
            });
            return e;
        }
        function toHex() {
            for (var d = [], d = 1 == arguments.length && arguments[0].constructor == Array ? arguments[0] : arguments, e = "", f = 0; f < d.length; f++)
                e += (16 > d[f] ? "0" : "") + d[f].toString(16);
            return e.toLowerCase();
        }
        var a = toNumbers("$testcookie_enc_key");
        var b = toNumbers("$testcookie_enc_iv");
        var c = toNumbers("$testcookie_enc_set");
        document.cookie = "BPC=" + toHex(slowAES.decrypt(c, 2, a, b)) + "; path=/";
        location.href = "$testcookie_nexturl";
    </script>
    <noscript>
        <p>JavaScript is required to access this site.</p>
    </noscript>
</body>
</html>

You will need the slowAES JavaScript library served from your web root. This encryption method requires bots to execute JavaScript, which significantly increases protection.

Whitelisting Trusted Networks

Search engine crawlers cannot execute JavaScript challenges. Therefore, you must whitelist their IP ranges to maintain SEO visibility. The module supports IPv4 and IPv6 CIDR notation:

# Google crawlers
testcookie_whitelist 66.249.64.0/19;
testcookie_whitelist 64.233.160.0/19;

# Bing crawlers
testcookie_whitelist 207.46.0.0/16;
testcookie_whitelist 40.77.167.0/24;

# Internal networks
testcookie_whitelist 10.0.0.0/8;
testcookie_whitelist 192.168.0.0/16;

For verified bot protection, consider the bot-verifier module. It validates that requests actually originate from claimed search engines through reverse DNS lookups.

Cookie Security Attributes

Modern browsers support several cookie security attributes. Configure them for better protection:

testcookie_httponly_flag on;    # Prevents JavaScript access to cookie
testcookie_secure_flag on;      # Requires HTTPS
testcookie_samesite Lax;        # Restricts cross-site requests
testcookie_expires 1d;          # Cookie lifetime (default: far future)
testcookie_path /;              # Cookie scope

Conditional Bypass

Sometimes you need to bypass the challenge for specific conditions. Use nginx variables with the testcookie_pass directive:

# Bypass for requests with valid API key
map $http_x_api_key $bypass_testcookie {
    default 0;
    "your-api-key-here" 1;
}

server {
    testcookie_pass $bypass_testcookie;
    # ... other configuration
}

You can also use testcookie_get_only on; to apply challenges only to GET and HEAD requests. This allows POST requests to pass through without challenge.

Testing Your Bot Protection

After configuration, verify that the module works correctly. Use these methods to test your setup.

Testing with curl

First, test a request without cookies:

curl -I http://example.com/

You should see a 307 redirect with a Set-Cookie header:

HTTP/1.1 307 Temporary Redirect
Set-Cookie: BPC=0d081cd38ab792e2cb10e03867345bb4; path=/; SameSite=None; Secure
Location: http://example.com/?attempt=1

Next, test with cookie handling enabled:

curl -I -c cookies.txt -b cookies.txt -L http://example.com/

This should show a 200 response after the redirect.

Checking the Logs

Add $testcookie_ok to your log format to track challenge results:

log_format main '$remote_addr - [$time_local] "$request" '
                '$status testcookie_ok=$testcookie_ok';

The variable shows:
– 1 when the challenge passed
– 0 when the challenge failed

Browser Testing

Open your browser’s developer tools and navigate to a protected page. In the Network tab, you should see:

1. Initial request returns 307 redirect
2. Cookie is set in response headers
3. Redirected request returns 200 OK

The process should be invisible to users because browsers handle redirects automatically.

Troubleshooting Common Issues

Several issues can arise during testcookie deployment. Here are solutions for the most common problems.

Infinite Redirect Loop

Symptom: Browsers show “too many redirects” error.

Cause: Missing testcookie_arg directive allows infinite attempts.

Solution: Add the attempt tracking parameter:

testcookie_arg attempt;
testcookie_max_attempts 3;

Legitimate Users Blocked

Symptom: Real users cannot access the site.

Cause: JavaScript disabled or cookies blocked in browser.

Solution: Provide a helpful fallback page that explains the requirements:

testcookie_fallback /enable-js.html;

location = /enable-js.html {
    internal;
    default_type text/html;
    return 200 '<html><body><h1>JavaScript Required</h1><p>Please enable JavaScript and cookies to access this site.</p></body></html>';
}

Search Engines Cannot Crawl

Symptom: Google Search Console shows crawl errors.

Cause: Search bots cannot execute JavaScript challenges.

Solution: Whitelist search engine IP ranges or use the bot-verifier module for verified crawlers. Consider combining testcookie with an NGINX honeypot for comprehensive protection.

Cookie Not Persisting

Symptom: Challenge passes but users get challenged again on next request.

Cause: Domain mismatch or secure flag issues.

Solution: Check your cookie settings:

testcookie_domain .example.com;    # Include subdomains
testcookie_secure_flag off;        # If not using HTTPS
testcookie_path /;                 # Ensure site-wide scope

Module Not Loading

Symptom: NGINX fails to start with “unknown directive” error.

Cause: Missing load_module directive.

Solution: Add to the top of nginx.conf, before the events block:

load_module modules/ngx_http_testcookie_access_module.so;

Verify the module file exists:

ls -la /usr/lib64/nginx/modules/ngx_http_testcookie_access_module.so

Performance Considerations

The testcookie module adds minimal overhead to request processing. Understanding its performance characteristics helps optimize your configuration.

Request Overhead

First-time visitors experience one additional redirect. This adds approximately 50-100ms of latency depending on network conditions. Returning visitors with valid cookies bypass the challenge entirely. The MD5 calculation for challenge generation is extremely fast and adds negligible CPU load.

Optimization Tips

Apply these optimizations for busy sites:

# Close connections after challenge to disrupt bot connections
testcookie_deny_keepalive on;

# Only challenge GET requests (POST passes through)
testcookie_get_only on;

# Set reasonable cookie lifetime
testcookie_expires 1d;

Comparison with Alternatives

The testcookie module offers several advantages over other approaches:

For comprehensive protection, combine testcookie with rate limiting and the NGINX honeypot approach.

Key Takeaways

The testcookie module provides effective NGINX bot protection without frustrating legitimate users. Here are the essential points to remember:

• Invisible protection: Legitimate browsers pass the challenge automatically
• Easy installation: Available from GetPageSpeed repository for all Enterprise Linux distributions
• Flexible configuration: Supports encryption, whitelisting, and custom templates
• Low overhead: Minimal performance impact compared to CAPTCHAs or WAFs
• Whitelist search engines: Use IP whitelists or the bot-verifier module to maintain SEO

For complete protection, consider combining testcookie with:
– Bot-verifier module for verified crawler access
– NGINX honeypot for additional bot trapping
– Rate limiting for DDoS mitigation

Validate your NGINX configuration with gixy to catch security issues before deployment. The testcookie module gives you modern bot protection that respects user experience while effectively filtering automated traffic.