Install ModSecurity NGINX: Complete WAF Guide

ModSecurity NGINX is the most powerful combination for protecting your web applications from malicious attacks. When you install this WAF module, you gain enterprise-grade Web Application Firewall capabilities. The module actively monitors, logs, and blocks malicious HTTP traffic before it reaches your application.

This comprehensive guide walks you through installing and configuring the security module on CentOS, RHEL, Rocky Linux, AlmaLinux, Fedora, Debian, and Ubuntu systems. You’ll use packages from the GetPageSpeed repository for easy installation. By the end, you’ll have a fully functional WAF protecting against SQL injection, XSS, and other common web attacks.

Why Choose ModSecurity NGINX?

This security module combination offers several compelling advantages for protecting your web server:

• Protection against OWASP Top 10 vulnerabilities including SQL injection (SQLi), cross-site scripting (XSS), and remote code execution (RCE)
• Real-time traffic monitoring with detailed logging and audit trails for forensic analysis
• Flexible rule engine supporting custom rules and the OWASP Core Rule Set (CRS)
• Low performance overhead thanks to the efficient libmodsecurity v3 library
• Per-location customization for fine-tuning security rules per endpoint
• SELinux compatibility built into GetPageSpeed packages

The connector module bridges your web server and libmodsecurity. It requires no Apache internals. This architecture results in fewer dependencies, improved performance, and reduced bugs compared to older ModSecurity 2.x implementations.

If you’re looking to protect your server from common attacks, adding this WAF is essential. You can also combine it with JWT authentication for layered security.

How the WAF Module Works

The security module operates as a dynamic component. It intercepts HTTP requests and responses at multiple phases:

1. Request Headers Phase – Analyzes incoming request headers, method, and URL
2. Request Body Phase – Inspects POST data, file uploads, and request payloads
3. Response Headers Phase – Monitors outgoing response headers
4. Response Body Phase – Scans response content for data leakage
5. Logging Phase – Records transaction details for audit purposes

When the WAF detects a rule violation, it can:

• Block the request with HTTP 403 Forbidden
• Log the event without blocking
• Redirect to a custom error page
• Drop the connection entirely

Prerequisites

Before installing the security module, ensure you have:

• NGINX from the GetPageSpeed repository (version 1.28.x or later)
• Root or sudo access to your server
• An active GetPageSpeed subscription (for RHEL-based distributions)

For optimal results, ensure your server has proper NGINX permissions configured.

Installation on RHEL-Based Systems

CentOS, RHEL, AlmaLinux, Rocky Linux

First, enable the GetPageSpeed repository:

sudo dnf install https://extras.getpagespeed.com/release-latest.rpm

Then install the ModSecurity NGINX module and OWASP Core Rule Set:

sudo dnf install nginx-module-security nginx-owasp-crs

This command installs these packages:

• nginx-module-security – The connector module (v1.0.4)
• libmodsecurity – The ModSecurity v3 library (v3.0.14)
• nginx-owasp-crs – OWASP Core Rule Set (v4.23.0)

Load the module by adding this line at the top of /etc/nginx/nginx.conf:

load_module modules/ngx_http_modsecurity_module.so;

Configuration

Basic Setup

The WAF requires configuration at two levels:

1. NGINX directives – Control where the module operates
2. SecRules – Define traffic patterns to detect

The GetPageSpeed packages install these pre-configured files:

Enabling the WAF in Your Server Block

Add these directives to your server block:

server {
    listen 80;
    server_name example.com;

    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec_includes.conf;

    location / {
        root /var/www/html;
        index index.html;
    }
}

Test and reload:

nginx -t && systemctl reload nginx

Directive Reference

The module provides five configuration directives:

modsecurity

Syntax: modsecurity on | off;
Default: off
Context: http, server, location

Enables or disables WAF processing:

server {
    modsecurity on;

    location /health {
        modsecurity off;
    }
}

modsecurity_rules_file

Syntax: modsecurity_rules_file <path>;
Default: none
Context: http, server, location

Specifies the rules configuration file:

modsecurity_rules_file /etc/nginx/modsec_includes.conf;

modsecurity_rules_remote

Syntax: modsecurity_rules_remote <key> <url>;
Default: none
Context: http, server, location

Downloads rules from a remote server:

modsecurity_rules_remote my-key https://rules.example.com/download;

modsecurity_rules

Syntax: modsecurity_rules '<rules>';
Default: none
Context: http, server, location

Embeds rules directly in the configuration:

location /api {
    modsecurity_rules '
        SecRuleEngine On
        SecRule ARGS "@contains admin" "id:1001,deny,status:403"
    ';
}

modsecurity_transaction_id

Syntax: modsecurity_transaction_id <string>;
Default: auto-generated
Context: http, server, location

Sets a custom transaction ID for log correlation:

log_format extended '$remote_addr [$time_local] "$request" $status $request_id';

server {
    modsecurity on;
    modsecurity_transaction_id "api-$request_id";
    access_log /var/log/nginx/access.log extended;
}

Testing Your Setup

Verify that your ModSecurity NGINX WAF blocks malicious requests properly.

Test XSS Blocking

curl -I "http://localhost/?x=<script>alert(1)</script>"

Expected response:

HTTP/1.1 403 Forbidden
Server: nginx/1.28.2

Test SQL Injection Blocking

curl -I "http://localhost/?id=1%20OR%201=1"

Expected response:

HTTP/1.1 403 Forbidden

Verify Module Loading

Check your error log:

grep ModSecurity /var/log/nginx/error.log

You should see:

ModSecurity-nginx v1.0.4 (rules loaded inline/local/remote: 0/717/0)
libmodsecurity3 version 3.0.14

Understanding Audit Logs

The WAF maintains detailed audit logs. GetPageSpeed packages configure concurrent logging with separate files per transaction.

Audit Log Locations

• Main index: /var/log/nginx/modsec_audit.log
• Transaction files: /var/log/nginx/modsec/YYYYMMDD/YYYYMMDD-HHMM/

Viewing Blocked Requests

Find recent blocks:

grep "Access denied" /var/log/nginx/error.log | tail -10

Examine concurrent audit logs:

find /var/log/nginx/modsec/ -type f -mmin -5 | xargs cat

For advanced log analysis, consider the NGINX error_log_write module for conditional logging.

OWASP Core Rule Set Configuration

OWASP CRS provides comprehensive attack protection. Version 4.23.0 includes rules for:

• SQL injection (942xxx rules)
• Cross-site scripting (941xxx rules)
• Remote code execution (932xxx rules)
• Local file inclusion (930xxx rules)
• PHP injection attacks (933xxx rules)
• Session fixation (943xxx rules)

CRS v4 uses anomaly scoring. Each violation adds points. Requests exceeding the threshold get blocked.

Customizing CRS Settings

Copy and edit the CRS setup file:

sudo cp /etc/nginx/owasp-modsecurity-crs/crs-setup.conf /etc/nginx/owasp-modsecurity-crs/crs-setup.conf.local

Common customizations include adjusting the anomaly threshold and paranoia level.

Handling False Positives

False positives occur when legitimate requests trigger rules. Check your NGINX error log to identify these issues.

Excluding Rules Per Location

Use modsecurity_rules to disable specific rules:

location /api/webhook {
    modsecurity_rules '
        SecRuleRemoveById 942100
        SecRuleRemoveById 941100
    ';
}

Rule Exclusion Best Practices

1. Check audit logs to identify the triggering rule ID
2. Test the exclusion in a staging environment first
3. Document all rule exclusions for security audits
4. Review exclusions periodically

Performance Optimization

The WAF adds processing overhead. For optimal performance with proxy configurations, follow these recommendations.

Selective WAF Protection

Apply the module only where needed:

server {
    location / {
        modsecurity on;
        modsecurity_rules_file /etc/nginx/modsec_includes.conf;
    }

    location /static/ {
        modsecurity off;
    }

    location /health {
        modsecurity off;
        return 200 "OK";
    }
}

PCRE Limits for ReDoS Prevention

Set limits in /etc/nginx/modsecurity.conf:

SecPcreMatchLimit 10000
SecPcreMatchLimitRecursion 10000

Request Body Limits

Configure appropriate limits:

SecRequestBodyLimit 13107200
SecRequestBodyNoFilesLimit 131072

Troubleshooting Common Issues

WAF Not Blocking Attacks

1. Verify module loaded: nginx -T 2>&1 | grep modsecurity
2. Check modsecurity on; in your server block
3. Verify SecRuleEngine On in modsecurity.conf

Audit Log Not Writing

Fix directory permissions:

sudo chown -R nginx:nginx /var/log/nginx/modsec
sudo chmod 770 /var/log/nginx/modsec

High Memory Usage

Reduce request body limits and disable response body scanning if not needed:

SecResponseBodyAccess Off

Monitoring and Maintenance

Keeping OWASP CRS Updated

Update rules regularly:

# RHEL-based
sudo dnf update nginx-owasp-crs

# Debian/Ubuntu
sudo apt-get update && sudo apt-get upgrade nginx-owasp-crs

Log Analysis Script

Create a monitoring script:

#!/bin/bash
BLOCKED=$(grep -c "Access denied" /var/log/nginx/error.log)
echo "Blocked requests: $BLOCKED"
grep "Access denied" /var/log/nginx/error.log | \
    grep -oP 'id "\K[0-9]+' | sort | uniq -c | sort -rn | head -5

Security Best Practices

When running the WAF in production:

1. Start in detection mode – Use SecRuleEngine DetectionOnly initially
2. Monitor audit logs – Review blocked requests daily
3. Tune gradually – Add rule exclusions only when necessary
4. Keep rules updated – Update OWASP CRS monthly
5. Use SELinux – The GetPageSpeed packages include SELinux policies
6. Combine with rate limiting – Use rate limiting for DDoS protection
7. Enable HTTPS – Always use SSL/TLS certificates with your WAF

Comparison with Other WAF Solutions

ModSecurity NGINX offers advantages over cloud-based WAF solutions:

For high-traffic sites, you might also consider Varnish caching in front of your ModSecurity NGINX setup.

Additional Resources

• ModSecurity v3 GitHub
• ModSecurity-nginx Connector
• OWASP CRS Documentation
• nginx-module-security Package

Conclusion

ModSecurity NGINX provides enterprise-grade WAF capabilities. It protects against SQL injection, XSS, and OWASP Top 10 vulnerabilities. The GetPageSpeed repository simplifies installation with pre-configured packages. You get the OWASP Core Rule Set, SELinux policies, and concurrent audit logging included.

By following this guide, you now have a fully configured ModSecurity NGINX WAF protecting your web applications. Regular monitoring and rule updates keep your applications protected against evolving threats.