ModSecurity Breach

Support request checklist

Introduction

This document describes the steps needed to collect the information necessary to successfully resolve a ModSecurity support request. You don't have to go through this document every time you need to ask something. For simpler problems just go ahead and ask a question. But if the information you provided is not sufficient you will be forwarded back to this document.

This is what you need to do:

  1. Read the document briefly to understand the process
  2. Go through the document item by item and collect the data
  3. Put all the information into an archive
  4. Describe your problem in an email to mod-security-users@lists.sourceforge.net, attaching the archive from the previous step.

You are about to submit a support request to a public mailing list. Before you do it you need to think about the data you have included in the report. Not only there are many people subscribed to the list (that's a good thing) but it is archived at many web sites and practically stored for ever. You should not send anything private, sensitive, or confidential.

Environment information

1. Specify the operating system

Which operating system are you running?

On Unix, the version can be provided using the uname command:

$ uname -a
Linux fedora1 2.4.22-1.2199.nptl #1 Wed Aug 4 12:25:07 EDT 2004 i686 athlon i386 GNU/Linux

On Red Hat systems the file /etc/redhat-release provides more information:

$ cat /etc/redhat-release
Fedora Core release 1 (Yarrow)

On Windows computers, system information is available from the Control Panel, option System (on the first tab). For example:

Microsoft Windows XP
Professional
Version 2002
Service Pack 2

2. Specify the Apache version

Which Apache are you running? You can get that information from the httpd binary when you execute it with the -V command:

$ ./httpd -V
Server version: Apache/2.0.52
Server built:   Jan 16 2005 10:37:17
Server's Module Magic Number: 20020903:9
Architecture:   32-bit
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D HTTPD_ROOT="/home/ivanr/apache2"
 -D SUEXEC_BIN="/home/ivanr/apache2/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

While you are there, it is also useful to list the available modules:

$ ./httpd -l
Compiled in modules:
  core.c
  mod_access.c
  mod_auth.c
  mod_include.c
  mod_log_config.c
  mod_env.c
  mod_setenvif.c
  prefork.c
  http_core.c
  mod_mime.c
  mod_status.c
  mod_autoindex.c
  mod_asis.c
  mod_cgi.c
  mod_negotiation.c
  mod_dir.c
  mod_imap.c
  mod_actions.c
  mod_userdir.c
  mod_alias.c
  mod_rewrite.c
  mod_so.c

3. Specify the ModSecurity version

You can find out the ModSecurity version from the name of the archive (the version number is part of the name, for example mod_security-1.8.6.tar.gz), or by looking in the source code (file mod_security.c):

#define MODULE_RELEASE "1.8.6"

If you find that you are using an older version of ModSecurity, please check the “Known issues” page (http://www.modsecurity.org/documentation/known-issues.html) to determine if your problem is already known. If it is, you will most likely find the fix already available (in a form of minor version upgrade). If the fix isn't available yet chances it will be very soon.

4. Brief system description

Briefly describe how is the system used. For example:

Apache is used to host a departmental intranet application. The
application was developed internally, runs in PHP, connects to
a MySQL database, and our IMAP server.

5. Information about the browser

Consider the following questions:

  • Which browser are you using? Go to Help > About and record the information displayed.
  • Do you think the problem is related to the browser you are using?
  • Does the same happen when you use some other browser?

II Configuration

6. Apache and ModSecurity configuration

Please supply copies of your Apache and ModSecurity configuration. In most cases the configuration is only contained in the httpd.conf file. Some administrators choose to separate configuration data logically into more files. If this is the case you can look for the Include configuration directive, which specifies another file to include. Include all such files in your report.

Your Apache configuration may contain sensitive information. If you think it does you should inspect the configuration files and remove sensitive information before you include the files in your report.

7. Filesystem layout

If you are experiencing filesystem problems you should include a listing of a part of the filesystem in the report. If you execute the following on a Unix system, the file fs-listing.txt will contain the list of all files in the /home/ivanr/apache2 directory.

$ ls -alR /home/ivanr/apache2 > ~/fs-listing.txt

On Windows, the same effect can be achieved with:

C:\> dir "c:\Program Files\Apache" /s > c:\fs-listing.txt

Feel free to include dumps of several directories if required.

8. Special circumstances to consider

Are there any special circumstances to consider when requesting support? For example, your Apache installation may be running from a chroot jail, where it has access only to a limited set of files.

III Logs

The final piece of information needed is contained in the log files. Since logs usually contain large amounts of information you need to take steps to only send the necessary data:

  1. Check the configuration to verify logging is enabled (see below)
  2. Locate all four sources of information (see below)
  3. Perform a request and monitor the files to verify the information is logged
  4. Rotate the logs so the files become empty
  5. Execute the test request
  6. Add the files to the problem report

If you are not getting any output from ModSecurity make sure it is configured and enabled with SecFilterEngine On.

9. Apache access log

Each request appears as a single line in the Apache access log. By default the access log is in the file /logs/access_log. When virtual hosting is used each virtual host may have a separate access log file. Examine the Apache configuration if you don't see the information about the test requests you make in the access log (look for the appearance of the TransferLog or CustomLog directives).

The following example shows traces of two requests (both have generated errors, as you shall see later in the text):

192.168.2.10 - - [16/Jan/2005:10:56:49 +0000] "GET /index.html?111 HTTP/1.1" 403 291
192.168.2.10 - - [16/Jan/2005:10:56:49 +0000] "GET /favicon.ico HTTP/1.1" 404 288

10. Apache error log

Apache errors are logged to the error log. ModSecurity will emit relevant warnings and errors to the error log too. By default the error log is written to a file located at /logs/error_log. It is entirely possible for different virtual hosts to have different error log files. To determine if the errors will appear in the file you are monitoring just perform a test request for a resource that does not exist on the server. A line warning about a page not found should appear in the log:

[Sun Jan 16 10:56:49 2005] [error] [client 192.168.2.10] File does
not exist: /home/ivanr/apache2/htdocs/favicon.ico

Otherwise examine the Apache configuration to find the correct error log location (look for the ErrorLog directive).

This is what a single ModSecurity error message looks like:

[Sun Jan 16 10:56:49 2005] [error] [client 192.168.2.10]
mod_security: Access denied with code 403. Pattern match "111" at
THE_REQUEST [hostname "192.168.2.101"] [uri "/index.html?111"]

11. ModSecurity Debug log

The ModSecurity debug log will, when configured properly, provide a ton of information for every HTTP request. To get the information, specify the file where you want the debug output to go and set the log level to 9:

SecFilterDebugLevel 9
SecFilterDebugLog logs/modsec_debug_log

The output will be in the file /logs/modsec_debug_log.

There can be pages of text for each request, especially if a file is being uploaded.

This is how a very short output looks like (one request, one ModSecurity rule only):

[16/Jan/2005:10:56:49 +0000] [192.168.2.101/sid#8f519d8][rid#9008830][/index.html]
sec_check_access, path=(null)
[16/Jan/2005:10:56:49 +0000] [192.168.2.101/sid#8f519d8][rid#9008830][/index.html]
Normalised REQUEST_URI: "/index.html?111"
[16/Jan/2005:10:56:49 +0000] [192.168.2.101/sid#8f519d8][rid#9008830][/index.html]
Parsing arguments...
[16/Jan/2005:10:56:49 +0000] [192.168.2.101/sid#8f519d8][rid#9008830][/index.html]
Adding parameter: [111][]
[16/Jan/2005:10:56:49 +0000] [192.168.2.101/sid#8f519d8][rid#9008830][/index.html]
Checking signature "111" at THE_REQUEST
[16/Jan/2005:10:56:49 +0000] [192.168.2.101/sid#8f519d8][rid#9008830][/index.html]
Checking against "/index.html?111"
[16/Jan/2005:10:56:49 +0000] [192.168.2.101/sid#8f519d8][rid#9008830][/index.html]
Signature check returned 403
[16/Jan/2005:10:56:49 +0000] [192.168.2.101/sid#8f519d8][rid#9008830][/index.html]
Access denied with code 403. Pattern match "111" at THE_REQUEST
[16/Jan/2005:10:56:49 +0000] [192.168.2.101/sid#8f519d8][rid#9008830][/index.html]
Rule match, returning code 403
[16/Jan/2005:10:56:49 +0000] [192.168.2.101/sid#8f519d8][rid#9008830][/index.html]
find_last_request: start with 9008830 "/index.html"
[16/Jan/2005:10:56:49 +0000] [192.168.2.101/sid#8f519d8][rid#9008830][/index.html]
sec_logger: start
[16/Jan/2005:10:56:49 +0000] [192.168.2.101/sid#8f519d8][rid#9008830][/index.html]
Audit log off here

12. ModSecurity Audit log

The ModSecurity audit log will, when enabled, contain the complete request and the headers from the response. To make sure request bodies are observed configure:

SecFilterScanPOST On

The following configuration will tell ModSecurity to log only violations:

SecAuditEngine RelevantOnly
SecAuditLog logs/audit_log

The output will be in the file /logs/audit_log. The example below shouls one audit log entry:

========================================
Request: 192.168.2.10 - - [17/Jan/2005:17:56:30 +0000] "GET /index.html?111 HTTP/1.1"
403 291
Handler: (null)
----------------------------------------
GET /index.html?111 HTTP/1.1
Host: 192.168.2.101:8080
User-Agent: mod_security regression test utility
Accept: text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
mod_security-message: Access denied with code 403. Pattern match "111" at THE_REQUEST
mod_security-action: 403

HTTP/1.1 403 Forbidden
Content-Length: 291
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1

Notes on working on a production server

Providing logs from a production server, possibly one used by dozens of people at any given time, can be challenging. For example, switching the debug log to level 9 will instantly slow down the web server. It is still possible to provide detailed logs by increasing the debug log level only for the parts where it is absolutely required.

For example, let us assume you only need to test one script, for example /test.php. To increase the debug log level for that script only do the following (assuming you have previously configured the file name for the debug log and set the default log level to zero):

<Location /test.php>
    SecFilterDebugLogLevel 9
</Location>    

Almost all ModSecurity configuration directives can be put inside Apache context tags. If you need to perform a test but you are afraid your ModSecurity configuration can affect your users you can even enable ModSecurity for one file only. Just put all the directives in a container tag.