Understanding and Analyzing Apache Log Files

If you’re responsible for keeping web servers running, you already know easy access to log messages is critical when troubleshooting issues. Apache provides comprehensive support for logging, and its highly customizable configuration allows you to tailor its logging to your exact needs. You can gain visibility into your web servers by logging everything from the initial request through to the URL mapping process and connection termination. And if this wasn’t enough, third-party modules provide additional logging capabilities such as support for application runtimes including PHP, Java, and CGI programs.

In this article, I’ll show you how to use the two different logging mechanisms in Apache and how to find the data you need from your logs when investigating problems.

The ErrorLog Directive

Errors and diagnostic warnings on web servers usually mean users can’t access the pages and resources they need, and the first place any admin will look is in the Apache server error log. So, the ErrorLog is the single most important Apache log file for troubleshooting. It contains the information you’ll need to understand why errors are occurring and which users are affected.

The location of the error log is controlled with the ErrorLog directive. By default, you can find it in a file (error.log) but it’s also possible to send the error log to a syslog daemon or pipe it to a program using Apache support for piped logs.

The ErrorLogFormat directive enables you to craft the way server error messages look and the data they contain in addition to the log message emitted by the Apache internal code. Apache sends a range of different messages to error.log, so it’s sometimes difficult to tell them apart and find the one you’re looking for. By taking advantage of the ErrorLogFormat directive, you’ll be able to find the messages you need faster.

Of course, you don’t have to specify your own format, since a default format is provided by Apache, but by using ErrorLogFormat, you can create more relevant and actionable log data.

For example, the following ErrorLogFormat directive displays a timestamp, the log message severity, the process ID of the web server, the source file and log where the log was generated, and finally the message itself:

ErrorLogFormat "[%t] [%l] [pid: %P] %F %M"

The format produces error messages that look like this:

[Thu Jan 23 11:36:12 2020] [notice] [pid: 1793] event.c(2974) AH00489: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Thu Jan 23 11:36:12 2020] [notice] [pid: 1793] log.c(1561) AH00094: Command line: '/usr/sbin/apache2'

Thr Apache ErrorLogFormat directive supports over 20 different format strings, which work just like C-style printf() format strings. This goes to show how flexible Apache logging capabilities are—Apache provides the ability to customize where server errors messages are sent and allows you to control their format, so you can include all the information you need.

The CustomLog Directive

Every time a user visits your site, Apache stores information about which paths they accessed in access.log. This gives you a useful way to monitor things like the most popular sections of your site, enables you to see the addresses from which users are connecting, and shows which HTTP status codes were returned by your server for each request. Connection addresses and HTTP status codes can be valuable clues when later troubleshooting and investigating issues.

Just like the error log, you can configure the format of the access log using format strings with the LogFormat directive. Each LogFormat option requires a unique identifier, so you can create more than one log format to store log messages in different ways.

For example, here’s an example of two different log formats each writing to separate files. The first format follows the Common Log Format (CLF is supported by many applications, so it’s often used as the logging format for web servers), and the second format displays just the referer and the URL request:

LogFormat "%h %l %u %t \"%r\" %>s %O" common
CustomLog ${APACHE_LOG_DIR}/access.log common

LogFormat "%{Referer}i -> %U" referer
CustomLog ${APACHE_LOG_DIR}/referer.log referer

Entries for these formats looks like this:

access.log

127.0.0.1 - - [23/Jan/2020:16:41:01 +0000] "GET / HTTP/1.1" 200 11229

referer.log

- -> /index.html

By combining a LogFormat directive with CustomLog, you can direct log messages to different files.

Not only can you write access log messages to multiple log files, you can also make writing log messages conditional on some attribute of the request. The SetEnvIf directive allows you to set environment variables if some condition is true. Then you can use the env= clause in your CustomLog directives to enable logging only if that environment variable is set.

Here’s an example logging all requests from localhost to a dedicated file named local.log:

SetEnvIf  Remote_Addr "127\.0\.0\.1" localaddr
CustomLog ${APACHE_LOG_DIR}/local.log common env=localaddr

This rule takes advantage of Apache Remote_Addr variable, which records the IP address of the client making the request. The complete list of Apache variables is available in the Apache documentation. If you’re using SolarWinds^® Papertrail^™ search, you can use those variables to quickly troubleshoot issues.

Analyzing Error and Access Logs

Once you’ve written log messages to your log files you need a way to extract them. Of course, you can send them directly to Papertrail where you can tail, search, and analyze live and stored logs. If you aren’t sending them to Papertrail, you can use standard file tools such as cat, less, and tail. If you’re using the tail command, it prints the last 10 lines of a file by default, but you can control exactly how many lines are displayed with the -n parameter. To read the last 25 lines from access.log, you need to do:

tail -n25 access.log

The Papertrail live tail feature lets you tail your logs and continuously see new events as they happen. You can pause, search, and scroll through the log history and even click on elements to jump to related log entries, such as all the ones with a specific IP address.

If your web server receives a lot of traffic, it’s a challenge to manually search through your logs because of their size. Instead, you need to turn to the Papertrail solution’s search syntax, which allows you to create powerful expressions matching the strings you’re trying to find. Here’s an example search string to find all messages containing 404 HTTP status codes:

HTTP 404

If you’re working with log files from multiple sources such as PHP, Apache, or JSON, it might be worthwhile to look at a log management tool like Papertrail.

Conclusion

Effectively managing web servers means you need to be able to understand their behavior and performance. When users start reporting pages not loading or connections timing out, you need a way to look under the hood of your web servers to see what’s happening. Actionable log messages are the key to keeping your site up and running.

Apache provides a heap of options for storing log messages to files. With the ErrorLog and CustomLog directives, you can customize where messages are written to and their format. Splitting log messages into multiple files is possible with more than one directive, and with Apache support for environment variables, you can take advantage of conditional statements to choose exactly when log messages should be written.