Best Tips for Monitoring and Filtering Your Web Server Logs

Last updated: August 2024

Server logs are essential and can never be left out of web development. There’s no such thing as a perfect website—even one owned by a big tech company is likely to have errors in production. Using web server logs, you can easily know where the problem is coming from and solve it on time. Hence, they are quite important when monitoring and filtering your web server.

In this post, I’m going to talk about what web server logs are, why you need to log your website page, how and the various types of logs. Moreover, I will also provide practical tips on monitoring and filtering web server logs using not just the Tail and egrep Commands but also the NGINX Server and get the most value out of them.

What are Web Server Logs?

Put simply, a web server log is a log file that shows information about every request made on a web server.

Logs are automatically created by the server and consist of files containing information such as errors, requests made to the server, and other information worth looking at. Even though these logs can be created automatically by the server, you can also create custom log management functionalities. These can be done within the web app. Today, almost all programming languages have log management modules or at least a framework. With good configurations, you can utilize the language logging system and get the same log messages from the server.

Logs contain different types of files, such as images. In most cases, you’ll only want to see a certain type of information from the server.

Types of Web Server Logs

The two major types of logs that are commonly used in a web server environment are:

Access Logs

Access logs capture information about the traffic requests coming into the web server. They typically include details such as the client IP address, requested URL, HTTP method, status code, bytes transferred, and timestamp of the request. Additional fields, such as the user agent string (browser type), cookies, transfer sizes, and referrer, can also be captured if configured. These logs are helpful for analyzing website traffic patterns, identifying popular pages, monitoring server performance, and detecting potential security threats.

The limitations of access logs however include difficulty distinguishing between human users and bots. There is also the challenge associated with proxy servers that masks the true origin of requests.

Error Logs

Error logs, on the other hand, capture information about issues, bottlenecks, or errors encountered by the web server when processing requests. These logs contain details about missing files, broken links, hung services, or diagnostic information about the server itself. Error logs are important for troubleshooting web server issues and diagnosing problems that may impact website functionality.
With error logs, security professionals can also identify potential attack attempts targeting vulnerable web services. These logs are often found in files named error.log (on Windows) or error_log (on Unix).

Why Do You Need Logging on Your Website?

Before we look at how to monitor your logs, let’s talk about why you should. Money comes from speed, and a website with good speed and without lag is what every user wants. You’ll need logs from the server to know if everything is working fine on your website. Sometimes, you don’t even notice your website is throwing errors until you have access to the log files. It’s not easy to know if the database is complaining about some bad queries you’ve written. You only know when you’re able to analyze the logs from the server. If you make use of web server logs, you’ll be able to serve your clients (users) better and generate more revenue.

How Can I Monitor My Web Server Logs?

Knowing how to monitor your web server logs is important. To monitor web server logs, you’ll need to use some tools to aid you, and these tools will depend on the type of server you’re using. Most servers today run on the Unix/Linux operating system (OS), and most common tools out there are Linux tools.

Before you can even start monitoring your website, you need to know what server your website is hosted on. There several servers out there, and the tools you’re going to use are mostly similar. Once you have an idea of what your web server is, you’ll be able to know when the logs are sitting on the server. There are four types of web servers: Apache, NGINX, LiteSpeed, and IIS.

Apache is the most popular web server. Most web servers use Apache because it supports almost all the operating systems we use today, including Mac OS X, Windows, Linux, Unix, and more. It’s easy to monitor web server logs on an apache server, but the first thing to know is where the logs are located. You can find logs on the Apache server by navigating to these directories on a Linux/Unix OS:

• /var/log/apache/access.log
• /var/log/apache2/access.log
• /etc/httpd/log/access_log (on MacOS X)
• /var/log/apache2/error.log

Using the Tail Command to View Log Messages

To monitor error logs from the Apache server, you can use the Linux tail command to view all the errors as they occur in real time. Here’s an example of how you can log the errors on the console using the tail Linux command.

-$ sudo tail -f /var/log/apache2/error.log

When you run the above command, you’ll be able to view all the errors in the terminal console. You can now go ahead and use the error message to troubleshoot the problem you’re having. The -f flag in the command helps make sure all the log messages are echoed to the console. Seeing errors as they happen in real time can help you solve the problem before it affects the user experience, saving you money. If a revenue-generating website is down for hours or even just minutes, you’ll lose revenue, so being able to fix errors in real time is crucial.

Tail and egrep Commands

When you combine these two commands, the power is now in your hands. With these two commands, you can specify the types of files you don’t want to see displayed in the console. When these commands are combined to filter a log file, you get rid of the unwanted files, allowing you to experience greater speed as you’re processing the files you need. Here’s an example using the tail and egrep commands to filter unwanted files:

tail -f /var/log/apache2/error.log | egrep -v "(.gif|.jpg|.png|.swf|.ico)"

Using the grep Command to View Log Messages

Global regular expression print, or grep, is a Linux command you can use to read a file and echo the output on the console or write the data in another file. The grep command can be used to filter data in the file. If you want to display only a specific IP address, here is how you can use the grep command to filter that particular IP address (e.g., when searching for an IP address like 127.0.0.1):

$ grep "^127\.0\.0\.1" /var/log/apache2/access.log

In the same way, you can search for a specific keyword in any file via error.log. You just have to pass the keyword as the first option before the file you’re looking into and you’ll get the desired results (e.g., when you’re searching for an error message like “syntax error”):

$ grep "syntax error" /var/log/apache2/error.log

Logging on the NGINX Server

NGINX is also a commonly used server. You can access the log messages from the NGINX server by navigating to the following directories:

• /var/log/nginx/access.log
• /var/log/nginx/error.log

The first step is knowing where your log files are. This is important because it’ll bring you closer to actually solving the issue at hand. You can use the cat command to open the file and read its content, and the command is as simple as the one below:

$ cat ./var/log/nginx/error.log

You can also filter the error.log file to get specific information, and you can still use the grep command just like you can when accessing logs via the Apache server. Here’s an example of how you can filter a log file using a grep command:

$ grep "syntax error" /var/log/nginx/error.log

Any syntactical error message will be filtered for you. You can use this to solve syntax errors you have in your code. As you can see, grep is a powerful command. If you can, utilize its power and do even more advanced regular expressions to filter logs.

Writing all these commands isn’t easy, and neither is viewing data on the console. Reading data on the console is difficult because it’s usually not well aligned. It’s even more difficult if you’re reading a larger file. Even though these commands will do the job, you have to be patient enough to avoid making errors as you write them. If writing commands isn’t something you enjoy, there are logging tools designed to help you log your web apps. SolarWinds^® Papertrail^™, for example, is a cloud-based logging tool you can easily set up and start logging with. You’ll have a better feel and also be able to search syntax and filter logs in real time, an important bonus.

Conclusion

Logging is an important technology in website hosting, as it helps you troubleshoot server issues and optimize websites. You can also use server logs to know where most of the traffic is coming from, and you can look for things such as what files are accessed most by the website and its users. If you’re a web developer, you’ll need logs from the server. These can be used either to fix errors or to optimize the website.

Server logs can help you with almost anything regarding website issues and performance. But analyzing logs on the console won’t be easy, so you can use a cloud-based logging tool like Papertrail. With Papertrail, you can search logs and filter based on what you need (e.g., filter an IP address).

This post was written by Mathews Musukuma. Mathews is a software engineer with experience in web and application development. Some of his skills include Python/Django, JavaScript, and the Ionic Framework. Over time, Mathews has also developed an interest in technical content writing.