Modern apps and services generate a constant stream of log data capturing the inner workings of the code and how users interact with it externally. Buried inside those logs are golden nuggets of information critical for troubleshooting problems in development and in production.
But without powerful tools to efficiently extract insights from your logs, they’re just useless files taking up space in the cloud. To really get value from log files, you need a string of features to dig into everything from application errors to performance bottlenecks.
SolarWinds® Papertrail™ is a great tool for this job. It provides a search query syntax for quickly finding the data you need in huge log files, a live tail feature to monitor log messages in real time, log velocity analytics for visualizing data, and many more features to help you quickly and efficiently find problems and keep your software running.
This article focuses on two scenarios developers deal with every day: working with app errors and uncovering performance issues. First, let’s look at how to use Papertrail to discover and analyze errors in your app logs.
Analyzing Application Errors
Tip 1 – Focused Searches
As much as we’d like to think we can write bug-free code, errors happen in production. It’s important to monitor them to catch issues affecting users in real-time. The event viewer in Papertrail allows you to search for specific keywords in your logs such as error critical or panic using a simple search syntax. This syntax also supports boolean operators such as AND and OR that allow you to group and restrict small expressions. For example, here’s a query string to find all the log messages containing the string “error” and “app01”:
error AND app01A negation operator (–) is also supported which provides a way to exclude results containing a given string. If you want to find all messageswith “error” but without the word “memory,” you can use this query:
error AND -memoryThe search syntax also supports attributes, which allow you to filter your logs using the value in specific fields of your log messages. For example, to find all the messages with a severity of critical you can use the following search query in the Papertrail event viewer:
severity:critSometimes it’s handy to be able to group multiple subqueries into a larger one, like if you or your coworkers have already built queries during previous searches, or to control the order of evaluation for subqueries. For this, you can use parentheses:
“app01” AND (“crit” OR “emerg”)If you remove the parentheses, it changes the meaning of the query.
Tip 2 – Go Real Time
For tricky bugs, viewing error messages in real time might be the only way to figure out how multiple services or components of your software are interacting. The Papertrail event viewer supports a live tail feature, which is like running tail -f on your log files—only live tail works across all your logs from within your web browser. This means there’s no need to log in to individual servers. So, no matter where your logs are generated, you can view them in real time. Additionally, you can search and filter the live stream of events to zone in on the parts you’re analyzing.
Tip 3 – Context Matters
Exceptions and stack traces are also important pieces of the puzzle when digging into app issues. Unfortunately, log messages including this type of data are split across multiple lines, which can make it difficult for traditional log management tools to match multiple messages from a single exception or stack trace. Papertrail offers context links to solve this problem. Context links allow you to easily filter log messages by clicking on identifiers in a log message. For example, to see the entire crash message for a server, clicking the server’s hostname in the log will discard all other messages and only show the logs from that server.
Investigating Performance Problems
Performance issues are easy to miss because they don’t result in an outright error or failure. Instead, performance exists on a spectrum. Users don’t hit a missing web page, things just gradually get slower and slower until users lose patience and go somewhere else. Detecting and analyzing these kinds of issues requires well-crafted logs, intuitive tools, and a deep sense of how your app works.
Tip 4 – Threshold Alerts to Catch Changes in Volume
Papertrail allows you to turn your saved searches into alerts by assigning a schedule. To catch changes in performance, however, what works well is to set an alert with a high minimum threshold. This type of alert will notify you when you have an unusually high volume of messages matching the alert condition. Threshold alerts can help you catch issues before there’s a service interruption and before your users start calling.
However, detecting potential performance problems is only the first step. Next, you need to be notified. Papertrail integrates with many third-party services such as Slack, PagerDuty, and Campfire, so your whole team can receive a message as soon as an alert is triggered.
Tip 5 – Visualize Log Volume to Spot Trends
What if you want to check how the performance of your app is changing over time? One of the features of Papertrail particularly useful for understanding this is velocity graph analytics. This feature allows you to create graphs and data visualizations from your log data, making it easy to see the volume of log messages matching different search criteria at a glance and how it changed over time. You can also configure the graph to update periodically as new log data is received, giving you a rolling snapshot of the behavior of your software.
Once you’ve identified an anomaly in log velocity analytics you want to investigate, the Papertrail event viewer allows you to jump to a specific period by seeking to a date and time.
Conclusion
Application errors and performance issues are a standard part of every developer’s daily life. But despite their commonplace nature, log analysis tools like Papertrail can make all the difference when you need to understand and fix them. Papertrail makes it effortless to work with large volumes of log data with its intuitive search query syntax which supports boolean operators, negation, and parentheses for combining smaller queries into large ones.
Emergency situations demand tools capable of inspecting logs in real time. The live tail feature from Papertrail lets you view log messages as they’re received. You can do everything you’d normally do when inspecting static log data including searching, filtering, and scrolling backwards and forwards in time. And Papertrail support for multiline messages makes it easy to find the entire contents of exceptions and stack traces.
Performance problems are fundamentally different from app errors and require different support from your analysis tools. Detecting when performance problems occur can be tackled with Papertrail alerts, which enable you to turn saved searches into a schedulable monitoring system. Alerts also integrate seamlessly with third-party communication tools such as Slack, PagerDuty, and Campfire, and with log velocity analytics, you can create data visualizations to spot high-level trends and patterns in your performance metrics.
To start a free trial with SolarWinds Papertrail, head on over to papertrail.com.