Logging is one of those Node.js functions that’s easy to take for granted — until some unexpected event results in hours, sometimes days, of searching through log files to pinpoint the source of a problem.
Node.js apps create basic text logs just like any other application. These logs can be easily generated using open source logging libraries, such as the Winston or Bunyan libraries. These libraries work fine for managing relatively small applications, but developers creating massive amounts of logs on a production server need to be able to quickly and efficiently troubleshoot infrastructure and application issues.
Logging Best Practices
There are a number of best practices to follow when setting up your logs. First, you should ensure logging is enabled in the application you are using. Below we will show you how you can turn on logging using popular libraries such as Winston and Bunyan.
Next, it’s important to categorize your logs so they are easy to maintain and search. Categorization can facilitate the process of searching through large log files, and filtering for the information you need. You’ll see some examples of how to do this using groups and attributes.
You should also log uncaught exceptions because if an event fires and there is no error listener, it propagates to the Node.js process and will print the stack trace and crash it. To avoid this, you can listen to an uncaught exception event on the process object or configure your logging framework to handle it. We’ll also show you how you can set this up later.
Node applications on a production server often contain millions of lines of log entries. When you need to troubleshoot for any reason, it can be a daunting task. During the development process, command line tools provide an easy interface for console-level log review. But analyzing large volumes of data on a production server requires an automated process that enables you to easily query and analyze your logs.
For production-scale logging, you need a robust set of automated troubleshooting tools that allows you to go beyond console-level review. In a production environment, it’s important to maintain a central log file and/or stream your logs to a management tool. In addition to notifying you about new issues, a log management tool can help you identify the source of unexpected errors, as well as bugs that may have been missed earlier in the development cycle.
The tools best-suited for your needs will be influenced by the way in which you manage your data. If your data is in an Apache log format such as Morgan logs, you’ll be better off with a management tool that can understand and search the Apache format natively. If the data is in JSON format, you need an analysis tool that understands JSON.
Use a Logging Library to Add Functionality
Popular node log management solutions, such as Bunyan and Winston, are designed to add functionality to the logging process. For example, you can format JSON output using Bunyan, a well-tested tool for node logging. Bunyan lets you create and format JSON versions of your logs. This is especially useful if you plan to use external services, such as Papertrail™, to manage, read, and analyze your logs.
Bunyan is a fast and easy-to-use logging module that includes a process id (pid), hostname, and timestamp when sending log messages. You can use Bunyan for custom transports in order to log data to various locations, such as Syslog daemons, Slack notifications, and rotating files.
You also can use Bunyan to log based on priority level. This facilitates the creation of custom loggers, which it refers to as child loggers, and allows for log customization. For example, this will enable you to log different information in your development and production environments.
To use Bunyan, create and name a Logger. You can use Bunyan’s logging API to customize the output format you prefer.
var bunyan = require('bunyan');
var log = bunyan.createLogger({name: 'papertrail-test'});
log.info('Welcome to the app'); // Just text
log.warn({port: bind}, 'port'); // Text plus metadata
This example, using Bunyan, demonstrates how the log level can be encoded using the number in the “level” property of the resulting JSON object.
var bunyan = require('bunyan');
var log = bunyan.createLogger({name: 'play', level: 'debug'});
log.info('<i>Book ‘Catch 22’ has been added to the database</i>');
It will produce the following log entry:
{
'name': 'papertrail',
'hostname': 'mainserv',
'pid': 32411,
'level': 30,
'msg': '<i>Book ‘Catch 22’ has been added to the database</i>',
'time': '2018-05-19T23:57:12.112Z',
'v': 0
}
Another logging option that can be particularly useful for handling complex application server logs is Winston. Using Winston, you can define and customize your logs. This includes defining destinations, adding metadata, and maintaining multiple asynchronous logs.
Winston is one of the most popular logging libraries available for Node.js. It has a large and robust feature set that is easy to install, configure, and use. Among its many features are the ability to use multiple transports, create custom transports, stream logs, and query logs.
For instance, with Winston you can define a custom logger that includes a timestamp, uses uppercase for the logging level, and adds specific metadata you define in a log message. Here are some examples of this process in action:
// Create custom logger
var logger = new(winston.Logger)({
transports: [
new(winston.transports.Console)({
timestamp: function() {
return Date.now();
},
formatter: function(options) {
return options.timestamp() + ' ' + options.level.toUpperCase() +
' ' + (options.message ? options.message : '') +
(options.meta && Object.keys(options.meta).length ? 'nt' +
JSON.stringify(options.meta) : '');
}
})
]
});
The following is an example of log level encoding using Winston:
var winston = require('winston');
winston.level = 'debug';
winston.log('info', '<i>Book ‘Catch 22’ has been added to the database</i>');
It will produce the following log:
info: <i>Book ‘Catch 22 has been added to database</i>
You can also log uncaught exceptions by configuring Winston when you create the logger.
const logger = createLogger({
transports: [
new transports.File({ filename: 'combined.log' })
],
exceptionHandlers: [
new transports.File({ filename: 'exceptions.log' })
]
});
Troubleshoot Problems Fast Using Papertrail
Cloud-hosted monitoring solutions are an increasingly popular option for production-scale log management, aggregation, and search. SolarWinds® Papertrail is a cloud-hosted log management service, which is designed to integrate with popular logging library solutions and help make it easy for you to consolidate all of your Node.js logs in one place.
Papertrail is intended to help you centralize your log management in the cloud. You can quickly and easily use the Papertrail solution to configure centralized logging from Node.js apps. Papertrail can handle logs directly from the Node.js app using a regular text log file, or by integrating with logging libraries, such as Winston or Bunyan.
Papertrail is purpose-built to provide instant log visibility. It can typically be set up in less than a minute, and provides features such as fast search, integration with multiple alerting services, flexible system groups, team-wide access, and webhook monitoring.
In addition to functionality, Papertrail may add value to the log data you already collect. It is designed to make it easier for you to identify error messages, app requests, slow DB queries, config changes, and application anomalies.
Papertrail features numerous benefits for handling Node log files, including:
Log aggregation. Papertrail allows you to consolidate all of your Node logs with syslogs, and other application and database logs, giving you easy access to application syslogs, text logs, and Apache logs — all in one location.
Tail and search logs. With Papertrail, you can search stored or streaming log data using search query syntax, which can help you quickly troubleshoot issues. You also can search large volumes of log messages using the robust search capabilities built-in to Papertrail. The Papertrail GUI can also expedite the process of identifying mission-critical issues that may need to be addressed in a production environment.
Instant alert notifications. Papertrail is designed to help minimize downtime. You can receive alerts via email, or send them to Slack, Librato, PagerDuty (or any custom HTTP webhooks of your choice.) Moreso, alerts can be accessed from a web page that enables customized filtering. For example, you can filter by name or tag.
Log analysis. Papertrail log archives can be loaded into third-party data management utilities, such as Redshift or Hadoop, to enable even more robust queries and analytics. With the Papertrail built-in log velocity analytics, you can visualize log throughput to quickly identify patterns and anomalies.
Scalability. You can also use Papertrail to scale to your log volume and desired searchable duration.
Encryption. Papertrail facilitates encrypted logs, which adds another layer of application security in the production environment. In addition, Papertrail supports optional TLS encryption and certificate-based destination host verification.
Configuring Node to send logs to Papertrail
Getting up and running with Papertrail is quick and easy. If you already have log files, you can transmit them to Papertrail using remote_syslog2. This small program will monitor the log files and send new lines to Papertrail.
> remote_syslog -D --tls -p 1234 /var/log/mysqld.log
If you prefer to use Winston, you can also send events asynchronously from Node.js using the Winston Papertrail transport. Update the host and port to the ones given to you in your Papertrail log destination settings.
require('winston-papertrail').Papertrail;
var winstonPapertrail = new winston.transports.Papertrail({
host: 'logs.papertrailapp.com',
port: 12345
});
var logger = new winston.Logger({
transports: [winstonPapertrail]
});
If you prefer to use Bunyan, you can use the node-bunyan-syslog stream. Update the host and port to the ones given to you in your Papertrail log destination settings.
var bsyslog = require('bunyan-syslog');
var log = bunyan.createLogger({
name: 'foo',
streams: [{
level: 'debug',
type: 'raw',
stream: bsyslog.createBunyanStream({
type: 'sys',
facility: bsyslog.local0,
host: 'logs5.papertrailapp.com',
port: 59738
})
}]
});
Conclusion
Papertrail goes beyond giving you centralized management of your logs. It can enable you to see more value from the log data you already collect. This is a log management tool designed to help you troubleshoot customer problems, resolve error messages, improve slow database queries, and implement config changes. Moreso, Papertrail can give you analytical tools to identify and resolve system anomalies and potential security issues.
Learn more about how Papertrail can give you frustration-free log management in the cloud and sign up for a trial or the free plan to get started.
The Papertrail trademarks, service marks, and logos are the exclusive property of Papertrail, Inc. or its affiliates. All other trademarks are the property of their respective owners.
