Container logging adds some complexities we don’t experience when using VMs or dedicated hardware. With containers, once the container dies, the logs and data for the container also die. This may not be a problem for small applications with little logging. But for more complex applications or applications running in production, you need to start thinking about log persistence and management.
For applications running in Docker containers, you need a way to easily access, search, and archive logs. In Docker, most applications will have multiple instances and multiple services working together, so centralized logging is essential. It can help you figure out what’s going on and what might be going wrong. Centralizing your logs makes it easier to query logs from different containers in one place long after those containers die. You can then start to correlate events between containers as well.
In this post, we’ll cover how to get started with centralized log management for applications running in Docker containers.
Background
The Docker daemon users the default logging driver each time you create a new Docker container. For the example application in this post, the default logger is the json-file driver. This means the logs will be stored within the container in JSON format.
If you want to see what logging driver your containers use, you can retrieve the driver using this command:
docker info --format '{{.LoggingDriver}}'To get started or just log locally, the default driver works fine. However, if you want logs to persist longer than the life of the container, the usefulness of the default driver diminishes. Once you start running multiple instances of your service, centralizing and correlating data will become essential for troubleshooting and optimizing performance
So what are the different logging options for Docker logs?
Options
When considering how I want to configure Docker logs, I look at the following.
Centralized vs. Sidecar
With a centralized service, all your Docker containers route their logs to one place. This is usually where I start because it’s simpler and doesn’t result in doubling the number of containers I have to manage. You can collect logs from all your containers, store them in a single container, and even send them to a different container for analysis or archiving.
Alternatively, with the sidecar approach, each of your Docker containers has a logging container associated with it. They share disk drive space access. This is great for large-scale or enterprise systems where fine-grained control and scaling are necessary. This approach, however, can add complexity and be challenging to set up and scale. A logging sidecar can require more resources than the default logging method, which can increase platform costs. You also need to make sure the sidecar and application containers are working as a single unit, or you could end up losing data.
Drivers, Remote Syslog, and Logspout
Next, let’s consider other logging drivers like remote syslog and logspout.
With these logging drivers, it’s easy to send your logs to syslog, Fluentd, or other daemons and forward your logs to remote log aggregators. One thing to keep in mind is if you use a driver other than json-file or local, you won’t be able to access logs at the container level with the typical “docker logs CONTAINER_NAME” command. If you’re already used to sending your logs to a remote log aggregator, this won’t be a problem. But if you’re used to troubleshooting issues inside Docker, it can make local debugging harder.
As an alternative, with remote syslog (remote_syslog2 or rsyslog), you’re sending your logs to a remote syslog service. They don’t live inside Docker and need to be monitored outside the Docker ecosystem, but you may already be using remote syslog to centralize logs from other applications. If you’re already using this method for non-Docker systems, it’s easy to do the same for your container logs.
Another option to consider is logspout (as well as other API-based tools), which runs inside Docker and automatically routes all container logs based on your configuration. This means the logspout container will only connect to external systems like SolarWinds® Papertrail™. However, you won’t be able to use logspout for non-Docker container logging. This Docker API-based logging tool allows your containers to keep their local logs while also sending them to logspout, so you won’t run into issues with each application container failing to connect to a remote system. Additionally, you can still configure logspout to send your logs to different places.
Blocking vs. Non-Blocking
Docker sends logs in one of two ways: blocking (the default) or non-blocking.
With blocking, Docker suspends operations in the container to send the log event. This ensures delivery of the log, but it may have performance implications. If you’re using the local or json-file driver, there’s not too much concern about latency. But if the driver you’re using sends logs to a remote system, then latency can affect the performance of your application.
Alternatively, with non-blocking, the container first writes the logs to a buffer. Then, when available, the driver sends the logs from the buffer. This reduces the risk of logging affecting your application performance. However, if the buffer fills up, it may result in lost logs, as the logging driver may not be able to keep up.
Decision Time
In the previous section, I reviewed points to consider for logging setup. So how do you choose a centralized log aggregator? Ultimately, it depends on your application needs and your comfort with the various technologies.
For my use case, I’ll be using logspout as my centralized log aggregator. I’m also going to start with blocking mode. Because the driver I’m using doesn’t send logs to an external system from my container, I’m less worried about latency. I can always change this later if I run into issues. Finally, I’ll be routing my logs from logspout to Papertrail so I can easily search and correlate my logs.
You can get more information from the SolarWinds documentation as well as the logspout documentation.
Diagnosing Issues
No matter what decisions you’ve made for your logging, you’ll need to debug issues from time to time.
Set Up Troubleshooting
When it comes to setting up your centralized logging, take it one step at a time. Work iteratively to add each feature, only moving on once you know you’re in a working state. For example, don’t route the logs to an external source until you know they’re being sent to logspout or syslog. Additionally, don’t add TLS until you know sending logs to your remote log management system works.
When you first run logspout, don’t worry about hooking it up to your log management system. Just add a –publish flag so you can curl the output before you send the logs anywhere.
$ docker run -d --name="logspout" \
--volume=/var/run/docker.sock:/var/run/docker.sock \
--publish=127.0.0.1:8000:80 \
gliderlabs/logspout
$ curl http://127.0.0.1:8000/logs
Then, add features piece by piece until you’re hooked up to Papertrail and configured how you want.
After a few iterations, my logspout startup looks similar to this:
$ docker run --restart=always -d \
--name='logspout' \
-v=/var/run/docker.sock:/var/run/docker.sock \
-e SYSLOG_HOSTNAME=`{{.ContainerName}} -s` gliderlabs/logspout \
multiline+syslog+tls://logsN.papertrailapp.com:XXXXX
Iterative configuration will help you troubleshoot where the problem resides.
Running Troubleshooting
Once you have your centralized logging set up, you may periodically run into issues where things go wrong. As always, break it down step by step. First, is your container sending logs? If you’re using the default json-file driver or logspout, you can still run “docker logs CONTAINER_NAME” to validate your container is logging. Next, make sure logs make it to logspout or syslog. Validate any environment variables or config settings you have and ensure they’re appropriate for your app.
At each step, if you find an issue, try restarting the container or logging service (logspout, fluentd, syslog) first. For example, if logspout was connected at one time but loses connection to Papertrail or other external services, it may need to be restarted.
If you continue to run into issues with your setup, you may want to consider another logging option. Your use case may require a change.
Summary
When optimizing your Docker apps, you should centralize your logging sooner rather than later to help resolve issues. Consider what options work best for your use case or experiment with a few of these options to see what works best.
No matter what choices you make for centralizing your Docker logs, you can always use Papertrail. It’ll provide an easy way to visualize, search, and correlate logs for your Docker containers.
This post was written by Sylvia Fronczak. Sylvia is a software developer who has worked in various industries with various software methodologies. She’s currently focused on design practices the whole team can own, understand, and evolve over time.