Local Kubernetes clusters are great for both developers and system engineers. When developing applications, programmers can use local clusters to make sure their application can be easily and correctly deployed without the need for configuring real infrastructure. System engineers can use them for testing, creating proofs of concept (POCs), or learning and trying new tools.
There are a few options for deploying local Kubernetes clusters, and they all work slightly differently. However, no matter how you deploy your cluster, you’re probably interested in learning how to aggregate logs from your containers. Logs are crucial for finding issues, debugging applications, and understanding how complex tools operate. In this post, you’ll learn how to read and aggregate logs from applications running on your local Kubernetes cluster.
Local vs. Cloud-Based Kubernetes
To understand the challenge of aggregating logs from local Kubernetes clusters, let’s quickly discuss how they differ from their cloud-based cousins.
First, not all local Kubernetes clusters are “real Kubernetes” clusters. Some of them—for example, kind—imitate Kubernetes in Docker. Of course, they still work as a normal Kubernetes cluster, but some may not be compatible with off-the-shelf solutions built for “normal” Kubernetes. It doesn’t happen often, but it’s something to keep in mind.
Second, the biggest difference is cloud-based managed clusters usually offer one-click logging solutions from cloud providers. Therefore, local clusters typically require much more work to achieve the same results. But this also depends on the solution you choose.
The third difference is the storage architecture. When using cloud-based Kubernetes, we usually use S3-like cloud storage to save the logs. Only from there are they picked up by a log management tool. On local clusters, however, we usually don’t have any separated centralized storage available.
Log Aggregation Options
Now, let’s focus on local clusters. Fortunately, the actual logging strategy doesn’t differ much for local clusters. With this said, you still have two main options: per-node or sidecar-based logging aggregation. Which one you should choose will depend on your specific use case. It’ll also depend on what you use your local Kubernetes cluster for. Is it only for testing new tools? Is it the official local development solution in your company? Depending on your needs, one of the following options may work better for you:
Node-Level Agents
The first option is to deploy tools like Fluentd, Filebeat, or Logstash to gather the logs from all nodes in your local cluster. One instance of a preferred application will then be deployed per node and will gather logs from all containers on the node. Typically, this is done using ReplicaSet or ReplicationController. An example deployment can be found here. Of course, the actual deployment will differ depending on where you want to send your logs, but the deployment itself will look similar.
The only thing you need to be aware of when using this method is sometimes, local Kubernetes clusters don’t have PersistentVolumes. Sometimes only specific types are available, like in the case of minikube, where by default only hostPath type PersistentVolumes is allowed. In such cases, you need to adjust your Fluentd config accordingly (either use only hostPath type PersistentVolumes or change to other storage options).
Sidecar Approach
Deploying a logging agent as a sidecar is another option, and it works pretty well with local clusters. In general, however, this option is a bit more complicated and requires more maintenance. With node-based agents, you only need to create one deployment with one configuration and you’re good to go. With the sidecar approach, whenever you need to change the configuration, you must restart all the pods.
On the other hand, since local clusters are often used only as a dev/POC environment, you may find a sidecar approach more beneficial because it gives you more flexibility. For example, you can ship the logs from only a few important applications and skip the rest. You can also create two or more configurations to send logs from the main application to one place and development containers to another. This can be useful when you don’t need to bombard your company’s centralized logging server with local debug messages, for example. Another use case for a sidecar approach is to aggregate logs from an application that doesn’t support logging to STDOUT/STDERR (so logs wouldn’t be picked up by an agent on a node level).
So how exactly does the sidecar agent work? You deploy a second container in the same pod as your application. Then, you use volume mounts to share the same directory for log files between the application pod and agent pod. The application then writes log files into the directory, and an agent from another container reads these files and sends them wherever you want. An example could look like this:
spec:
containers:
- name: your_application
image: your_app:latest
volumeMounts:
- name: log-storage
mountPath: /var/log
- name: sidecar-log-agent
image: fluent/fluentd:latest
volumeMounts:
- name: log-storage
readOnly: true
mountPath: /mnt/log
volumes:
- name: log-storage
emptyDir: {}
Things to Consider
As you can see, aggregating logs from local Kubernetes clusters doesn’t differ much from cloud-based clusters. However, we’ve talked only about gathering logs so far—you still need to ship them somewhere and analyze them. This is another story entirely and where most of the differences lie. You can either spend days setting up your own Elastic cluster, or you can use a solution like Solarwinds® Papertrail™, which basically offloads all the installation and maintenance.
Frustration-free log management. (It’s a thing.)
Aggregate, organize, and manage your logs with PapertrailSetting up log aggregation from Kubernetes clusters in Papertrail just got easier. Papertrail recently announced rKubeLog, a lightweight, open-source Kubernetes connector that doesn’t require special permissions or daemons. The rKubeLog connector is perfect for logging applications running in local Kubernetes clusters or any environment where you don’t have ownership of the infrastructure or can’t deploy Daemonsets, such as EKS or nodeless clusters.
However, it’s not just about the installation and maintenance. Let’s take a step back and discuss what we need logs for in general. Whenever issues happen, logs are the first things you look at, right? Basically, logs are an important tool to help you keep your application up and running. Therefore, the better the log management solution, the less downtime you get. And this is why, although a good log aggregation strategy is important, the capabilities of the log management tool you select are critical.
Having all your logs in one place certainly helps you find the root cause of the issues faster. But you still need to read and understand them to find the problem. This can take some time—even more so on local clusters, since we usually enable debug logging there. How can we improve this? Tools like Papertrail can help you find the root cause of issues with proactive monitoring, the ability to view and search live log streams, contextual searching, and alerting. The quicker you find the root cause, the fewer users affected.
Summary
Logging is crucial for solving issues. Local Kubernetes clusters require a slightly different approach for log aggregation. Depending on which tool you use for local Kubernetes, you may need to change the configuration of your log agents. In general, however, you can use the same node-based or sidecar approaches. Log aggregation is only the first step, though; you must choose what to do next. Do you ship the logs to a custom Elastic cluster? Do you ship them to a cloud-based solution? Or do you go to the next level and sign up for Papertrail, which can help you understand your logs better and offers a lightweight Kubernetes log collector? How you decide to address log aggregation and management will have an impact on your ability to identify and resolve issues quickly.
This post was written by Dawid Ziolkowski. Dawid has 10 years of experience as a network/system engineer, has worked in DevOps, and has recently worked as a cloud-native engineer. He’s worked for an IT outsourcing company, a research institute, a telco, a hosting company, and a consultancy company, so he’s gathered a lot of knowledge from different perspectives. Nowadays, he’s helping companies move to the cloud and/or redesign their infrastructure for a more cloud-native approach.