Administrator Guide
- Documentation for Papertrail
Collect Logs
Collect Logs: Apps & Services
Collect Logs: Hosting Services
Collect Logs: Integrations
- Fluentd
- Librato
- Amazon CloudWatch
- StatHat
- Amazon SNS
- HostedGraphite
- Datadog
- Zapier
- VictorOps
- New Relic traces
- Slack
- Campfire
- PagerDuty
- New Relic Insights
- Pushover
- AppOptics
- Geckoboard
- OpsGenie
- Honeybadger
Manage Logs
Send Logs for Analytics
SolarWinds Users & Orgs
Support and Security
- Security and Compliance
- Support Policy
What's New
- Papertrail Release Notes

Encrypting with TLS (SSL)

Log messages can be delivered to Papertrail using TLS-encrypted syslog over TCP, as well as over UDP. Papertrail also supports TCP without TLS, though it isn’t often used.

This page describes how to configure rsyslog or syslog-ng for encrypted logging. If you’re using remote_syslog2 rather than rsyslog or syslog-ng, its README contains TLS setup instructions.

When configuring logging on your first system, consider configuring plaintext logging, verifying that it works, and then changing to TLS.

rsyslog.conf

Download root certificates

Save https://papertrailapp.com/tools/papertrail-bundle.pem into /etc/papertrail-bundle.pem on the log sender. For example:

$ sudo curl -o /etc/papertrail-bundle.pem https://papertrailapp.com/tools/papertrail-bundle.pem

Its MD5 checksum is currently 1062c59e49c4585a9acfaad740a79c5d, but it is updated periodically. Do not hard-code the checksum in deployment scripts. (To ensure a repeatable result, embed the entire bundle instead.)

rsyslog trusts these root CA keys to validate the key presented by Papertrail, preventing man-in-the-middle attacks.

Add TLS configuration

When using TLS, we strongly recommend running the latest minor version of your rsyslog release. Where possible, run rsyslog v7 or v8; these fix many bugs with TLS.

The instructions below assume rsyslog 4.0+.

On many distros, also install the rsyslog-gnutls package (including CentOS, Fedora, Debian, and Ubuntu). Alpine Linux requires a similar package named rsyslog-tls.

Starting with unencrypted logging, find the line that sends to Papertrail, similar to

*.*     @logsN.papertrailapp.com:XXXXX`

Above that line, paste:

$DefaultNetstreamDriverCAFile /etc/papertrail-bundle.pem # trust these CAs
$ActionSendStreamDriver gtls # use gtls netstream driver
$ActionSendStreamDriverMode 1 # require TLS
$ActionSendStreamDriverAuthMode x509/name # authenticate by hostname
$ActionSendStreamDriverPermittedPeer *.papertrailapp.com

See Download root certificates to download the required CA file. Finally, on the Papertrail destination line, change the @ before the hostname to @@ (2 at-signs), which tells rsyslog to use TCP. For example:

*.*     @@logs.papertrailapp.com

If the destination line used a different hostname and/or a port other than 514, update the values in configuration to match. For example, to log to logsN.papertrailapp.com on port XXXXX with TCP and TLS:

*.*     @@logsN.papertrailapp.com:XXXXX

Restart rsyslog

Restart rsyslog so it detects the TLS-over-TCP destination:

$ sudo /etc/init.d/rsyslog restart

Connection error handling (recommended)

After logging is working, we strongly recommend adding this configuration to make rsyslog queue locally and reconnect if the TCP connection drops.

Without this extra configuration, rsyslog may not reconnect to Papertrail or may block on inbound syslog() calls from apps, both of which are bad. Head over here and paste it in.

Troubleshooting

could not load module '/usr/lib/rsyslog/lmnsd_gtls.so',
rsyslog error -2078 [try http://www.rsyslog.com/e/2068 ]

First, make sure that module exists by running ls against the path in the error, such as:

$ ls -la /usr/lib/rsyslog/lmnsd_gtls.so

If it doesn’t exist, install the related package (often called rsyslog-gnutls or rsyslog-tls), or if you compiled rsyslog from source, compile the module.

Second, ensure that the user that runs rsyslog has permissions to read Papertrail’s public key (in the instructions above, /etc/papertrail-bundle.pem). On many distributions, rsyslog starts as root and then drops to a user. In that case, run: chmod 644 /etc/papertrail-bundle.pem to let all users read the key file.

Finally, this may appear if you are using $ModLoad lmnsd_gtls to explicitly load the TLS module, and that configuration option occurs before the $DefaultNetstreamDriverCAFile has been defined. Explicitly loading the module is rarely required and the configuration above does not use it. We recommend removing that $ModLoad lmnsd_gtls option and relying on autoloading. If your lmnsd_gtls needs to be explicitly loaded, such as because it is in a non-default location, move the $DefaultNetstreamDriverCAFile config line above the $ModLoad line.

For more generic troubleshooting information, see Troubleshooting remote syslog reachability.

For additional logging tips, see Advanced Unix logging tips, notably Aggregate local log files with rsyslog and Tweak queue options for connection failure.

syslog-ng.conf

Download root certificates

Download and extract root CA certificates for syslog-ng:

$ sudo mkdir /etc/syslog-ng/cert.d
$ cd /etc/syslog-ng/cert.d
$ curl https://papertrailapp.com/tools/papertrail-bundle.tar.gz | sudo tar xzf -

The MD5 checksum of papertrail-bundle.tar.gz is currently 004e8ff71cd19dc4550650fd8f5f8450, but it is updated periodically. Do not hard-code the checksum in deployment scripts. (To ensure a repeatable result, embed the entire bundle instead.)

syslog-ng trusts these root CA certificates to validate the authenticity of the key presented by Papertrail, preventing man-in-the-middle attacks.

Add TLS configuration

Starting with unencrypted logging, find the line that sends to Papertrail. It should be in the format: destination d_papertrail { .. }.

In that stanza, remove the existing udp(..) line. Replace it with the new TLS-over-TCP destination:

tcp("logs.papertrailapp.com" port(514) tls(ca_dir("/etc/syslog-ng/cert.d")) );

If the udp configuration used a different hostname and/or a port other than 514, update the values in the new tcp configuration to match. For example, to log to logsN.papertrailapp.com on port XXXXX with TCP and TLS:

destination d_papertrail {
  tcp("logsN.papertrailapp.com" port(XXXXX) tls(ca_dir("/etc/syslog-ng/cert.d")) );
};

Restart syslog-ng

Restart syslog-ng so it detects the TLS-over-TCP destination:

$ sudo killall -HUP syslog-ng

Verify (recommended)

To verify that messages are encrypted, run a packet sniffer like tcpdump, generate a log message, and confirm that the cleartext body is not shown. For example, to output the payload of packets to logsN.papertrailapp.com:

$ sudo tcpdump -s 1500 -X src or dst logsN.papertrailapp.com

Change logsN to your Papertrail host, as shown under Log Destinations. You should see packets flowing, and they should not contain human-readable log text.

Troubleshooting

Consider setting up cleartext logging first, then moving to TLS encryption once cleartext works.

If cleartext logging is working, but TLS is not, attempt a TLS-encrypted TCP connection to Papertrail using the s_client feature of openssl.

Run:

$ openssl s_client -showcerts -connect logsN.papertrailapp.com:XXXXX -CAfile /etc/papertrail-bundle.pem

where logsN and XXXXX are the name and port number shown under log destinations and /etc/papertrail-bundle.pem is the path to the certificate that can be downloaded here.

For more generic troubleshooting information, see Troubleshooting remote syslog reachability.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.