Troubleshooting reachability

After configuring a system to log to Papertrail, if logs aren’t appearing, or aren’t appearing as expected, these checks help verify end-to-end reachability from your system to Papertrail (with few or no system changes). This can identify whether a problem is with the system/logger configuration, network reachability, or Papertrail configuration.

We’re here to help and we love this stuff, so don’t feel like you need to go through this process alone. Contact us.

Tests

Basic IP reachability (telnet)
Hostname format (hostname)
Sending data (tcpdump)
Examine logger daemon activity (strace)
System logging config (logger)
Use remote_syslog2 (remote_syslog2)
Troubleshooting using Windows
Interpreting tests

Configuration and limits

Message length
Message format
Firewall policies
Idle timeouts
Rate limits
Port reaping
GNU syslogd

Tests

Before diving into these troubleshooting steps, it’s always worth confirming that there are no firewall policies in place that may be preventing a connection to Papertrail. Also, confirm your account destination is configured to accept new messages and set to accept traffic using the same protocol(s) as the sender (UDP, TCP plain text, TCP+TLS).

Basic IP Reachability

To see whether your system can reach Papertrail, run:

$ telnet logs.papertrailapp.com 514

You should see a connection established (“Connected to…”) but won’t see any output. Here’s an example:

$ telnet logs.papertrailapp.com 514
Connected to logs.papertrailapp.com.
Escape character is '^]'.
^]
telnet> close

Note that this only tests TCP reachability to the default syslog port. It doesn’t test UDP reachability. Replace logs and 514 with the hostname and port number for the account destination to test TCP connectivity to alternate ports.

See Troubleshooting using Windows for more specific Windows instructions, since the telnet command might not be installed.

Hostname Format

The syslog packet includes a hostname field. Papertrail accepts syslog messages with hostnames that can exist in DNS (letters, numbers, periods, hyphens, and to be permissive, underscores).

Occasionally, the operating system will have an invalid hostname like (none), or the syslog daemon will be transmitting a hostname value different from the operating system hostname.

OS hostname

Run:

$ hostname
$ hostname -f

Check the output to verify that the system hostname doesn’t contain parentheses or other non-DNS characters.

Syslog hostname

Using syslog-ng? Look for the chain_hostnames option in syslog-ng.conf. If found, comment out that line, run /etc/init.d/syslog-ng restart, and retry the test.

This will ensure that syslog-ng does not transmit hostnameA@hostnameB as the hostname. (If you use chained hostnames, contact us or consider keep_hostname).

Sending data

To see whether your logger(s) are actually sending packets to Papertrail, run:

$ sudo tcpdump -n -s 1500 -X port XXXXX

…where XXXXX is the port that the logger is sending to Papertrail on. While tcpdump is running, generate some log messages. You should see roughly one packet in the tcpdump output for each log message.

The output from tcpdump may be extremely verbose. To output it to a file, such as to send to Papertrail support, run:

$ sudo tcpdump -n -s 1500 -X port 11111 -w tcpdump.pcap

Examine logger daemon activity

To see the actual behavior of the logger daemon (like rsyslog or remote_syslog2), use strace. For example:

$ sudo strace -s 500 -tfp 22222

…where 22222 is the process ID of the logger, as seen in the second column of ps auxw. While strace is running, generate some log messages. The output from strace may be extremely verbose. To output it to a file, such as to send to Papertrail support, run:

$ sudo strace -s 500 -tfp 22222 -o strace.log

If the strace program is not found, install your distribution’s package with a command like sudo yum install strace or sudo apt-get install strace.

System Logging Config

To generate a log message that uses your system syslogd config (/etc/rsyslog.conf, /etc/syslog-ng.conf, or another syslog config), run:

$ logger "test"

to generate a message containing “test” as a syslog message.

If your system syslog is not set up properly (or cannot communicate with the remote syslog destination), this test message will also be affected by that problem. Try the instructions below to send a message directly to a host & port, without using the system’s syslog.

Use remote_syslog2

Install remote_syslog2, create a placeholder log file, then tell remote_syslog2 to watch it and send new lines to a hostname/port provided by Papertrail, and not to daemonize:

$ touch /tmp/testfile
$ remote_syslog -d logsN.papertrailapp.com -p XXXXX -D /tmp/testfile

Replace logsN and XXXXX with the host and port provided on log destinations. Then in another terminal or with remote_syslog2 backgrounded, append lines to the file:

$ echo "This is a test message" >> /tmp/testfile

Each message should appear in Papertrail momentarily.

Stop remote_syslog2 with Ctrl-C when finished testing.

Troubleshooting using Windows

Note for Windows users: the telnet command is not installed by default since Windows Vista. To enable this feature quickly use the command below:

dism /online /Enable-Feature /FeatureName:TelnetClient

The telnet command will be available right after dism completes. To run the test:

$ telnet logs.papertrailapp.com 514

You should see a brief message “Connecting to..” followed by an empty screen. If connection fails, telnet will respond with:

Connecting To logs.papertrailapp.com...Could not open connection to the host, on port 514: Connect failed

To capture packets on Windows, use Wireshark. Select the network, click Capture Options, and enter capture filter: port XXXXX (replace XXXXX with the port number which is used by the logger). This capture filter will catch traffic sent using both protocols: TCP and UDP. Click Start to see if there’s any traffic sent out.

Interpreting tests

The tests above are intended to help guide to the likely cause. Choose the test result below.

Test message arrived

The problem is probably with system or app server log configuration. Start here.

For example, consider running the log sender in debug mode and in the foreground to look for error messages, such as rsyslogd -dn or syslog-ng -dF.

Message arrived but was attributed to incorrect system

The problem is probably with the name which the sender is transmitting to Papertrail (and which Papertrail uses to identify to sender). Contact us and we’ll make specific suggestions for your hosting environment.

Test message did not arrive

There are a few possible causes:

The most likely is network reachability. If the system is behind a firewall, verify that the firewall policy permit log messages destined to the relevant port.
If the system is behind NAT and you provided Papertrail with a source IP, verify that the public NAT IP matches the IP in Papertrail’s system settings.
Ensure that “Yes, recognize logs from new systems” is enabled under the log destinations settings page, or that the system has been added using papertrail-add-system.

I’m not sure

Message Length

UDP

Papertrail accepts UDP packets of up to 1024 bytes (1 KB) in total packet length, which is about 950 characters for the log message text. That’s a very long line – 5-10 lines of typical dense text.

Note that the limit is applied separately to each packet (log line). Almost all lines longer than 1 KB are spread across multiple lines anyway (for example, tracebacks). As a result, this limit is almost never encountered.

1024 bytes is a protocol requirement defined in RFC 3164, which nearly all senders enforce (either by dropping longer messages or splitting them into multiple conforming messages). Even without the RFC requirement, UDP messages would be constrained by the Internet MTU of 1500 bytes minus the UDP header.

TCP

Papertrail accepts messages over TCP that are up to 100,000 bytes (~100 KB) long. 100 KB was chosen as the outer edge of what we could imagine a use for and to ensure Papertrail can handle messages from default configurations of all common syslog daemons. Messages larger than 100KB will be dropped, and the connection temporarily closed.

Message Format

Papertrail accepts messages that are formatted according to either RFC 3164 or RFC 5424. Senders may use either or both. No configuration within Papertrail is necessary.

While RFC 5424 defines the hostname field to be up to 255 characters, Papertrail truncates the hostname value to 120 characters.

Papertrail also accepts many types of malformed messages, generally because the message formats are generated by commonly-used software and hardware.

Sending over TCP

By default, Papertrail accepts messages over TCP with TLS. Unencrypted TCP is also supported, but not enabled by default. To enable it, visit Destinations.

Firewall policies

To add an explicit firewall rule with iptables that allows outgoing traffic based on a destination port (recommended), run these 2 commands. Change 514 to to the destination port:

$ iptables -I OUTPUT 1 --protocol udp --dport 514 -j ACCEPT
$ iptables -I OUTPUT 1 --protocol tcp --dport 514 -j ACCEPT

To allow based on destination IP, add rules allowing traffic to 67.214.208.0/20, 173.247.96.0/19, and 169.46.82.160/27:

$ iptables -I OUTPUT 1 -d 67.214.208.0/20 -j ACCEPT
$ iptables -I OUTPUT 1 -d 173.247.96.0/19 -j ACCEPT
$ iptables -I OUTPUT 1 -d 169.46.82.160/27 -j ACCEPT

…then generate another test message. If this works, make sure to save these firewall rules for the next reboot.

When using IP subnet-based egress firewall rules, please sign up for Papertrail’s log destination changes mailing list. This is only necessary if your network configuration requires you to know about every new or deprecated log destination subnet. It’s not necessary if you use a current IP as your log destination only because the sender doesn’t support sending to a domain. Papertrail tracks inbound messages and will email account contacts before removing any destinations, so you’ll always hear from us if you need to change an existing IP.

SELinux

If your system enforces SELinux policies and the destination port is not 514, you may need to add the port to the set of ports which syslogd can open network connections to.

To confirm that SELinux is the cause, execute strace -tt -s 500 -fp PID -o strace.log (replace PID with the pid of your logging software), and then grep for EACCESS:

$ grep EACCESS strace.log

Output similar to the following will indicate that a policy change is required:

connect(6, {sa_family=AF_INET, sin_port=htons(XXXX), sin_addr=inet_addr("67.214.X.X")}, 16) = -1 EACCES (Permission denied)

To permit access, on RHEL6:

$ yum install policycoreutils-python
$ semanage port -a -t syslogd_port_t -p tcp 12345

To see current port settings:

$ semanage port --list

Juniper/Netscreen ScreenOS

To allow log traffic to pass through the firewall, configure it via the Web interface as follows:

Creating a Custom Service

Navigate to Policy > Policy Elements > Services > Custom
Set the service name to Papertrail
Set the Service Timeout to Use protocol default
Select either UDP or TCP depending on how you wish to log
Set the low Source Port to 0 and the high Source Port to 65535
Set the low and high Destination Port to the port of the log destination that you wish to send log data to
Click Ok to save the changes

Applying the Service

Navigate to Policy > Policies
Select Trust from the From dropdown and Untrust from the To dropdown
Click New at the top right to create a new policy
Under Name specify Papertrail
Set the Source Address & Destination Address to Any from the drop down
Click the Multiple button and select the Papertrail service from the list
Set the Action to Permit
Click Ok to save the policy

Idle timeouts

Papertrail automatically closes idle TCP connections after 15 minutes of inactivity. Log senders should already handle this as part of a full implementation, since connections are routinely affected by Internet reachability changes. Senders can elect to immediately reconnect, or leave the connection closed until new log messages are available.

In addition to Papertrail’s timeout, Google Compute Engine and Microsoft Azure also have their own.

Rate limits

The following rate limits apply for log messages transmitted to Papertrail:

For UDP:

3,000 messages per second per source IP
10,000 messages per second to a log destination (port)

For TCP:

10 new connections per second per source IP to a log destination (port)

For more information on why they were introduced, see Introducing Syslog Rate Limits.

Hitting these limits? Contact us. Please include the logger name and associated configuration file.

Port reaping

If a free Papertrail account hasn’t seen any new events for one year, then any assigned destination ports may be removed for future reassignment. For this reason, it is important to fully remove Papertrail from any system before placing it into hibernation, if there is any possibility of the system being reanimated after a year or longer. (Otherwise, it may inadvertently send logs to a different Papertrail account.)

This policy allows Papertrail to ensure resources are used efficiently, while the one-year grace period ensures ports aren’t reassigned unless an account is truly inactive. However, if you have concerns about the implementation of this policy, contact us.

GNU syslogd

Test

Until recently, a few distributions (CentOS, Gentoo) provided GNU syslogd rather than syslog-ng or rsyslog. GNU syslogd only supports logging to the default syslog port (514), not user-selected destination ports. To see whether you run GNU syslogd, check:

Did you edit /etc/syslog.conf (rather than another filename)?
Run: syslogd -v. Does it say syslogd 1.4.1 (or another version)?

If so, this may be the problem.

Workarounds

Papertrail has a few workarounds for GNU syslogd. Choose one:

Tell Papertrail your system’s IP address

Visit Add Systems and select “I’d like to aggregate system/OS logs from something else…“.
Under Less common setup methods below the search area, click My syslog daemon only sends to port 514.
Provide Papertrail with this device’s Internet-facing public (external NAT) IP.

Use the standalone `remote_syslog2` log collector

remote_syslog2 can send both app and system log files. It doesn’t modify the system-wide logging configuration at all and Papertrail does not need the IP addresses of your servers. Setup instructions.

Upgrade to syslog-ng or rsyslog

These are better loggers by any measure. CentOS and Gentoo were the last distributions to switch.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.