Administrator Guide
- Documentation for Papertrail
Collect Logs
Collect Logs: Apps & Services
Collect Logs: Hosting Services
Collect Logs: Integrations
- Fluentd
- Librato
- Amazon CloudWatch
- StatHat
- Amazon SNS
- HostedGraphite
- Datadog
- Zapier
- VictorOps
- New Relic traces
- Slack
- Campfire
- PagerDuty
- New Relic Insights
- Pushover
- AppOptics
- Geckoboard
- OpsGenie
- Honeybadger
Manage Logs
Send Logs for Analytics
SolarWinds Users & Orgs
Support and Security
- Security and Compliance
- Support Policy
What's New
- Papertrail Release Notes

Unix and BSD text log files (remote_syslog2)

Configure centralized logging from text log files in Unix with remote_syslog2. Use remote_syslog2 to aggregate logs from any text file, like app log files, in Unix and BSD.

Why remote_syslog2

To send log files to Papertrail from applications and daemons that don’t support syslog, run Papertrail’s tiny standalone remote_syslog2 daemon. It tracks one or more log files and sends new entries to Papertrail in realtime.

remote_syslog2 works for any text log files, has no impact to or reliance on the system syslog daemon or its logging configuration, is easy to set up, and forwards logs directly to Papertrail. No adjustments to syslog.conf, rsyslog.conf, or syslog-ng.conf are required.

Some applications that typically log to text files, such as MySQL, Apache, and Tomcat (log4j), have internal support for logging directly to syslog. For apps like these, either their internal syslog or remote_syslog2 can be used to send logs to Papertrail.

Setup

Automated (Configuration Management)

remote_syslog2 can be deployed and configured with configuration management tools. Papertrail provides official support for Chef, Puppet, and Salt:

These configuration management modules only configure and deploy remote_syslog2. To configure and manage the system’s syslog daemon configuration, see Configuring remote syslog from Unix/Linux and BSD/macOS.

Manual

Install remote_syslog2

Download the current release. To extract it and copy the binary into a system path, run:

$ tar xzf ./remote_syslog*.tar.gz
$ cd remote_syslog
$ sudo cp ./remote_syslog /usr/local/bin

RPM and Debian packages are also available.

Configure

Paths to log file(s) can be specified on the command-line, or save log_files.yml.example as /etc/log_files.yml. Edit it to define:

the path to this app’s log file, and any other log file(s) to watch.
the destination host and port provided under log destinations. If no destination port was provided, set host to logs.papertrailapp.com and remove the port config line to use the default port (514).

Start

Start the daemon:

$ sudo remote_syslog

Logs should appear in Papertrail within a few seconds of being written to the on-disk log file. Problem? See Troubleshooting.

remote_syslog requires read permission on the log files it is monitoring.

Auto-start

remote_syslog2 can be automated to start at boot using init scripts (examples) or your preferred daemon invocation method, such as monit or god. See remote_syslog --help or the full README on GitHub.

Troubleshooting

remote_syslog2 is daemonized as remote_syslog. When you’re looking for the process, look for remote_syslog (not remote_syslog2).

The system rebooted and `remote_syslog` didn’t start

Install an init file.

Logs not appearing?

Please feel free to contact us, either instead of following these instructions or while trying them. We enjoy helping, and these steps are only here if they let you save time by troubleshooting independently.

There are a few reasons remote_syslog might not be sending logs.

Is `remote_syslog` running?

Check with ps auxww | grep [r]emote_syslog. Exactly one process should be shown, like this:

root     24501    0.0 0.4    13952 8864 ? S   Mar01 2:45 remote_syslog

Does it have the correct access permissions?

In the example above, the process is running as user root. If a user other than root is shown, does that user running remote_syslog have permission to write to /var/run/? /var/run/ is the default location for PID files (background).

If remote_syslog isn’t already running as root, try running it as root to test (such as with sudo), or specify an alternate location for the PID file (such as with remote_syslog --pid-file=/tmp/some.pid).

Did it stop running?

If remote_syslog starts but subsequently stops running, try leaving it attached to the terminal (rather than daemonizing) using remote_syslog -D.

Is it able to connect and find the log files?

If remote_syslog is running, try stopping it and restarting it manually in the foreground, logging at DEBUG level, with:

$ remote_syslog -D --log="=DEBUG"

After it starts, it’ll print detailed logs about its activity to the terminal. If it’s unable to connect, check Troubleshooting reachability. If it’s unable to forward one or more of the log files, (for example, if it prints Cannot forward filename.log, it may not exist), check the filenames and permissions.

After the initial message Forwarding file: filename.log, it’s normal for remote_syslog to print Skipping filename.log because it is already running on subsequent checks of the currently watched files.

Are the log files being updated?

remote_syslog only reads new additions to the log files after it was started. If there’s no new activity, no logs will appear in Papertrail. To check whether a file is active, run:

$ tail -Fn0 filename.log

to watch for new lines coming in; this is very similar to what remote_syslog2 does. To verify that new lines would be sent once added, try:

$ echo "Papertrail test line" >> filename.log

If that line appears, any new lines added to the log will also appear in Papertrail.

Is it running, but suddenly stopped sending data?

…perhaps across many systems at the same time? This could be a firewall policy change or log rotation. However, remote_syslog plays well with all common log rotation systems (with no changes to the configuration of either), so we’d like to hear about this problem.

Is `remote_syslog` monitoring log files that are stored on an NFS share?

If so, enable polling via the --poll switch.

Detailed troubleshooting

The next step is to see what’s actually being sent (with tcpdump), or what remote_syslog is actually doing, typically with strace:

$ strace -tt -s 500 -fp 12345

…where 12345 is the process ID of remote_syslog, obtained from the second column of ps auxww. This will output every call that it makes. We suggest sending the strace output to a file:

$ strace -tt -s 500 -fp 12345 -o strace.log

Feel free to send the strace.log file to us via Contact us. We’ll look for a few things. Did the OS notify remote_syslog of writes to the log files? Did remote_syslog call sendto() (UDP) or connect() and send() (TCP) to try sending the message? What happened?

Extra

See other options with:

$ remote_syslog --help

Question?

Can we help? Contact us.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.