Administrator Guide
- Documentation for Papertrail
Collect Logs
Collect Logs: Apps & Services
Collect Logs: Hosting Services
Collect Logs: Integrations
- Fluentd
- Librato
- Amazon CloudWatch
- StatHat
- Amazon SNS
- HostedGraphite
- Datadog
- Zapier
- VictorOps
- New Relic traces
- Slack
- Campfire
- PagerDuty
- New Relic Insights
- Pushover
- AppOptics
- Geckoboard
- OpsGenie
- Honeybadger
Manage Logs
Send Logs for Analytics
SolarWinds Users & Orgs
Support and Security
- Security and Compliance
- Support Policy
What's New
- Papertrail Release Notes

Web Hooks

React to log events: when Papertrail receives new log events that match one of your saved searches, it can notify services like Campfire, Librato, or a HTTP URL that you provide (webhook).

Sample app

Here’s the source to a working example webhook handler on GitHub. This tiny Sinatra app accepts a webhook request from Papertrail containing log matches. It then submits them to a remote Web service, Prowl.

Webhooks operated by Papertrail

Papertrail operates a set of webhooks for common services like Slack and email. These are available in Papertrail’s Web interface. From the Dashboard, click the pencil icon on the saved search, then click New Alert.

Papertrail hosts these alert processors.

You can also write your own standalone webhook handler. The rest of this page covers how to do that.

Example use

You have a monitoring system running in your datacenter or on another hosted service. New log events with the string segfault are an important failure case, and they should be sent to the monitoring system so it can page administrators.

Papertrail will notify your monitoring system every minute that new matches are found, and will also provide the contents of the new log matches.

Papertrail can notify every hour or day for less-urgent message types, as for generating summaries.

Setup

In Events, search for the relevant query (in the example use above, “segfault”)
Click Save Search and give the search a name.
Select Save & Setup an Alert.
Select Webhook as the alert type.
Define your webhook URL and callback frequency (minute, hour, or day).

If the webhook only needs a count of matching logs, not all raw logs, enable the Send only counts option. See Count-only webhooks.

Callback

The callback is a POST request in the format used by the github-services suite of webhook handlers. The POST body contains a single parameter called payload, which contains a JSON hash.

Typically, the most important key in this hash is events: an array of hashes of log events (one hash per event).

Example payload

This is a POST body containing 2 events:

payload = {
  "events": [
    {
      "id": 7711561783320576,
      "received_at": "2011-05-18T20:30:02-07:00",
      "display_received_at": "May 18 20:30:02",
      "source_ip": "208.75.57.121",
      "source_name": "abc",
      "source_id": 2,
      "hostname": "abc",
      "program": "CROND",
      "severity": "Info",
      "facility": "Cron",
      "message": "message body"
    },
    {
      "id": 7711562567655424,
      "received_at": "2011-05-18T20:30:02-07:00",
      "display_received_at": "May 18 20:30:02",
      "source_ip": "208.75.57.120",
      "source_name": "server1",
      "source_id": 19,
      "hostname": "def",
      "program": "CROND",
      "severity": "Info",
      "facility": "Cron",
      "message": "A short event"
    }
  ],
  "saved_search": {
    "id": 42,
    "name": "Important stuff",
    "query": "cron OR server1",
    "html_edit_url": "https://papertrailapp.com/searches/42/edit",
    "html_search_url": "https://papertrailapp.com/searches/42"
  },
  "max_id": 7711582041804800,
  "min_id": 7711561783320576,
  "frequency": "1 minute"
}

For readability, the example above is not URL-encoded. See Encoding below. Also, whitespace and line breaks have been added.

The following hash keys are defined for each log event:

id
received_at
display_received_at
source_name
source_ip
source_id
facility
severity
program
message
hostname

HTTP and HTTPS URLs are supported.

To learn more about the meaning of each key and the data types used, see the descriptions in Search API: Event keys.

Encoding

The payload JSON hash is encoded with application/x-www-form-urlencoded, as is standard for all POST requests. Apps rarely need to consider this because almost all Web frameworks automatically URL-decode the POST contents. In those cases, the app only needs to decode JSON.

Here’s a very minimal example of how the request is generated. This uses the Ruby irb console and the payload event only contains a hostname, but it shows the format of the URL-encoded POST:

>> payload = { :events => [ { :hostname => 'abc' } ] }
=> {:events=>[{:hostname=>"abc"}]}
>> puts payload.to_json
{"events":[{"hostname":"abc"}]}
>> puts URI.escape(payload.to_json)
%7B%22events%22:[%7B%22hostname%22:%22abc%22%7D]%7D

Parsing

In Ruby with the Ruby on Rails framework, this line will turn the callback POST params into a Ruby hash called payload:

payload = Yajl::Parser.parse(params[:payload])

Implementation considerations

Up to 25,000 events are included in a single callback. That was chosen as a compromise based on:

the practical payload of a single HTTP POST over the Internet (especially one that could happen every minute)
the number of events that a typical receiving app might be equipped to handle in an unexpected case
trying to support apps that do things we didn’t expect or plan for

The attribute reached_record_limit is set in the top-level hash when the 25,000 limit is reached. If you need more, just ask and we’ll try to make it work.

Testing

To feed real log data from your Papertrail account into a local development environment, use a service such as:

UltraHook

Alternatively, you can host your code in the cloud. A list of providers can be found here.

Count-only webhooks

Most webhooks act on the message body or other parts of the log message. However, for webhooks which only need the number of matching events, or do not need any information at all (beyond that the alert fired), enable the “Count-only webhooks” option.

Since a webhook invocation may cover thousands of matching messages, sending only counts can turn a 5 MB JSON payload into only a few KB.

When this option is enabled, the example is modified. No events hash key is included. Instead, a counts hash key contains an array of hashes, one hash per sending system seen during the period.

Each sender hash has the sending system’s name and ID (more). It also has a timeseries key containing a count of messages received from that sender at each Unixtime timestamp. The key is a Unixtime timestamp (as a JSON integer). The value is the number of matching messages generated by the sender during that 1 second interval.

Here is an example payload for a count-only webhook invocation covering 16 total events. For example, the sender arthur sent 5 messages at timestamp 1453248895.

Other than the new counts and omitted events keys, the payload is identical to the provided example:

payload = {
  "counts": [
    {
      "source_name": "arthur",
      "source_id": 4,
      "timeseries": {
        "1453248892": 2,
        "1453248895": 5,
        "1453248905": 5,
        "1453248948": 1
      }
    },
    {
      "source_name": "ford",
      "source_id": 3,
      "timeseries": {
        "1453248927": 3
      }
    }
  ],
  "saved_search": {
    "id": 42,
    "name": "Important stuff",
    "query": "cron OR server1",
    "html_edit_url": "https://papertrailapp.com/searches/42/edit",
    "html_search_url": "https://papertrailapp.com/searches/42"
  },
  "max_id": 7711582041804800,
  "min_id": 7711561783320576,
  "frequency": "1 minute"
}

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.