Administrator Guide
- Documentation for Papertrail
Collect Logs
Collect Logs: Apps & Services
Collect Logs: Hosting Services
Collect Logs: Integrations
- Fluentd
- Librato
- Amazon CloudWatch
- StatHat
- Amazon SNS
- HostedGraphite
- Datadog
- Zapier
- VictorOps
- New Relic traces
- Slack
- Campfire
- PagerDuty
- New Relic Insights
- Pushover
- AppOptics
- Geckoboard
- OpsGenie
- Honeybadger
Manage Logs
Send Logs for Analytics
SolarWinds Users & Orgs
Support and Security
- Security and Compliance
- Support Policy
What's New
- Papertrail Release Notes

Search syntax

If you’ve used Google search, you’re familiar with 95% of Papertrail’s search syntax. Type what you have, whether it’s an IP address, an error string, the name of a program or system, a user’s session, or fragments of any of those things.

From there, build up sets of what you do and don’t want to see. Here’s a few examples for the Ruby on Rails framework to give you an idea.

Background

This came from our experience using Papertrail for our logs, from watching early users try different syntaxes, and most importantly, from seeing what we and others actually wanted to find. By far the most common type of search was a set of desired program names or message strings (important things) combined with a few exclusions (noise).

This Boolean syntax makes queries like that fast to compose and easy to read/update.

Check out Linking to logs to quickly integrate Papertrail searches into your app’s internal admin interface.

Example searches

These 6 searches demonstrate most of the syntax. The sender name, program/file name, and message are all searchable:

"Accepted password" sudo
-CROND
(www42 some.log) OR (nyc-db2 other.log)
(message-id postfix/smtpd) OR -google.com
10.1.2 -("TTL exceeded" OR "1 packet")

Searches can also be constrained by an attribute, so Papertrail considers only that field:

program:(production.log staging.log) '"action"=>"index"' -("count(*)" OR abc)

For particulars on searching JSON objects please refer to the JSON Search Syntax article.

Quick reference

While in the event viewer, click the question mark icon on the right side of the search box to see these examples without leaving the viewer:

Tips

To get the narrowest set of results as fast as possible:

Create groups that represent different sets of systems, then search those groups (instead of searching an all-encompassing group). Aside from making search faster, properly-scoped groups make the Dashboard easier to interpret, since searches are tied to the group they’re actually related to rather than being lumped together. As an example, Production Web servers may be far fewer logs than Production servers, and when a search is specific to Web logs, it probably logically belongs there.
Use attributes when searching only specific parts of a log message. For example, abc sender:www42 will be faster than abc www42.
Provide (or link with) a time/date or rough estimate. Papertrail will start searching from the time you seek to in the event viewer. Seek to a time at the end of or after the incident so that the search starts there.
Integrate contextual “Show related logs” links from your app’s admin dashboard. Your dashboard can generate links to logs with timestamps (such as a timestamp from a user’s last login or when a background job completed) and search queries. The additional search keywords (beyond what you would manually type) may return fewer unrelated matches and it saves time entering a search.

Search Syntax

Papertrail supports combining terms and phrases using AND, OR, and exclusion operators, as well as order of operations (parentheses). The message text, program/file name, and system name (as shown in Papertrail’s dashboard) are searched.

AND

By default, terms in the same search are automatically ANDed. This search:

something somethingelse

is equivalent to this search:

something AND somethingelse

OR

To match one or more of multiple terms, use the OR keyword. For example:

something OR somethingelse

Note: OR must be capitalized.

Phrases

Enclose phrases in single- or double-quotes. For example:

something OR "something else"

Single- and double-quotes are interchangeable, as long as the same type of quote is used to start and end the same phrase. To search for a phrase that contains a quote, use the other type of quote. For example, to match this punctuation-filled phrase:

{"action"=>"index",

enclose it in single-quotes, like this:

'{"action"=>"index",'

Backslashes

Backslashes can be used to escape single- and double-quotes within words or phrases. Therefore, to find a literal backslash, it must be doubled up, as in C:\ or “C:\“.

Order of operations

To indicate order of operations for matching, use parentheses. For example:

something ("something else" OR "a third thing")

says to match either of the 2 terms inside the parentheses, then AND it with the term outside the parentheses.

Contrast that search with the same search without parentheses:

something "something else" OR "a third thing"

which is ambiguous about whether the OR applies to just the middle term, or to both of the first 2 terms. (Without parentheses, terms are matched left to right).

AND and OR can be freely used inside and outside of parentheses.

Exclusion (negation)

Prepend - to a term, quoted phrase, or parenthesized set. For example, to show all logs except a certain word or quoted phrase:

-"exclude this"

Exclude all matches in a parenthesized set:

something -("but not" OR "something else")

Exclusions can be used anywhere in a query. For example:

something -("but not" OR "something else") "but this too"

Case sensitivity

Searches are case insensitive. This query:

Completed in

is equivalent to this query:

completed in

Attributes

Papertrail supports 5 attributes: sender, program, message, severity, and facility. All are used in key:value fashion. An example search that constrains results to those from the ssh program containing “something bad” but not “noise”, or those with an error severity, is:

("something bad" program:ssh -noise) OR severity:error

It’s possible to include more than one valid value for an attribute. Messages matching either one are returned (OR):

program:(raid5tools ethtool)

Severity and facility attributes are those from the syslog specification. If severity is a part of the displayed log message, simply use a text search like error or INFO to find those messages.

Use attributes:

to constrain matching to only a single field, either to eliminate false positives or search less data (and increase speed)
to search facility or severity, which are not normally examined

Attributes and attribute-less constraints can be used together, and can be combined with negation.

All attributes except message can only have a single value per message, so AND is never relevant. Because of this, all attributes default to OR, as in the program: example above. program:(a b) means program:(a OR b).

Additionally, any of the attribute names host, source, or system can be used interchangeably with sender.

facility and severity support substring matching, so this matches any facility value beginning with local:

facility:local

As a result, they also support shorthand, like this. No need to type the full severity values:

severity:(crit emerg)

For more examples, see Papertrail’s attribute announcement.

Punctuation

Other than quotes and backslashes, no special adaptations should be needed to use punctuation in a term or phrase. For example, to see all occurrences of IP addresses within 10.20.30.0 to 10.20.30.255, use this query:

10.20.30.

The periods are interpreted as part of the string. Other punctuation works the same way, and no adaptation or escaping is needed.

Similarly, the entire log message is considered for matching. The search:

bcd

will match lines containing any of these:

abcde
123bcd456
456bcd
456abcde

Time/Date Search

To search for a time or date, click the clock icon to the right of the search box and enter a time, date, or combination. The time or date can be in almost any format, and will be parsed and displayed in realtime. Here’s a screenshot:

Timezones

When searching by time in the Web interface, the searched-for timestamp is interpreted in the timezone set in the user profile.

Papertrail tracks the time a log message was received, and automatically displays it in the profile timezone. Users can have different timezones within the same organization, so there’s less need to perform mental timezone conversions.

Sender name and program name

While sender name and program name/filename can be used in searches, they aren’t part of the phrase that Papertrail matches against. For example, this log message:

www42 httpd: GET / HTTP/1.0

would be matched by any of these queries:

www42 GET
httpd GET
www42 httpd GET
www42 httpd "GET /"
www42 OR "GET /"

However, that message would not be matched by this phrase query because the program name and separating colon are not included in the phrase:

"httpd: GET"

Linking to Searches

This document has moved to Linking to logs.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.