Constrain searches with attributes (like severity)

As of today, Papertrail has a new search operator: an attribute. Use an
attribute-specific keyword to examine only a specific part of a log
message.

Here’s a live example returning messages with the severities emergency, critical, or error.

Example

This search will search only the program/tag element of log messages,
rather than the whole message.

program:sshd

These attributes are available: severity, facility, sender, program, and message.

You’re probably already familiar with AND (Papertrail’s default), OR, negation (-), and parenthesis. Attribute constraints can be used in the same searches as other constraints, like this:

'something bad' program:ssh -noise

Why are these useful?

The program, sender, and message attributes permit more granular searches against data which Papertrail already examined. The severity
and facility attributes allow new portions of log messages to be
examined.

We added these to:

• Build more powerful searches. program:(apache OR sshd) severity:crit is useful on its face.
• Incorporate fields which weren’t previously searchable. Until today,
  the facility and severity were only included in archive exports and
  API responses. They’re now searchable.
• Permit faster searches. Papertrail is subject to the laws of physics, as much as we try to optimize around them. When you know what you want and can codify it, Papertrail can examine less data and return results faster.
• Eliminate false positives. A search for sshd finds all logs containing the string sshd anywhere, whether the message was sent from the sshd program or the body simply contained sshd. Use attributes to search for either specific case.

Play it out

In designing the grammar for attributes, we tried to maintain our starting point that “it just works.” A user would rightly expect this search to work, and it does:

('something bad' program:ssh -noise)
OR severity:error

We took that further: values can be nested to reduce typing, and without reducing the readability of the search. For example:

('something bad' program:ssh -noise)
OR program:(raid5tools ethtool)

Negation (-) works uniformly:

-('something bad' program:ssh)
OR -program:(raid5tools ethtool)
OR -noise

Grammar police

We debated how to handle multiple constraints within an attribute which
can have only one. In Papertrail, the search a b means a AND b. That’s logical because AND is possible.

However, that doesn’t make sense for attributes which a log message can only have one of. program:(abc AND def) is almost never true. (The
“almost” is if abc and def are different portions of a single program value, a search which would not intentionally be performed.)

4 of the 5 attributes – all except message – can only have a single value per message, so AND is never relevant. Because of this, we chose to default all attributes to OR. program:(a b) meansprogram:(a OR b).

Saving typing

Portions of severity or facility names work fine too. For example, this
works:

severity:(crit emerg)

No need to type all of critical and emergency. One argument can
match multiple values, too. For example, this will match logs with any
facility containing local (like local0 or local7):

facility:local

All operators, and all searches, work with all Papertrail Search methods (Web, command-line, API, and inbound links). Finally, a simple query like sshd still works just like it did: Papertrail searches the message itself, sender, and program.

Finally, since senders often represent different things, the attributeshost, system, and source all work as aliases for sender. Use the one(s) which make sense to you.

Suggestions?

We hope these operators allow more granular searches. As a fringe benefit, because your search is more tightly constraining, Papertrail is able to search less data and can often return results faster.

If you try these and have ideas, please send them over. Enjoy!