Almost everything we touch at Papertrail has a timestamp, which means it also has an expressed or implied time zone. Here’s more about why and how we work with time zones, then an introduction to new time zone-aware search alerts.
Who Cares?
Modern time zones came about in 1884, when a Canadian railroad engineer proposed a “universal day” based on a world meridian. At the International Meridian Conference, the Observatory of Greenwich was chosen as the meridian and zones were created every 15 degrees of longitude (proceedings).
Papertrail encounters a few issues that 19th century railroads did:
- Zones can change. The items we care about – log messages, users, servers – can all be mobile. Like trains and passengers, they can start, traverse, and end in different places.
- Reasonable people expect different things. Sysadmin Sally might have servers in 6 countries, set them to GMT, and convert to local time in her head. She probably expects something different than Developer Dan, who logs in once a week to troubleshoot a problem (and shouldn’t need to care about that conversion). Sally and Dan may be coworkers.
- Humans aren’t the only audience. Aside from operations, Papertrail needs to present the right zone and level of zone awareness to webhooks, permanent archives, and other targets.
- The status quo isn’t great. Log aggregation often transfers the burden on to the user, either by “punting” and treating timestamps as strings rather than times (so everyone has to do what Sally does) or by converting them to a single zone.
Our goal is to deliver a service that doesn’t require knowing any of that.
OK, How?
As much as possible, our starting point is the principle of least astonishment: Papertrail should just work, and then be tweakable where needed. We’ve spent effort to:
- Let servers and apps live anywhere and configured however the operator wants. Multiple data centers, cloud presences, and PaaS deployments are the norm. They often don’t – and should’t need to – use the same time zone.
- Deliver a sequential view of events across all infrastructure, without requiring reliable time sync. Troubleshooting interactions between 2 servers is very difficult when the events are out of order, and if Papertrail blindly trusted local timestamps, they often would be.
- Support distributed teams. Each engineer should be able to view messages in whatever zone is most logical to them. Same for timestamps provided by staff, like when seeking to a time.
- Present a consistent view of inconsistent logs. Some log messages contain explicit timestamps (ISO 8601), others have implicit zones (known only to the server), and some senders may generate logs in multiple zones concurrently (for example, OS, Web server, and app events). Manually tracking and normalizing these is a particularly demeaning use of humans.
- Make it easier for log consumers (search alerts, API clients) to ignore zones. Transfer the burden of localization on to Papertrail. More below.
Time Zone-Aware Log Alerts
As of March 29, 2012, search alerts can now receive event timestamps in any timestamp. Hit the Dashboard, find the saved search you have an alert for, mouseover it, and click “Edit.” Under “Manage Alerts,” choose the alert.
You’ll see a time zone dropdown. Choose the zone that Papertrail should convert the timestamps of matching messages to.
The Making Of Zone-Aware Alerts
Here’s the back story. We tried a few incarnations and found that even teams which are spread across time zones usually have a zone that the team thinks of as standard (what server clocks are set to, a particular office, UTC, etc.).
We also found that for emails, automatically changing timestamps to each recipient’s Papertrail account zone sounds ideal, but it’s actually not very intuitive because other than email, most alert methods inherently present a single time zone to many recipients. For example, a single chat room alert is seen by many viewers.
(There’s also an unwritten expectation with team-wide alerts that the whole team received the same thing. When a coworker pastes a log message in IM, converting it from their time zone to yours isn’t even on your radar.)
Also, treating email uniquely would mean that for the same issue, the same individual could receive a Campfire alert in one zone and an email in another. Besides violating the principle of least astonishment, that’s just dumb. There’s a better way: render each event’s timestamp as a link directly to that message in the Web viewer (where the time is localized to you). So Papertrail does that.
That’s why alert timestamps are set once for the alert. It helps that we’ve actually tried, and lived with, these decisions.
I hope you enjoyed this introduction to time zones in Papertrail. As always, if something isn’t working well, doesn’t seem obvious, or simply can be better, tell us.