Knowing what your system is doing

A customer emails: "the site was slow yesterday afternoon."

You have logs. You grep for errors and find none, because it was not throwing errors, it was being slow. You have no idea whether it was slow for everyone or for one person, or which part was slow.

You cannot answer, and you cannot tell whether it is still happening.

Three different things

• Logs: what happened, one event at a time. Good for a specific request.
• Metrics: numbers over time. Good for "is this normal?"
• Traces: one request's journey through several services. Good for "where did the time go?"

Most teams have logs and think they have monitoring. Logs answer questions you already know how to ask.

Write logs a machine can read

Bad:  "Error processing order"
Good: {"event":"order_failed","order_id":"A-91","reason":"card_declined",
       "user_id":"u_22","duration_ms":812}

The second can be counted, grouped and alerted on. The first can only be read.

And put a request id on everything, passed between services, so one customer's problem can be followed end to end.

Measure percentiles, not averages

An average response time of 200 ms can hide that one request in twenty takes four seconds. The average is fine and those users are leaving.

Watch p95 and p99. They describe the experience of your least lucky users, which is where complaints come from.

Alert on symptoms

Alert on what a user would notice: error rate up, latency up, queue growing without limit. Not on CPU at 80%, which may be entirely healthy.

Every alert that fires and needs no action trains somebody to ignore the next one, and the next one is the real one.