Monitoring vs Observability: What the Difference Is
Monitoring checks a known signal, and observability answers a new question. Learn the SLI, the alert, and the cardinality.
By Velloreth
The distinction
| Monitoring | Observability | |
|---|---|---|
| Question | is a known thing broken? | what is happening? |
| Signal | a known metric | any signal |
| Change | a new dashboard | a new query |
| Scope | a subset | the whole |
Monitoring is a subset of observability.
The golden signals
| Signal | Meaning |
|---|---|
| Latency | how long a request takes |
| Traffic | how many requests |
| Errors | how many fail |
| Saturation | how full a resource is |
They cover the user experience and the resource pressure.
A starting point, not the whole set.
The SLI and the SLO
An SLI is a measured signal.
The fraction of requests that succeed
The fraction of requests faster than 300ms
An SLO is the target.
99.9 percent over 30 days
An error budget is one minus the SLO.
0.1 percent of 30 days = about 43 minutes
The budget decides when to stop shipping a feature and fix the reliability.
The alert
Alert on a symptom, not on a cause.
| Bad | Good |
|---|---|
| CPU over 80% | the error rate over 1% |
| Memory over 90% | the p99 latency over 1s |
| A disk over 80% | the requests failing |
A cause is a guess, and a symptom is what the user experiences.
A noisy alert trains the team to ignore it, which is worse than no alert.
A multi-window alert reduces the noise.
The error rate over 5% for 2 minutes
AND over 1% for 1 hour
The cardinality
The number of distinct values a label can take.
A high-cardinality label explodes the metric series.
A user id belongs in a log or a trace, not in a metric label.
A high-cardinality signal is what makes a new question possible, which is the observability part.
The workflow
1. A metric alerts.
The p99 latency is over 1s
2. A trace locates.
The slow span is POST /payments
3. A log explains.
The payment failed with code card_declined
That sequence is the investigation, and each signal has a role.
The dashboards
A dashboard is for a human, and it should answer a question.
A dashboard with fifty panels answers nothing.
A dashboard per service with the golden signals is a good start.
A dashboard that is not looked at is a cost.
The runbook
A runbook is the steps for a known alert.
Alert: the error rate is over 1%
1. Check the deploy history
2. Check the dependency status
3. Roll back if a deploy is the suspect
4. Escalate if the cause is unknown
It reduces the time to mitigate, which is the metric that matters.
A runbook that is not updated is a trap.
The tools
| Signal | Tools |
|---|---|
| Metric | Prometheus, CloudWatch, Datadog |
| Log | Loki, Elasticsearch |
| Trace | Jaeger, Tempo |
| All three | OpenTelemetry |
OpenTelemetry is the vendor-neutral standard.
The maturity model
A team moves through a few stages, and knowing the stage explains which practice to adopt next. The first stage is a manual check: someone looks at a dashboard when a user complains, and the time to detect is measured in hours. The second is an alert on a cause, such as a high CPU, which fires often and is often wrong, so the team learns to ignore it. The third is an alert on a symptom with an SLO, which fires when the user is affected and is rare enough to be trusted. The fourth is a trace and a high-cardinality log, which turns an investigation from a guess into a query. The fifth is a review of the incidents and the alerts, which removes the alert that nobody acts on and adds the one that was missing. A team that skips to the fourth stage without the third has the tooling and not the discipline, and the alerts are still ignored.
The cost of observability
The signals are not free, and the cost is the reason a team has to choose what to keep. A metric is cheap because it is aggregated, and the cost is driven by the number of series, which is why a high-cardinality label is expensive. A log is the usual cost surprise, because every event is stored, and the volume grows with the traffic and the level. A trace is in between, and the sampling controls it. The practical controls are a log level that is right for production, a retention that matches the investigation window, a sampling rate for the traces, and a review of the metric labels for a high cardinality. A team that measures the cost per signal can afford the signals that matter, and a team that does not eventually turns off the one it needs.
Common mistakes
An alert on a cause. A false alarm.
A noisy alert. An ignored alert.
No SLO. No target.
No error budget. No decision.
A high-cardinality metric label. An exploded series.
A dashboard with no question. A cost.
No runbook. A slow mitigation.
No trace. A slow request is a mystery.
A metric with no alert. A silent failure.
The three signals in three tools with no link. A slow investigation.
A quotable summary
Monitoring checks a known signal against a threshold, and observability is the ability to ask a new question of the system without changing it, so monitoring is a subset of observability. An SLI is a measured signal and an SLO is the target, with the error budget as the allowed failure, and an alert should fire on a symptom rather than a cause. A high-cardinality signal is what makes a new question possible, and a metric alerts, a trace locates, and a log explains.
Related reading
For the related topics, see metrics-logs-and-traces and incident-postmortems. For the broader context, logging-levels-guide and load-balancing-algorithms.