Skip to content
Velloreth
Technology 5 min read

Monitoring vs Observability: What the Difference Is

Monitoring checks a known signal, and observability answers a new question. Learn the SLI, the alert, and the cardinality.

By

The distinction

MonitoringObservability
Questionis a known thing broken?what is happening?
Signala known metricany signal
Changea new dashboarda new query
Scopea subsetthe whole

Monitoring is a subset of observability.

The golden signals

SignalMeaning
Latencyhow long a request takes
Traffichow many requests
Errorshow many fail
Saturationhow full a resource is

They cover the user experience and the resource pressure.

A starting point, not the whole set.

The SLI and the SLO

An SLI is a measured signal.

The fraction of requests that succeed
The fraction of requests faster than 300ms

An SLO is the target.

99.9 percent over 30 days

An error budget is one minus the SLO.

0.1 percent of 30 days = about 43 minutes

The budget decides when to stop shipping a feature and fix the reliability.

The alert

Alert on a symptom, not on a cause.

BadGood
CPU over 80%the error rate over 1%
Memory over 90%the p99 latency over 1s
A disk over 80%the requests failing

A cause is a guess, and a symptom is what the user experiences.

A noisy alert trains the team to ignore it, which is worse than no alert.

A multi-window alert reduces the noise.

The error rate over 5% for 2 minutes
AND over 1% for 1 hour

The cardinality

The number of distinct values a label can take.

A high-cardinality label explodes the metric series.

A user id belongs in a log or a trace, not in a metric label.

A high-cardinality signal is what makes a new question possible, which is the observability part.

The workflow

1. A metric alerts.

The p99 latency is over 1s

2. A trace locates.

The slow span is POST /payments

3. A log explains.

The payment failed with code card_declined

That sequence is the investigation, and each signal has a role.

The dashboards

A dashboard is for a human, and it should answer a question.

A dashboard with fifty panels answers nothing.

A dashboard per service with the golden signals is a good start.

A dashboard that is not looked at is a cost.

The runbook

A runbook is the steps for a known alert.

Alert: the error rate is over 1%
1. Check the deploy history
2. Check the dependency status
3. Roll back if a deploy is the suspect
4. Escalate if the cause is unknown

It reduces the time to mitigate, which is the metric that matters.

A runbook that is not updated is a trap.

The tools

SignalTools
MetricPrometheus, CloudWatch, Datadog
LogLoki, Elasticsearch
TraceJaeger, Tempo
All threeOpenTelemetry

OpenTelemetry is the vendor-neutral standard.

The maturity model

A team moves through a few stages, and knowing the stage explains which practice to adopt next. The first stage is a manual check: someone looks at a dashboard when a user complains, and the time to detect is measured in hours. The second is an alert on a cause, such as a high CPU, which fires often and is often wrong, so the team learns to ignore it. The third is an alert on a symptom with an SLO, which fires when the user is affected and is rare enough to be trusted. The fourth is a trace and a high-cardinality log, which turns an investigation from a guess into a query. The fifth is a review of the incidents and the alerts, which removes the alert that nobody acts on and adds the one that was missing. A team that skips to the fourth stage without the third has the tooling and not the discipline, and the alerts are still ignored.

The cost of observability

The signals are not free, and the cost is the reason a team has to choose what to keep. A metric is cheap because it is aggregated, and the cost is driven by the number of series, which is why a high-cardinality label is expensive. A log is the usual cost surprise, because every event is stored, and the volume grows with the traffic and the level. A trace is in between, and the sampling controls it. The practical controls are a log level that is right for production, a retention that matches the investigation window, a sampling rate for the traces, and a review of the metric labels for a high cardinality. A team that measures the cost per signal can afford the signals that matter, and a team that does not eventually turns off the one it needs.

Common mistakes

An alert on a cause. A false alarm.

A noisy alert. An ignored alert.

No SLO. No target.

No error budget. No decision.

A high-cardinality metric label. An exploded series.

A dashboard with no question. A cost.

No runbook. A slow mitigation.

No trace. A slow request is a mystery.

A metric with no alert. A silent failure.

The three signals in three tools with no link. A slow investigation.

A quotable summary

Monitoring checks a known signal against a threshold, and observability is the ability to ask a new question of the system without changing it, so monitoring is a subset of observability. An SLI is a measured signal and an SLO is the target, with the error budget as the allowed failure, and an alert should fire on a symptom rather than a cause. A high-cardinality signal is what makes a new question possible, and a metric alerts, a trace locates, and a log explains.

For the related topics, see metrics-logs-and-traces and incident-postmortems. For the broader context, logging-levels-guide and load-balancing-algorithms.

Frequently asked questions

What is the difference between monitoring and observability?
Monitoring checks a known set of signals against a threshold, so it answers whether a known thing is broken. Observability is the ability to ask a new question of the system without changing it, which is what a high-cardinality log and a trace provide. Monitoring is a subset of observability.
What is an SLI and an SLO?
An SLI is a service level indicator, a measured signal such as the fraction of requests that succeed. An SLO is a service level objective, the target for that signal, such as 99.9 percent over 30 days. An error budget is the allowed failure, which is one minus the SLO.
Why alert on a symptom rather than a cause?
Because a cause is a guess about what will break, and a symptom is what the user experiences. An alert on a high CPU may fire when the service is fine, and an alert on a high error rate fires when the user is affected. Alert on the symptom, and use the cause to investigate.
What is a golden signal?
One of four signals that describe a service: latency, traffic, errors and saturation. They are a starting point for the metrics to collect, because they cover the user experience and the resource pressure.
What is cardinality and why does it matter?
Cardinality is the number of distinct values a label can take. A high-cardinality label such as a user id explodes the number of metric series, which is a cost and a performance problem. A high-cardinality value belongs in a log or a trace, which is what makes a new question possible.
What is an alert that is too noisy?
An alert that fires when nothing is wrong, which trains the team to ignore it. The fix is to alert on a symptom with a threshold that matters, to group the related alerts, and to remove an alert that nobody acts on. A noisy alert is worse than no alert.