Metrics, Logs and Traces: The Three Pillars Explained
Observability rests on three signals. Learn what each answers, the cardinality, the sampling, and how they combine.
By Velloreth
The three signals
| Signal | Records | Answers | Cost |
|---|---|---|---|
| Metric | a number over time | how many, how fast | low |
| Log | an event | what happened | high |
| Trace | a request path | where the time went | medium |
They combine: a metric alerts, a trace locates, a log explains.
The metric
A numeric measurement over time.
http_requests_total{status="200"} 12345
http_request_duration_seconds_bucket{le="0.1"} 900
The types:
| Type | Meaning |
|---|---|
| Counter | only increases |
| Gauge | goes up and down |
| Histogram | a distribution |
| Summary | a quantile |
A counter is for a request count.
A gauge is for a queue depth.
A histogram is for a latency, and it gives a percentile.
The cardinality
The number of distinct values a label can take.
| Label | Cardinality |
|---|---|
status | low |
method | low |
path | medium |
user_id | high |
request_id | very high |
A high-cardinality label explodes the series count, which is a cost and a performance problem.
A user id belongs in a log or a trace, not in a metric label.
A path with an id in it should be templated, so /users/123 becomes /users/:id.
The log
A timestamped event with detail.
{ "level": "ERROR", "message": "payment failed", "order_id": "abc", "code": "card_declined" }
It is expensive to store and slower to query.
It answers what happened, with the detail.
A structured log is queryable, and a free-form string is not.
The trace
The path of one request across the services.
GET /checkout [================] 420ms
POST /payments [======] 180ms
GET /inventory [==] 40ms
POST /orders [========] 200ms
Each span is one operation with a start, a duration and a parent.
It shows where the time went, which is what a metric cannot.
A trace id is propagated in a header, so the services join the spans.
The sampling
Storing every trace is expensive.
| Strategy | Decides | Keeps |
|---|---|---|
| Head-based | at the start | a random fraction |
| Tail-based | after the trace | the slow and the failed |
A tail-based sample is more useful, because it keeps the interesting traces.
A head-based sample is simpler and it may miss the slow one.
A consistent sample keeps the whole trace, not a part of it.
The combination
A metric alerts.
The p99 latency is over 1s for 5 minutes
A trace locates.
The slow span is POST /payments
A log explains.
The payment failed with code card_declined
That sequence is the workflow, and each signal has a role.
The tools
| Signal | Tools |
|---|---|
| Metric | Prometheus, CloudWatch, Datadog |
| Log | Loki, Elasticsearch, CloudWatch Logs |
| Trace | Jaeger, Tempo, X-Ray |
| All three | OpenTelemetry |
OpenTelemetry is the vendor-neutral standard, and it is the common choice.
The cost
A metric is cheap, because it is aggregated.
A log is expensive, because every event is stored.
A trace is in between, and the sampling controls the cost.
A high-volume log is the usual cost surprise, so the level and the retention matter.
Common mistakes
A high-cardinality metric label. An exploded series.
A path with an id in a metric. The same problem.
No trace. A slow request is a mystery.
No sampling. An unbounded trace cost.
A free-form log. A hard query.
No correlation id. A hard trace.
A metric with no alert. A silent failure.
An alert on a cause rather than a symptom. A missed incident.
No retention policy. An unbounded cost.
The three signals in three tools with no link. A slow investigation.
A quotable summary
A metric is a number over time, a log is a timestamped event, and a trace is the path of one request across the services. A metric is cheap and aggregated and answers how many or how fast, a log is detailed and expensive and answers what happened, and a trace shows where the time went. A high-cardinality value belongs in a log or a trace rather than a metric label, and the three combine: a metric alerts, a trace locates, a log explains.
Related reading
For the related topics, see monitoring-vs-observability and logging-levels-guide. For the broader context, incident-postmortems and sre-basics.