Skip to content
Velloreth
Technology 4 min read

Metrics, Logs and Traces: The Three Pillars Explained

Observability rests on three signals. Learn what each answers, the cardinality, the sampling, and how they combine.

By

The three signals

SignalRecordsAnswersCost
Metrica number over timehow many, how fastlow
Logan eventwhat happenedhigh
Tracea request pathwhere the time wentmedium

They combine: a metric alerts, a trace locates, a log explains.

The metric

A numeric measurement over time.

http_requests_total{status="200"} 12345
http_request_duration_seconds_bucket{le="0.1"} 900

The types:

TypeMeaning
Counteronly increases
Gaugegoes up and down
Histograma distribution
Summarya quantile

A counter is for a request count.

A gauge is for a queue depth.

A histogram is for a latency, and it gives a percentile.

The cardinality

The number of distinct values a label can take.

LabelCardinality
statuslow
methodlow
pathmedium
user_idhigh
request_idvery high

A high-cardinality label explodes the series count, which is a cost and a performance problem.

A user id belongs in a log or a trace, not in a metric label.

A path with an id in it should be templated, so /users/123 becomes /users/:id.

The log

A timestamped event with detail.

{ "level": "ERROR", "message": "payment failed", "order_id": "abc", "code": "card_declined" }

It is expensive to store and slower to query.

It answers what happened, with the detail.

A structured log is queryable, and a free-form string is not.

The trace

The path of one request across the services.

GET /checkout                     [================] 420ms
  POST /payments                  [======]          180ms
  GET /inventory                  [==]               40ms
  POST /orders                    [========]        200ms

Each span is one operation with a start, a duration and a parent.

It shows where the time went, which is what a metric cannot.

A trace id is propagated in a header, so the services join the spans.

The sampling

Storing every trace is expensive.

StrategyDecidesKeeps
Head-basedat the starta random fraction
Tail-basedafter the tracethe slow and the failed

A tail-based sample is more useful, because it keeps the interesting traces.

A head-based sample is simpler and it may miss the slow one.

A consistent sample keeps the whole trace, not a part of it.

The combination

A metric alerts.

The p99 latency is over 1s for 5 minutes

A trace locates.

The slow span is POST /payments

A log explains.

The payment failed with code card_declined

That sequence is the workflow, and each signal has a role.

The tools

SignalTools
MetricPrometheus, CloudWatch, Datadog
LogLoki, Elasticsearch, CloudWatch Logs
TraceJaeger, Tempo, X-Ray
All threeOpenTelemetry

OpenTelemetry is the vendor-neutral standard, and it is the common choice.

The cost

A metric is cheap, because it is aggregated.

A log is expensive, because every event is stored.

A trace is in between, and the sampling controls the cost.

A high-volume log is the usual cost surprise, so the level and the retention matter.

Common mistakes

A high-cardinality metric label. An exploded series.

A path with an id in a metric. The same problem.

No trace. A slow request is a mystery.

No sampling. An unbounded trace cost.

A free-form log. A hard query.

No correlation id. A hard trace.

A metric with no alert. A silent failure.

An alert on a cause rather than a symptom. A missed incident.

No retention policy. An unbounded cost.

The three signals in three tools with no link. A slow investigation.

A quotable summary

A metric is a number over time, a log is a timestamped event, and a trace is the path of one request across the services. A metric is cheap and aggregated and answers how many or how fast, a log is detailed and expensive and answers what happened, and a trace shows where the time went. A high-cardinality value belongs in a log or a trace rather than a metric label, and the three combine: a metric alerts, a trace locates, a log explains.

For the related topics, see monitoring-vs-observability and logging-levels-guide. For the broader context, incident-postmortems and sre-basics.

Frequently asked questions

What are the three pillars of observability?
Metrics, logs and traces. A metric is a numeric measurement over time, a log is a timestamped event with detail, and a trace is the path of one request across the services. Each answers a different question, and together they let you understand a system without guessing.
What is the difference between a metric and a log?
A metric is an aggregated number, so it is cheap to store and fast to query, and it answers how many or how fast. A log is a detailed event, so it is expensive to store and slower to query, and it answers what happened. A metric alerts and a log explains.
What is a trace?
A record of one request as it passes through several services, made of spans. Each span is one operation with a start, a duration and a parent, so the trace shows where the time went. It is what turns a slow request into a specific slow call.
What is cardinality?
The number of distinct values a label can take. A label such as a status code has a low cardinality, and a label such as a user id has a high cardinality. A high-cardinality label explodes the number of metric series, so it belongs in a log or a trace instead.
What is sampling?
Recording only a fraction of the traces, because storing every trace is expensive. A head-based sample decides at the start, and a tail-based sample decides after the trace is complete, which lets it keep the slow and the failed traces. A tail-based sample is more useful and more complex.
What is the difference between monitoring and observability?
Monitoring checks a known set of signals against a threshold, so it answers whether a known thing is broken. Observability is the ability to ask a new question of the system without changing it, which is what a high-cardinality log and a trace provide. Monitoring is a subset of observability.