Skip to content
Velloreth
Technology 5 min read

Canary Deployments: Releasing to a Small Slice First

Send a small share of traffic to the new version, watch the metrics, and roll forward or back. Learn the routing, the metrics and the traps.

By

What a canary is

A canary sends a small share of traffic to the new version.

95% -> v1 (stable)
 5% -> v2 (canary)

The team watches the metrics, then increases the share or rolls back.

The name comes from the canary in a coal mine, which gave an early warning.

Why it works

A test suite covers the paths the author thought of. A canary covers the paths real users take.

A canary catches:

  • A runtime error on a code path the tests missed
  • A latency regression from a slow query
  • A memory leak that appears under load
  • A configuration difference between the environments
  • A dependency that behaves differently in production

The blast radius is small. Five percent of users see the problem, not all of them.

The traffic split

By weight. The load balancer sends a share to each version.

# A service mesh rule
traffic:
  - version: v1
    weight: 95
  - version: v2
    weight: 5

By header or cookie. A tester opts in, which is useful for a manual check.

By user id hash. A given user stays on one version, which avoids a confusing experience where a page changes between requests.

By geography or a cohort. A region or a set of accounts.

The choice matters. A weight split is simple and a hash split is consistent. A random split per request is the worst, because a user sees both versions.

The metrics

The technical metrics:

MetricWhat it catches
Error ratean exception, a 5xx
Latency p95, p99a slow query, a slow dependency
CPU, memorya leak, a hot loop
Saturationa connection pool, a queue
Restart counta crash loop

The business metrics:

MetricWhat it catches
Conversion ratea slower page, a broken flow
Signup completiona form regression
Cart abandonmenta checkout regression
Support ticketsa confusing change

The business metric is the one that catches a regression that is not an error. A page that is 300ms slower is not an error, and it can reduce conversions.

The decision

Define the criterion before the release.

Promote if:
  error rate < 0.5%
  p95 latency < 400ms
  conversion rate within 2% of the baseline

Roll back if:
  error rate > 1%
  p95 latency > 600ms
  conversion rate down more than 5%

Automate the promotion. A human who watches a dashboard and decides it seems fine has made a guess.

A staged rollout: 1%, then 5%, then 25%, then 50%, then 100%, with a check at each step.

The rollback

A rollback must be one action.

The options:

  • Shift the weight back to zero
  • Revert the flag
  • Redeploy the previous image

The first two are instant. A redeploy takes minutes, which is minutes of a bad experience.

The requirement: the old version must still be running, or the image must be available.

The database problem

A schema change is the hard part.

The rule: a migration must be backward compatible, so both versions can run at once.

The pattern — expand and contract:

  1. Expand: add the new column, nullable, with no code change
  2. Deploy: the code writes to both and reads the new one
  3. Backfill: fill the new column for the old rows
  4. Contract: remove the old column in a later release

A migration that drops a column breaks the old version, which is still serving traffic.

The traps

A non-representative canary. If the canary gets only the internal users, it does not see the real traffic.

A canary that is too small. Five percent of a low-traffic site is a handful of requests, which is not a sample.

No success criterion. A guess, not a decision.

A shared resource. Both versions write to the same cache or the same queue, and the canary affects the stable version.

A sticky session that is not sticky. A user sees both versions.

A canary that runs forever. It becomes a second production version that nobody maintains.

A rollback that requires a redeploy. Minutes of a bad experience.

A migration that is not backward compatible. The old version breaks.

Canary versus blue-green

CanaryBlue-green
Traffic shiftgradualall at once
Blast radiussmalllarge
Rollout speedslowinstant
Infrastructureboth versionstwo environments
Best fora risky changea fast switch

A canary is the safer default for a change that might regress. Blue-green is better when the switch must be instant.

Common mistakes

No defined metric. A guess.

A canary that is not representative. A misleading result.

A sample that is too small. Noise.

A shared cache. The canary affects the stable version.

A non-backward-compatible migration. The old version breaks.

A rollback that is not one action. A slow recovery.

A canary left running. A second version to maintain.

No business metric. A silent conversion regression.

A manual promotion. Slow and inconsistent.

A quotable summary

A canary deployment routes a small share of production traffic to the new version while the rest continues on the old one, and the team promotes or rolls back based on a metric defined before the release. The split is by weight, header or user hash, the metrics are the error rate, the latency and a business metric, and the migration must be backward compatible so both versions can run at once. A rollback must be one action, not a redeploy.

For the related topics, see blue-green-deployments and ci-cd-pipeline-basics. For the broader context, incident-response-solo-developers and docker-basics-guide.

Frequently asked questions

What is a canary deployment?
It is a release where a small share of production traffic, often one to five percent, is routed to the new version while the rest continues to the old one. The team watches the error rate and the latency, and then either increases the share or rolls back. The name comes from the canary in a coal mine.
How is traffic split for a canary?
By a load balancer weight, by a header or a cookie, by a user id hash, or by a service mesh rule. A weight split is the simplest and a header split is the most controllable, because a tester can opt in. A user id hash keeps a given user on one version, which avoids a confusing experience.
What metrics should I watch?
The error rate, the latency at the 95th and 99th percentile, the saturation of the CPU and memory, and a business metric such as the checkout completion rate. The business metric is the one that catches a regression that is not an error, such as a slower page that reduces conversions.
How long should a canary run?
Long enough to collect a meaningful sample, which depends on the traffic. A site with a thousand requests a minute needs minutes, and a site with a thousand a day needs hours. The rule is to decide the sample size before the release, not to watch until it feels fine.
What is the difference between a canary and a blue-green deployment?
A blue-green deployment runs two full environments and switches all traffic at once. A canary runs both versions and shifts a share gradually. A canary gives a smaller blast radius and a slower rollout, and blue-green gives an instant switch and a larger blast radius.
What is the biggest mistake with a canary?
Watching the canary without a defined success criterion. A team that looks at a dashboard and decides it seems fine has not made a decision, it has made a guess. Define the metric and the threshold before the release, and automate the promotion.