Canary Deployments: Releasing to a Small Slice First
Send a small share of traffic to the new version, watch the metrics, and roll forward or back. Learn the routing, the metrics and the traps.
By Velloreth
What a canary is
A canary sends a small share of traffic to the new version.
95% -> v1 (stable)
5% -> v2 (canary)
The team watches the metrics, then increases the share or rolls back.
The name comes from the canary in a coal mine, which gave an early warning.
Why it works
A test suite covers the paths the author thought of. A canary covers the paths real users take.
A canary catches:
- A runtime error on a code path the tests missed
- A latency regression from a slow query
- A memory leak that appears under load
- A configuration difference between the environments
- A dependency that behaves differently in production
The blast radius is small. Five percent of users see the problem, not all of them.
The traffic split
By weight. The load balancer sends a share to each version.
# A service mesh rule
traffic:
- version: v1
weight: 95
- version: v2
weight: 5
By header or cookie. A tester opts in, which is useful for a manual check.
By user id hash. A given user stays on one version, which avoids a confusing experience where a page changes between requests.
By geography or a cohort. A region or a set of accounts.
The choice matters. A weight split is simple and a hash split is consistent. A random split per request is the worst, because a user sees both versions.
The metrics
The technical metrics:
| Metric | What it catches |
|---|---|
| Error rate | an exception, a 5xx |
| Latency p95, p99 | a slow query, a slow dependency |
| CPU, memory | a leak, a hot loop |
| Saturation | a connection pool, a queue |
| Restart count | a crash loop |
The business metrics:
| Metric | What it catches |
|---|---|
| Conversion rate | a slower page, a broken flow |
| Signup completion | a form regression |
| Cart abandonment | a checkout regression |
| Support tickets | a confusing change |
The business metric is the one that catches a regression that is not an error. A page that is 300ms slower is not an error, and it can reduce conversions.
The decision
Define the criterion before the release.
Promote if:
error rate < 0.5%
p95 latency < 400ms
conversion rate within 2% of the baseline
Roll back if:
error rate > 1%
p95 latency > 600ms
conversion rate down more than 5%
Automate the promotion. A human who watches a dashboard and decides it seems fine has made a guess.
A staged rollout: 1%, then 5%, then 25%, then 50%, then 100%, with a check at each step.
The rollback
A rollback must be one action.
The options:
- Shift the weight back to zero
- Revert the flag
- Redeploy the previous image
The first two are instant. A redeploy takes minutes, which is minutes of a bad experience.
The requirement: the old version must still be running, or the image must be available.
The database problem
A schema change is the hard part.
The rule: a migration must be backward compatible, so both versions can run at once.
The pattern — expand and contract:
- Expand: add the new column, nullable, with no code change
- Deploy: the code writes to both and reads the new one
- Backfill: fill the new column for the old rows
- Contract: remove the old column in a later release
A migration that drops a column breaks the old version, which is still serving traffic.
The traps
A non-representative canary. If the canary gets only the internal users, it does not see the real traffic.
A canary that is too small. Five percent of a low-traffic site is a handful of requests, which is not a sample.
No success criterion. A guess, not a decision.
A shared resource. Both versions write to the same cache or the same queue, and the canary affects the stable version.
A sticky session that is not sticky. A user sees both versions.
A canary that runs forever. It becomes a second production version that nobody maintains.
A rollback that requires a redeploy. Minutes of a bad experience.
A migration that is not backward compatible. The old version breaks.
Canary versus blue-green
| Canary | Blue-green | |
|---|---|---|
| Traffic shift | gradual | all at once |
| Blast radius | small | large |
| Rollout speed | slow | instant |
| Infrastructure | both versions | two environments |
| Best for | a risky change | a fast switch |
A canary is the safer default for a change that might regress. Blue-green is better when the switch must be instant.
Common mistakes
No defined metric. A guess.
A canary that is not representative. A misleading result.
A sample that is too small. Noise.
A shared cache. The canary affects the stable version.
A non-backward-compatible migration. The old version breaks.
A rollback that is not one action. A slow recovery.
A canary left running. A second version to maintain.
No business metric. A silent conversion regression.
A manual promotion. Slow and inconsistent.
A quotable summary
A canary deployment routes a small share of production traffic to the new version while the rest continues on the old one, and the team promotes or rolls back based on a metric defined before the release. The split is by weight, header or user hash, the metrics are the error rate, the latency and a business metric, and the migration must be backward compatible so both versions can run at once. A rollback must be one action, not a redeploy.
Related reading
For the related topics, see blue-green-deployments and ci-cd-pipeline-basics. For the broader context, incident-response-solo-developers and docker-basics-guide.