CI/CD Pipelines: A Practical Introduction
Continuous integration and delivery automate the path from commit to production. Learn the stages, the gates, and the pitfalls.
By Velloreth
The two ideas
Continuous integration (CI) means merging to a shared branch frequently and verifying every change automatically. The goal is to find integration problems early, when they are small.
Continuous delivery (CD) means every change that passes the pipeline is ready to deploy. Continuous deployment goes further: every passing change is deployed automatically.
The difference is the manual step. Delivery has an approval; deployment does not.
The stages
A typical pipeline:
install → lint → typecheck → test → build → deploy
Install — fetch dependencies. The slowest stage, and the one to cache.
Lint — catch style and obvious errors. Fast, so run it early.
Typecheck — catch type errors. Fast for a small project, slower for a large one.
Test — run the test suite. Split into unit and integration if the suite is slow.
Build — produce the artifact. Run it once and pass the artifact on.
Deploy — ship it. To a preview environment for a pull request, and to production on merge.
Fail fast. Order the stages so the quickest and most likely to fail runs first. A lint failure should not wait for a ten-minute test suite.
A GitHub Actions example
name: CI
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
- run: npm ci
- run: npm run lint
- run: npm run typecheck
- run: npm test
build:
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
- run: npm ci
- run: npm run build
- uses: actions/upload-artifact@v4
with:
name: dist
path: dist
cache: npm caches the dependency download, which is usually the biggest time saving.
needs: test makes the build wait for the tests.
Caching
The dependency install dominates the pipeline time. Caching it is the highest-value optimisation.
- uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
The cache key is derived from the lockfile, so a dependency change invalidates it and a source change does not.
For other ecosystems, cache the relevant directory.
- uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
Parallelism
Independent jobs run in parallel.
jobs:
lint:
runs-on: ubuntu-latest
steps: [...]
test:
runs-on: ubuntu-latest
steps: [...]
Both start at once, so the pipeline takes the time of the slower one rather than the sum.
Matrix builds run the same job across several configurations.
strategy:
matrix:
node: [18, 20, 22]
This is how you test across versions without duplicating the job.
Artifacts
A build artifact is the output of a stage, passed to later stages.
- uses: actions/upload-artifact@v4
with:
name: dist
path: dist
# In a later job
- uses: actions/download-artifact@v4
with:
name: dist
Passing the artifact means the build runs once. Rebuilding in every stage wastes time and risks a different result.
Environments and approvals
A deployment to production can require approval.
deploy:
needs: build
runs-on: ubuntu-latest
environment:
name: production
url: https://example.com
steps:
- run: ./deploy.sh
The environment can be configured with required reviewers, so the job pauses until someone approves.
Preview deployments
A pull request can deploy to a preview environment, which makes review concrete.
- name: Deploy preview
if: github.event_name == 'pull_request'
run: ./deploy-preview.sh
The reviewer sees the change running, not just the diff. This catches problems that a code review misses.
Secrets
Never put a secret in the workflow file. Use the platform’s secret store.
- run: ./deploy.sh
env:
API_KEY: ${{ secrets.API_KEY }}
Fork pull requests do not get secrets by default, which is deliberate: a malicious pull request could exfiltrate them. Use a separate workflow with pull_request_target only if you understand the risk.
Keeping the pipeline fast
Cache dependencies. The biggest win.
Run stages in parallel.
Only run affected tests. A monorepo tool can determine which packages changed.
Use a smaller image. A slim runner starts faster.
Split the test suite. Run unit tests on every push and the full suite on merge.
Fail fast. Order stages by speed and likelihood of failure.
A pipeline that takes 20 minutes gets bypassed. A pipeline that takes 3 minutes is used.
Flaky pipelines
A flaky pipeline is worse than a slow one, because it trains people to ignore failures.
Common causes:
- Flaky tests. Fix them; a retry hides the problem.
- Timing assumptions. A test that waits a fixed time fails under load.
- Shared state. Tests that depend on each other’s order.
- Network calls. A test that hits a real service fails when the service is slow.
- Resource limits. A job that runs out of memory on a busy runner.
The fix: treat a flaky test as a bug, not an inconvenience. A retry is a temporary measure, not a solution.
Security in the pipeline
A pipeline has access to production credentials, which makes it a target.
Pin action versions. A third-party action runs with your secrets, so a compromised action can exfiltrate them. Pin to a commit SHA rather than a moving tag for anything security-sensitive.
Limit the token permissions. A workflow gets a token by default, and it may have more access than it needs.
permissions:
contents: read
Set the minimum, and grant more only where required.
Do not echo secrets. A set -x or a debug print can write a secret to the log, where it persists.
Use OIDC for cloud access. A short-lived token from the cloud provider is safer than a long-lived key stored as a secret.
Review pull requests from forks carefully. A fork pull request does not get secrets by default, which is deliberate. A workflow that uses pull_request_target and checks out the fork’s code is a known vulnerability.
Measuring the pipeline
A pipeline is a product, and it has users: the developers who wait for it.
Track the duration. A pipeline that grows from 3 minutes to 20 is a problem, and the growth is usually gradual.
Track the failure rate. A high failure rate means the pipeline is flaky, and a flaky pipeline gets ignored.
Track the time to first failure. A pipeline that fails after 15 minutes wastes 15 minutes; one that fails in 30 seconds wastes 30 seconds.
Track the queue time. A pipeline that waits for a runner is slow for a reason the code cannot fix.
The numbers to watch are duration, failure rate, and time to first failure. Improving them is usually a matter of caching, parallelism, and ordering.
Practical guidance
- Cache dependencies.
- Run independent jobs in parallel.
- Fail fast — order stages by speed.
- Pass artifacts rather than rebuilding.
- Use preview deployments for pull requests.
- Store secrets in the platform’s secret store.
- Fix flaky tests rather than retrying.
- Keep the pipeline under a few minutes.
For the related guides, see dockerfile-best-practices and git-branching-strategies. For the broader context, git-hooks-guide and kubernetes-basics.
Quotable summary
Continuous integration means merging to a shared branch frequently and verifying every change automatically, while continuous delivery means every change is deployable and continuous deployment means every change is deployed with no manual step. A pipeline is a sequence of stages, and each stage should fail fast and give a clear reason, so the quickest and most likely to fail runs first. Cache dependencies and run independent stages in parallel to keep the pipeline fast, and pass build artifacts between stages rather than rebuilding. A pipeline that is slow or flaky gets bypassed, so treat its speed and reliability as a feature, and fix flaky tests rather than retrying them.