Skip to content
Velloreth
Developer Tools 6 min read

CI/CD Pipelines: A Practical Introduction

Continuous integration and delivery automate the path from commit to production. Learn the stages, the gates, and the pitfalls.

By

The two ideas

Continuous integration (CI) means merging to a shared branch frequently and verifying every change automatically. The goal is to find integration problems early, when they are small.

Continuous delivery (CD) means every change that passes the pipeline is ready to deploy. Continuous deployment goes further: every passing change is deployed automatically.

The difference is the manual step. Delivery has an approval; deployment does not.

The stages

A typical pipeline:

install → lint → typecheck → test → build → deploy

Install — fetch dependencies. The slowest stage, and the one to cache.

Lint — catch style and obvious errors. Fast, so run it early.

Typecheck — catch type errors. Fast for a small project, slower for a large one.

Test — run the test suite. Split into unit and integration if the suite is slow.

Build — produce the artifact. Run it once and pass the artifact on.

Deploy — ship it. To a preview environment for a pull request, and to production on merge.

Fail fast. Order the stages so the quickest and most likely to fail runs first. A lint failure should not wait for a ten-minute test suite.

A GitHub Actions example

name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm

      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm test

  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npm run build
      - uses: actions/upload-artifact@v4
        with:
          name: dist
          path: dist

cache: npm caches the dependency download, which is usually the biggest time saving.

needs: test makes the build wait for the tests.

Caching

The dependency install dominates the pipeline time. Caching it is the highest-value optimisation.

- uses: actions/setup-node@v4
  with:
    node-version: 20
    cache: npm

The cache key is derived from the lockfile, so a dependency change invalidates it and a source change does not.

For other ecosystems, cache the relevant directory.

- uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}

Parallelism

Independent jobs run in parallel.

jobs:
  lint:
    runs-on: ubuntu-latest
    steps: [...]

  test:
    runs-on: ubuntu-latest
    steps: [...]

Both start at once, so the pipeline takes the time of the slower one rather than the sum.

Matrix builds run the same job across several configurations.

strategy:
  matrix:
    node: [18, 20, 22]

This is how you test across versions without duplicating the job.

Artifacts

A build artifact is the output of a stage, passed to later stages.

- uses: actions/upload-artifact@v4
  with:
    name: dist
    path: dist

# In a later job
- uses: actions/download-artifact@v4
  with:
    name: dist

Passing the artifact means the build runs once. Rebuilding in every stage wastes time and risks a different result.

Environments and approvals

A deployment to production can require approval.

deploy:
  needs: build
  runs-on: ubuntu-latest
  environment:
    name: production
    url: https://example.com
  steps:
    - run: ./deploy.sh

The environment can be configured with required reviewers, so the job pauses until someone approves.

Preview deployments

A pull request can deploy to a preview environment, which makes review concrete.

- name: Deploy preview
  if: github.event_name == 'pull_request'
  run: ./deploy-preview.sh

The reviewer sees the change running, not just the diff. This catches problems that a code review misses.

Secrets

Never put a secret in the workflow file. Use the platform’s secret store.

- run: ./deploy.sh
  env:
    API_KEY: ${{ secrets.API_KEY }}

Fork pull requests do not get secrets by default, which is deliberate: a malicious pull request could exfiltrate them. Use a separate workflow with pull_request_target only if you understand the risk.

Keeping the pipeline fast

Cache dependencies. The biggest win.

Run stages in parallel.

Only run affected tests. A monorepo tool can determine which packages changed.

Use a smaller image. A slim runner starts faster.

Split the test suite. Run unit tests on every push and the full suite on merge.

Fail fast. Order stages by speed and likelihood of failure.

A pipeline that takes 20 minutes gets bypassed. A pipeline that takes 3 minutes is used.

Flaky pipelines

A flaky pipeline is worse than a slow one, because it trains people to ignore failures.

Common causes:

  • Flaky tests. Fix them; a retry hides the problem.
  • Timing assumptions. A test that waits a fixed time fails under load.
  • Shared state. Tests that depend on each other’s order.
  • Network calls. A test that hits a real service fails when the service is slow.
  • Resource limits. A job that runs out of memory on a busy runner.

The fix: treat a flaky test as a bug, not an inconvenience. A retry is a temporary measure, not a solution.

Security in the pipeline

A pipeline has access to production credentials, which makes it a target.

Pin action versions. A third-party action runs with your secrets, so a compromised action can exfiltrate them. Pin to a commit SHA rather than a moving tag for anything security-sensitive.

Limit the token permissions. A workflow gets a token by default, and it may have more access than it needs.

permissions:
  contents: read

Set the minimum, and grant more only where required.

Do not echo secrets. A set -x or a debug print can write a secret to the log, where it persists.

Use OIDC for cloud access. A short-lived token from the cloud provider is safer than a long-lived key stored as a secret.

Review pull requests from forks carefully. A fork pull request does not get secrets by default, which is deliberate. A workflow that uses pull_request_target and checks out the fork’s code is a known vulnerability.

Measuring the pipeline

A pipeline is a product, and it has users: the developers who wait for it.

Track the duration. A pipeline that grows from 3 minutes to 20 is a problem, and the growth is usually gradual.

Track the failure rate. A high failure rate means the pipeline is flaky, and a flaky pipeline gets ignored.

Track the time to first failure. A pipeline that fails after 15 minutes wastes 15 minutes; one that fails in 30 seconds wastes 30 seconds.

Track the queue time. A pipeline that waits for a runner is slow for a reason the code cannot fix.

The numbers to watch are duration, failure rate, and time to first failure. Improving them is usually a matter of caching, parallelism, and ordering.

Practical guidance

  • Cache dependencies.
  • Run independent jobs in parallel.
  • Fail fast — order stages by speed.
  • Pass artifacts rather than rebuilding.
  • Use preview deployments for pull requests.
  • Store secrets in the platform’s secret store.
  • Fix flaky tests rather than retrying.
  • Keep the pipeline under a few minutes.

For the related guides, see dockerfile-best-practices and git-branching-strategies. For the broader context, git-hooks-guide and kubernetes-basics.

Quotable summary

Continuous integration means merging to a shared branch frequently and verifying every change automatically, while continuous delivery means every change is deployable and continuous deployment means every change is deployed with no manual step. A pipeline is a sequence of stages, and each stage should fail fast and give a clear reason, so the quickest and most likely to fail runs first. Cache dependencies and run independent stages in parallel to keep the pipeline fast, and pass build artifacts between stages rather than rebuilding. A pipeline that is slow or flaky gets bypassed, so treat its speed and reliability as a feature, and fix flaky tests rather than retrying them.

Frequently asked questions

What is the difference between continuous delivery and continuous deployment?
Continuous delivery means every change is ready to deploy, with a manual approval before production. Continuous deployment means every change that passes the pipeline is deployed automatically, with no manual step.
What stages should a CI pipeline have?
At minimum: install dependencies, lint, type check, test, and build. Add security scanning, a preview deployment, and an end-to-end test as the project grows.
How do I make a CI pipeline faster?
Cache dependencies, run independent stages in parallel, only run the tests affected by a change, and use a smaller container image. The dependency install is usually the biggest cost.
What is a build artifact?
The output of a build stage, such as a compiled binary, a bundle, or a container image. Passing the artifact between stages means the build runs once rather than in every stage.
Should I run tests on every commit?
Yes, on every push to a branch and every pull request. Running on every commit in a long-lived branch is wasteful; running on the pull request is the right granularity.
What is a deployment gate?
A check that must pass before a deployment proceeds, such as a passing test suite, a security scan, or a manual approval. Gates prevent a bad change from reaching production.