Skip to content
Velloreth
Programming 6 min read

Logging Best Practices for Production Systems

Good logs make incidents short and bad logs make them long. Learn log levels, structured logging, what to log, and what never to log.

By

What logs are for

Logs answer a specific question: what happened, and when?

They are one of three observability signals, and it helps to know which to reach for.

SignalAnswers
LogsWhat happened, in what order, with what context
MetricsHow many, how often, how fast
TracesWhere the time went across services

Logs are the most detailed and the most expensive. A metric costs almost nothing to store; a log line costs storage, bandwidth, and query time. That cost is why volume control matters.

Log levels

Levels exist to control volume. The rule is that production runs at info or above, and debug is enabled temporarily when investigating.

LevelUse forExample
errorFailures needing actionPayment failed, database unreachable
warnAnomalies, not failuresRetry succeeded, cache miss rate high
infoSignificant state changesUser created, order shipped, deploy finished
debugDevelopment detailRequest payload, query parameters
traceVery fine-grainedEvery function entry

The most common mistake is logging at error for things that are not errors. A validation failure from user input is not an error — it is expected. Logging it at error trains the team to ignore errors, which defeats the level’s purpose.

The second most common mistake is logging at info in a hot path. A log line per request is fine; a log line per item in a 10,000-item batch is not.

Structured logging

Log key-value pairs, not formatted strings.

// Bad: a string that must be parsed to be queried
console.log(`User ${userId} placed order ${orderId} for $${total}`);

// Good: structured fields
logger.info('order placed', {
  userId,
  orderId,
  total,
  currency: 'USD',
  itemCount: items.length,
});

The structured version is queryable. You can ask “show me all orders over $1000” or “count orders by user” without writing a regex. The string version requires parsing, which is fragile and slow.

Most languages have a structured logger: pino and winston for Node, structlog for Python, zap and logrus for Go, logback with JSON encoder for Java.

What to log

State changes. A user was created, an order was shipped, a subscription was cancelled. These are the events you will want to reconstruct later.

External calls. Every call to a database, API, or queue, with the outcome and duration.

const start = Date.now();
try {
  const result = await paymentGateway.charge(amount);
  logger.info('payment charged', {
    durationMs: Date.now() - start,
    provider: 'stripe',
    amount,
  });
  return result;
} catch (error) {
  logger.error('payment failed', {
    durationMs: Date.now() - start,
    provider: 'stripe',
    amount,
    error: error.message,
  });
  throw error;
}

Errors with context. Not just the message — the inputs, the ids, and the state that led to the failure.

Startup and shutdown. Configuration loaded, migrations applied, server listening. These are invaluable when a deploy goes wrong.

Slow operations. Anything exceeding a threshold, so you can find performance problems before users report them.

What not to log

Secrets. Passwords, tokens, API keys, session ids, private keys. Logs are widely readable, often shipped to third-party services, and retained for months.

Personal data. Names, emails, addresses, phone numbers, and anything else covered by privacy regulation. If you must log an identifier, use a user id, not an email.

Full request bodies. They contain secrets and personal data, and they are large.

Inside loops. A log per iteration produces enormous volume. Log a summary or only the failures.

Duplicate information. If the framework already logs the request, do not log it again.

Noise. “Entering function”, “exiting function”, “step 1 complete” — these add volume without information.

Redaction

Do not rely on developers remembering to redact. Redact at the logging layer.

const logger = pino({
  redact: {
    paths: [
      'req.headers.authorization',
      'req.headers.cookie',
      'password',
      'token',
      'creditCard',
      '*.email',
    ],
    censor: '[REDACTED]',
  },
});

The redaction list is configuration, so it is reviewable and consistent. A developer who logs a whole request object gets the sensitive fields stripped automatically.

Correlation ids

In a distributed system, one request touches several services. A correlation id ties the log lines together.

// middleware: generate or propagate an id
app.use((req, res, next) => {
  req.id = req.headers['x-request-id'] ?? randomUUID();
  res.setHeader('x-request-id', req.id);
  next();
});

// every log line includes it
logger.info('order placed', { requestId: req.id, orderId });

Now you can query all log lines for one request across every service, in order. Without it, correlating logs by timestamp is guesswork.

The id should be generated at the edge and propagated in a header. Services that receive it should reuse it rather than generating a new one.

Log volume

Volume is the main operational cost, and it grows faster than traffic.

Sample high-volume logs. Log 1% of successful requests and 100% of failures.

Use levels properly. debug off in production, info for state changes only.

Aggregate in loops. Log the count and the failures, not every iteration.

Set retention. Keep 7–30 days of searchable logs and archive older ones to cheap storage.

Watch the cost. A log line that costs nothing to write can cost a lot to store and query at scale.

Practical guidance

  • Use structured logging with key-value fields.
  • Choose the level deliberately — error means action is needed.
  • Log state changes and external calls with duration and outcome.
  • Never log secrets or personal data, and redact at the logging layer.
  • Include a correlation id propagated across services.
  • Do not log inside loops — aggregate instead.
  • Sample high-volume paths and keep 100% of failures.
  • Set retention and watch the storage cost.

For the related practices, see error handling patterns that actually work and incident response for solo developers. For the broader context, reading stack traces and zero trust for small teams.

Quotable summary

Logs answer what happened and when, while metrics answer how much and traces answer where the time went. Use structured logging with key-value fields rather than string concatenation, so logs are queryable by user id, status code, or duration. Log levels control volume: error for failures needing action, warn for anomalies, info for state changes, and debug for development. Never log secrets, tokens, passwords, or personal data, and redact at the logging layer rather than relying on developers to remember. Include a correlation id generated at the edge and propagated across services, so you can reconstruct one request’s full path. The main operational cost is volume, so sample high-volume paths, aggregate in loops, and set retention.

Frequently asked questions

What should I log in production?
Log state changes, external calls with their outcome and duration, errors with context, and anything you would want to know during an incident. Do not log every function entry, and never log secrets or personal data.
What is structured logging?
Structured logging emits logs as key-value pairs, usually JSON, rather than as formatted strings. This makes them queryable: you can filter by user id or status code instead of grepping text.
What are the standard log levels?
Error for failures needing action, warn for anomalies that are not failures, info for significant state changes, and debug for development detail. Some systems add trace for very fine-grained output.
How do I avoid logging sensitive data?
Never log passwords, tokens, API keys, or full personal data. Redact fields at the logging layer rather than relying on developers to remember, and review logs before shipping a feature.
What is a correlation id?
A correlation id is a unique identifier generated per request and included in every log line for that request, across all services. It lets you reconstruct the full path of one request through a distributed system.
Should I log in a loop?
Rarely. Logging inside a loop over thousands of items produces enormous volume and hides the signal. Log a summary instead, or log only the failures.