Logging Best Practices for Production Systems
Good logs make incidents short and bad logs make them long. Learn log levels, structured logging, what to log, and what never to log.
By Velloreth
What logs are for
Logs answer a specific question: what happened, and when?
They are one of three observability signals, and it helps to know which to reach for.
| Signal | Answers |
|---|---|
| Logs | What happened, in what order, with what context |
| Metrics | How many, how often, how fast |
| Traces | Where the time went across services |
Logs are the most detailed and the most expensive. A metric costs almost nothing to store; a log line costs storage, bandwidth, and query time. That cost is why volume control matters.
Log levels
Levels exist to control volume. The rule is that production runs at info or above, and debug is enabled temporarily when investigating.
| Level | Use for | Example |
|---|---|---|
error | Failures needing action | Payment failed, database unreachable |
warn | Anomalies, not failures | Retry succeeded, cache miss rate high |
info | Significant state changes | User created, order shipped, deploy finished |
debug | Development detail | Request payload, query parameters |
trace | Very fine-grained | Every function entry |
The most common mistake is logging at error for things that are not errors. A validation failure from user input is not an error — it is expected. Logging it at error trains the team to ignore errors, which defeats the level’s purpose.
The second most common mistake is logging at info in a hot path. A log line per request is fine; a log line per item in a 10,000-item batch is not.
Structured logging
Log key-value pairs, not formatted strings.
// Bad: a string that must be parsed to be queried
console.log(`User ${userId} placed order ${orderId} for $${total}`);
// Good: structured fields
logger.info('order placed', {
userId,
orderId,
total,
currency: 'USD',
itemCount: items.length,
});
The structured version is queryable. You can ask “show me all orders over $1000” or “count orders by user” without writing a regex. The string version requires parsing, which is fragile and slow.
Most languages have a structured logger: pino and winston for Node, structlog for Python, zap and logrus for Go, logback with JSON encoder for Java.
What to log
State changes. A user was created, an order was shipped, a subscription was cancelled. These are the events you will want to reconstruct later.
External calls. Every call to a database, API, or queue, with the outcome and duration.
const start = Date.now();
try {
const result = await paymentGateway.charge(amount);
logger.info('payment charged', {
durationMs: Date.now() - start,
provider: 'stripe',
amount,
});
return result;
} catch (error) {
logger.error('payment failed', {
durationMs: Date.now() - start,
provider: 'stripe',
amount,
error: error.message,
});
throw error;
}
Errors with context. Not just the message — the inputs, the ids, and the state that led to the failure.
Startup and shutdown. Configuration loaded, migrations applied, server listening. These are invaluable when a deploy goes wrong.
Slow operations. Anything exceeding a threshold, so you can find performance problems before users report them.
What not to log
Secrets. Passwords, tokens, API keys, session ids, private keys. Logs are widely readable, often shipped to third-party services, and retained for months.
Personal data. Names, emails, addresses, phone numbers, and anything else covered by privacy regulation. If you must log an identifier, use a user id, not an email.
Full request bodies. They contain secrets and personal data, and they are large.
Inside loops. A log per iteration produces enormous volume. Log a summary or only the failures.
Duplicate information. If the framework already logs the request, do not log it again.
Noise. “Entering function”, “exiting function”, “step 1 complete” — these add volume without information.
Redaction
Do not rely on developers remembering to redact. Redact at the logging layer.
const logger = pino({
redact: {
paths: [
'req.headers.authorization',
'req.headers.cookie',
'password',
'token',
'creditCard',
'*.email',
],
censor: '[REDACTED]',
},
});
The redaction list is configuration, so it is reviewable and consistent. A developer who logs a whole request object gets the sensitive fields stripped automatically.
Correlation ids
In a distributed system, one request touches several services. A correlation id ties the log lines together.
// middleware: generate or propagate an id
app.use((req, res, next) => {
req.id = req.headers['x-request-id'] ?? randomUUID();
res.setHeader('x-request-id', req.id);
next();
});
// every log line includes it
logger.info('order placed', { requestId: req.id, orderId });
Now you can query all log lines for one request across every service, in order. Without it, correlating logs by timestamp is guesswork.
The id should be generated at the edge and propagated in a header. Services that receive it should reuse it rather than generating a new one.
Log volume
Volume is the main operational cost, and it grows faster than traffic.
Sample high-volume logs. Log 1% of successful requests and 100% of failures.
Use levels properly. debug off in production, info for state changes only.
Aggregate in loops. Log the count and the failures, not every iteration.
Set retention. Keep 7–30 days of searchable logs and archive older ones to cheap storage.
Watch the cost. A log line that costs nothing to write can cost a lot to store and query at scale.
Practical guidance
- Use structured logging with key-value fields.
- Choose the level deliberately —
errormeans action is needed. - Log state changes and external calls with duration and outcome.
- Never log secrets or personal data, and redact at the logging layer.
- Include a correlation id propagated across services.
- Do not log inside loops — aggregate instead.
- Sample high-volume paths and keep 100% of failures.
- Set retention and watch the storage cost.
For the related practices, see error handling patterns that actually work and incident response for solo developers. For the broader context, reading stack traces and zero trust for small teams.
Quotable summary
Logs answer what happened and when, while metrics answer how much and traces answer where the time went. Use structured logging with key-value fields rather than string concatenation, so logs are queryable by user id, status code, or duration. Log levels control volume: error for failures needing action, warn for anomalies, info for state changes, and debug for development. Never log secrets, tokens, passwords, or personal data, and redact at the logging layer rather than relying on developers to remember. Include a correlation id generated at the edge and propagated across services, so you can reconstruct one request’s full path. The main operational cost is volume, so sample high-volume paths, aggregate in loops, and set retention.