Secret Management: Storing and Rotating Credentials
Secrets in code and config files leak. Learn where to store them, how to inject them, and how to rotate them without downtime.
By Velloreth
The rule
A secret must never be committed to a repository. Not in a config file, not in a test fixture, not in a comment, and not in a private repository. A secret in a repository is copied to every developer’s machine, included in every backup, and exposed by a single misconfigured permission.
The reason is not that developers are careless. It is that a repository is designed to be copied, and a secret is designed to be held by as few places as possible. The two are incompatible.
What counts as a secret
A secret is a value that grants access. A database password, an API key, a private key, a signing secret, a session secret, a webhook token.
Configuration is a value that does not. A port number, a log level, a feature flag, a public URL, a public key.
The test: would publishing this value cause harm? If yes, it is a secret. If no, it is configuration and it can be committed.
The grey area is a value that is not secret by itself but is sensitive in combination, such as an internal hostname. Treat it as configuration but avoid publishing it unnecessarily.
Where to store secrets
A dedicated secret manager is the right answer for a production system. AWS Secrets Manager, Google Secret Manager, Azure Key Vault, and HashiCorp Vault all provide storage, access control, audit logging, and rotation.
The platform’s secret store is the right answer for a small project. GitHub Actions secrets, Cloudflare environment variables, and Vercel environment variables are encrypted at rest and injected at runtime.
The CI platform’s secrets are the right answer for a build. A deploy token belongs in the CI secret store, not in the repository.
A local .env file is acceptable for development, and it must be in .gitignore. It is not a production mechanism, because it is a file on disk that is easy to copy and hard to audit.
Injecting secrets
At runtime, from the environment. The application reads the secret from an environment variable that the platform injects. The value is never in the image or the repository.
const dbPassword = process.env.DB_PASSWORD;
if (!dbPassword) throw new Error('DB_PASSWORD is not set');
Fail fast if a secret is missing. An application that starts with an empty password and fails later is harder to diagnose than one that refuses to start.
Validate the shape, not the value. Check that the secret is present and non-empty. Do not log it, not even a prefix.
Fetch from the manager at startup. For a secret manager, the application authenticates with an identity it already has, such as an instance role, and fetches the secret. The identity is the credential, and it is not a secret in the repository.
import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';
const client = new SecretsManagerClient({ region: 'us-east-1' });
const { SecretString } = await client.send(
new GetSecretValueCommand({ SecretId: 'prod/db' })
);
const { password } = JSON.parse(SecretString);
Scanning for secrets
Scan the working tree and the history. A secret that was deleted in a later commit is still in the history, and a clone retrieves it.
# Scan the working tree
gitleaks detect --no-git
# Scan the full history
gitleaks detect --log-opts="--all"
Add a pre-commit hook. A hook that scans the staged changes catches a secret before it is committed, which is far cheaper than rotating it afterwards.
Add a CI check. A scan in the pipeline catches a secret that a developer bypassed the hook to commit.
Enable the platform’s scanning. GitHub secret scanning and push protection detect a known token pattern and can block the push.
Rotating secrets
Rotate on a schedule. A rotation that never happens is a rotation that does not exist. Ninety days is a common interval for a database password, and a shorter one for a high-value credential.
Rotate immediately after exposure. A secret that was committed, logged, or shared in a message must be assumed compromised. Rotation is the urgent step; removing it from the history is the cleanup.
Rotate when a person leaves. Anyone who had access to a secret should not retain it after they leave.
Automate the rotation. A manual rotation is skipped under pressure. A secret manager can rotate a database credential on a schedule and update the application.
Rotating without downtime
The pattern that makes a rotation a non-event is to support two valid credentials at once.
Step 1: add the new credential. Create a second password for the database user, or a second API key, while the old one still works.
Step 2: deploy the application to use the new one. The old credential is still valid, so a rolling deploy has no window where neither works.
Step 3: confirm the new credential works. Check the logs and the health endpoint.
Step 4: remove the old credential. Once every instance is on the new one, revoke the old.
The overlap is what removes the downtime. A rotation that replaces the credential in one step has a window where the old is revoked and the new is not yet deployed.
Common mistakes
Committing a secret. The most common and most damaging. Use a secret manager or the platform’s store.
A secret in a Docker image. The image is pushed to a registry and pulled by every environment. Inject at runtime instead.
A secret in a log. A request logger that records the headers captures the Authorization header. Redact it.
A secret in an error message. A connection error that includes the password leaks it to the log and the client.
One credential for every service. A leak in one service compromises all of them. Give each service its own credential.
A rotation that is manual. It will be skipped. Automate it.
Assuming a private repository is safe. It is copied, backed up, and permissioned. Treat it as public for the purpose of secrets.
A quotable summary
A secret is a value that grants access and must never be committed to a repository, private or not. Store secrets in a dedicated manager or the platform’s encrypted store, and inject them at runtime through an identity the application already has. Scan the working tree and the full history, because a deleted file is still in the history. Rotate on a schedule and immediately after any exposure, and support two valid credentials during a rotation so the change has no downtime.
Related reading
For the configuration side, see environment-variables-best-practices and for the pipeline side, ci-cd-pipeline-basics. For the broader context, password-storage-best-practices and zero-trust-for-small-teams.