What are the best practices for managing Kubernetes secrets?

Kubernetes secrets hold your most sensitive data — passwords, API keys, certificates. Here’s how to handle them properly.

1. Use Kubernetes Secrets objects

Store sensitive information in Secret objects, not in ConfigMaps, environment variables, or container images.
Secrets are base64-encoded, not encrypted. Treat them as plaintext until you enable encryption at rest.

2. Enable encryption at rest

Configure the API server to encrypt secrets in etcd using an encryption provider (AES-256 recommended). If etcd is compromised, the secrets stay protected.
Add an EncryptionConfiguration file to the API server config:

yaml kind: EncryptionConfiguration apiVersion: apiserver.config.k8s.io/v1 resources: - secrets providers: - aescbc: keys: - name: key1 secret: <base64-encoded-key> - identity: {}

3. Restrict access with RBAC

Grant least-privilege access to secrets. Define roles that allow only the verbs and resources needed.
Example: a role that permits reading secrets in the default namespace:

yaml kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: default name: secret-reader rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list", "watch"]

4. Never hardcode secrets

Keep secrets out of application code, Dockerfiles, and deployment manifests. Reference them via environment variables or volume mounts.

Mount a secret as an environment variable:
yaml env: - name: DB_PASSWORD valueFrom: secretKeyRef: name: db-secret key: password

5. Use external secret managers

Integrate with HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or Google Secret Manager for stronger encryption, audit trails, and automated rotation.
Sync them into Kubernetes with the External Secrets Operator or similar controllers.

6. Rotate secrets regularly

Rotate credentials on a schedule and after any suspected exposure. Automate rotation through your external secret manager or custom controllers.
Design applications to reload secrets without restart — use volume mounts (which update automatically) or a sidecar that signals reload.

7. Audit and monitor access

Enable Kubernetes audit logging and ship logs to a SIEM. Alert on anomalous access patterns — unusual users, namespaces, or times.
Tools like Falco, Prometheus/Grafana, or commercial security platforms can surface suspicious activity.

8. Keep secrets out of CI/CD logs

Never echo secrets in pipeline scripts. Use your CI/CD system’s secret store (GitHub Actions secrets, GitLab CI variables, Jenkins credentials) and inject them at runtime.

9. Prevent secret leakage in application logs

Configure loggers to redact sensitive fields. Many frameworks log entire request bodies or environment variables by default — disable that.

10. Encrypt secrets in transit

The Kubernetes API server uses TLS by default. For service-to-service traffic, enforce mTLS with a service mesh (Istio, Linkerd, Cilium) or manual certificate management.

11. Isolate secrets by namespace

Scope secrets to the namespace that needs them. Avoid cluster-wide secrets unless absolutely necessary. This limits blast radius if a namespace is compromised.

12. Make secrets immutable when possible

Set immutable: true on secrets that shouldn’t change. This prevents accidental or malicious updates and forces deliberate rotation.

yaml kubectl create secret generic my-secret --from-literal=username=admin --from-literal=password=secret --immutable=true

13. Back up secrets securely

Include secrets in your etcd backup strategy. Encrypt the backup files and store them in a separate security domain (different account, region, or provider).

14. Clean up unused secrets

Run periodic audits (e.g., kubectl get secrets --all-namespaces with label selectors) to find and delete orphaned secrets. Stale secrets expand your attack surface for no reason.