Skip to content

Rotate Secrets

Rotate Doppler service tokens. Covers generation, propagation, and verification.

Purpose

Doppler service tokens are long-lived read-only credentials. Rotate on: suspected compromise, engineer offboarding, or scheduled rotation (recommended: every 90 days).

When to use

  • Suspected token compromise
  • Engineer with token access leaves the team
  • Scheduled rotation cadence

Steps

1. Generate new token in Doppler UI

  1. Go to Doppler -> oep project -> select config (oep-stg / mansety-prd / us-prd)
  2. Service Tokens -> Generate Token -> name: k8s-<env>-<date> -> scope: read-only
  3. Copy token value (shown once)

2. Update GH secret

gh secret set DOPPLER_SERVICE_TOKEN_OEP_STG --repo unipuka/soa --body "<new-token>"
# repeat for mansety-prd / us-prd

3. Update K8s Secret via TF

Pass new token as TF env var and apply:

export TF_VAR_doppler_service_token_oep_stg="<new-token>"
direnv exec oep-infra terraform -chdir=oep-infra apply -target=kubernetes_secret.doppler_token_oep_stg

4. Force ESO re-sync

kubectl annotate externalsecret -n oep-stg --all \
  force-sync=$(date +%s) --overwrite

5. Delete old token in Doppler UI

After confirming ESO sync is healthy, delete the old token.

BD-3 secrets rotation

Stubs for new secrets added in BD-3. Full rotation flows filled in by the consuming ticket.

CF Origin certs (BD-3.8)

See tls-and-certs.md (created in BD-3.8) for the full rotation runbook. Short form: reissue in CF dashboard -> doppler secrets set CF_ORIGIN_CERT_<ZONE>=... + CF_ORIGIN_KEY_<ZONE>=... -> ESO syncs K8s TLS Secret within refreshInterval (60s).

Loki Spaces keys (BD-3.12)

See monitoring.md Loki section (created in BD-3.12) for full rotation flow. Short form: create new key in DO dashboard scoped to the same bucket -> update Doppler LOKI_SPACES_KEY_<ENV> + LOKI_SPACES_SECRET_<ENV> -> restart Loki components so they pick up the new ExternalSecret value -> delete old key in DO dashboard.

Argo CD admin password (BD-3.5/BD-3.14)

# Generate new password
NEW_PWD=$(openssl rand -base64 32 | tr -d '/+=' | head -c 32)
NEW_BCRYPT=$(htpasswd -bnBC 10 "" "$NEW_PWD" | tr -d ':\n' | sed 's/\$2y/\$2a/')
# Verify
TMPFILE=$(mktemp); echo "x:$NEW_BCRYPT" > "$TMPFILE"
htpasswd -bvB "$TMPFILE" x "$NEW_PWD" && rm -f "$TMPFILE"
# Store
doppler secrets set --project oep --config base \
  "ARGOCD_ADMIN_PASSWORD=$NEW_PWD" "ARGOCD_ADMIN_PASSWORD_BCRYPT=$NEW_BCRYPT"
unset NEW_PWD NEW_BCRYPT
# ESO re-syncs argocd-secret patch within refreshInterval
# Force immediate: kubectl annotate externalsecret -n argocd argocd-admin-password force-sync=$(date +%s) --overwrite

Grafana admin password (BD-3.11)

NEW_PWD=$(openssl rand -base64 32 | tr -d '/+=' | head -c 32)
doppler secrets set --project oep --config base "GRAFANA_ADMIN_PASSWORD=$NEW_PWD"
unset NEW_PWD
# Force Grafana to pick up: kubectl rollout restart deployment -n monitoring kube-prometheus-stack-grafana

Rollback

If new token doesn't work, generate another or restore from prior token if still in Doppler UI history.

V&V

  • kubectl get clustersecretstore all Valid
  • kubectl get externalsecret -n oep-stg all SecretSynced
  • doppler secrets download --token=$DOPPLER_SERVICE_TOKEN_OEP_STG --no-file --format json returns expected shape

Laravel app key rotation

DANGER - High blast radius. Plan a maintenance window. APP_KEY rotation immediately invalidates all existing sessions, cookies, and encrypted data the moment pods restart. All users will be logged out. Signed URLs generated with the old key will fail.

Laravel 11+: Set APP_PREVIOUS_KEYS alongside the new APP_KEY before restarting pods so Laravel can decrypt data encrypted with the old key during the transition. Remove APP_PREVIOUS_KEYS only after confirming no decryption errors in logs (see steps below).

Laravel < 11: No APP_PREVIOUS_KEYS fallback. All sessions and encrypted data become unreadable immediately. Accept full logout impact or migrate data before rotating.

Set both keys in Doppler before restarting pods (Laravel 11+):

doppler secrets set APP_KEY="base64:<new-key>" \
                    APP_PREVIOUS_KEYS="base64:<old-key>" \
  --project oep --config oep-stg

After all pods restart and traffic is healthy, retire the old key:

doppler secrets delete APP_PREVIOUS_KEYS --project oep --config oep-stg

When rotating APP_KEY or any other app-level secret in Doppler:

  1. Set new value in Doppler:
doppler secrets set APP_KEY="base64:..." --project oep --config oep-stg
  1. Wait 60s for ESO refreshInterval to sync the K8s Secret:
kubectl -n oep-stg get externalsecret -w
# Watch for LastSyncTime to update
  1. Rolling-restart all workloads to pick up new env:
for workload in api worker websockets; do
  kubectl rollout restart deployment/<release>-$workload -n oep-stg
done
  1. Verify new key active:
curl https://api-staging.unipuka.app/health

Note: No Reloader installed yet - manual restart required. Future: add reloader.stakater.com/auto: "true" annotation when Reloader is deployed.