Skip to content

ADR 0001: Secret Management Approach

Status: Accepted Date: 2026-05-16 Ticket: UNI-49 (BD-0.3)

Context

The soa Laravel backend requires ~30-50 runtime secrets per tenant env (DB credentials, Redis URLs, API keys for Sentry/Bugsnag/Pusher/Typesense/Firebase, APP_KEY, etc.). These previously lived in AWS SSM Parameter Store and were injected via ECS task-definition environment variables.

Moving to DOKS (Kubernetes), we need a secret management strategy that:

  • Is portable (not tied to AWS SSM or any single provider)
  • Does not store secret values in Git or Terraform state
  • Supports per-tenant config isolation
  • Enables rotation without pod restarts (where possible)
  • Works with External Secrets Operator (ESO) -> K8s Secret -> envFrom pattern

Decision

Doppler is the source of truth for all backend runtime secrets.

Project structure

One Doppler project per service:

Project Configs
oep base, oep-stg, mansety-prd, us-prd

base holds shared keys (Sentry DSN, Bugsnag API key, Firebase base config, Cloudflare Origin certs). Per-env configs (oep-stg, mansety-prd, us-prd) inherit from base and override with env-specific values (DB_HOST, DB_PASSWORD, APP_KEY, REDIS_URL, etc.).

Secret/config boundary

What Where
Runtime secrets (credentials, keys, DSNs) Doppler only. Never in Git. Never in Helm values.
Static config (hostnames, replicas, resource sizes) Helm values-<tenant>.yaml in infra-ops. Never in Doppler.
TLS certs (Cloudflare Origin certs) Doppler base config (keys CF_ORIGIN_CERT_<host>, CF_ORIGIN_KEY_<host>). ESO pulls into K8s TLS Secrets.

Service token flow

  1. Doppler service tokens generated manually in the Doppler UI (read-only scope, one per non-base config).
  2. Token stored in 3 places only:
  3. Doppler itself (source)
  4. GH secret at unipuka/soa repo level: DOPPLER_TOKEN_OEP_STG, DOPPLER_TOKEN_MANSETY_PRD, DOPPLER_TOKEN_US_PRD (CI usage)
  5. K8s Secret in external-secrets ns: doppler-token-<env> (ESO usage)
  6. TF creates the K8s Secret from var.doppler_service_token_<env> (sensitive TF input, passed via TF_VAR_* env at apply time, never committed).
  7. TF does NOT manage service token lifecycle via the Doppler provider (avoids token values in TF state).

ESO sync model

Doppler config (oep-stg)
  -> ClusterSecretStore (doppler-token-oep-stg K8s Secret)
    -> ExternalSecret (per workload, refreshInterval: 60s)
      -> K8s Secret (<release>-app-env)
        -> Pod envFrom

One ClusterSecretStore per Doppler config (3 total). All workloads (api, worker, websockets, scheduler, migrate hook) reference the same ClusterSecretStore for their tenant.

Consequences

Positive:

  • Secrets never in Git or Terraform state
  • Per-tenant isolation at the Doppler config level
  • Config inheritance (base -> env) reduces duplication
  • Provider-portable: ESO + ClusterSecretStore pattern works with Vault, AWS Secrets Manager, etc. - swap provider without changing K8s manifests
  • 60s refresh enables key rotation without pod restarts
  • Doppler UI provides audit trail + access control

Negative/Trade-offs:

  • Manual service token generation adds an ops step (mitigated by rotation runbook)
  • TF must be re-applied when service tokens are rotated (to update the K8s Secret)
  • ESO refresh lag of up to 60s before pods see a rotated secret

Alternatives Considered

AWS Secrets Manager via ESO: Considered but rejected. Tight coupling to AWS; migration goal is to reduce AWS dependency.

HashiCorp Vault: Out of scope per plan Section 3. Would require additional infra to operate.

SOPS + AGE keys in Git: Rejected. Encrypted blobs in Git create key management complexity and the decryption key becomes the new secret.

GH Org-level secrets as primary store: Current state on AWS. Rejected for DO because secrets cannot be synced to K8s without an additional bridge; also org-level scope is wider than needed.

Rotation Runbook

See docs/runbooks/rotate-secrets.md.