Skip to content

ADR-0002: Platform Architecture

Status: Accepted Date: 2026-05-23 Ticket: BD-3.0

Context

Migration from AWS ECS to DigitalOcean Kubernetes (DOKS) is complete. All 3 tenants (oep-stg, mansety-prd, us-prd) are running on oep-prd-cluster with the full platform layer: Argo CD GitOps, ESO secret sync, Cilium Gateway API ingress, kube-prometheus-stack observability, and namespace network isolation. This ADR captures the architectural decisions made during BD-3 that shaped the current platform.

Decision

All locked decisions captured from the BD-3 grill rounds:

GitOps - Argo CD

  • App-of-Apps pattern: one root Application per Argo CD install, pointing at bootstrap/platform-apps/. Each platform concern is a separate child Application. ApplicationSet (List generator) reserved for tenant workloads (BD-6.2).
  • Argo CD install: TF helm_release in oep-infra/argocd.tf. Pinned chart version. TF is source of truth for chart version + values. No bootstrap.sh.
  • server.insecure=true: argocd-server listens on plain HTTP; Gateway terminates TLS once at the front. Avoids TLS-in-TLS chain.
  • Dex disabled: auth handled by Cloudflare Access OIDC (BD-3.14). No Dex, no GitHub OAuth for Argo.
  • Argo CD RBAC: policy.default = role:admin for now (small team). Future tightening tracked as follow-up once team grows.
  • Argo CD exposure: ClusterIP + port-forward initially; HTTPRoute on shared Gateway after BD-3.7; CF Access SSO in BD-3.14.

Auth - Cloudflare Access

  • ArgoCD: CF Access self-hosted app at argocd.unipuka.app. Policy: email_domain = unipuka.com via existing Google Workspace IdP (configured in BD-2.3). Argo trusts CF Access JWT via oidc.config pointing at CF Access JWKS endpoint.
  • Grafana: same CF Access edge gate at grafana.unipuka.app. Grafana consumes Cf-Access-Authenticated-User-Email header via auth-proxy; auto-creates users with Admin role.
  • Break-glass: built-in admin credentials retained in Doppler base for both services, accessible only via port-forward (bypasses CF Access).
  • DNS: proxied=true (orange-cloud) for argocd.unipuka.app and grafana.unipuka.app so CF Access intercepts at the edge. Same pattern as docs.unipuka.app.

Sync Waves

Explicit argocd.argoproj.io/sync-wave annotations:

Wave Applications
0 ESO, ClusterSecretStores
1 TLS Secrets, ResourceQuota, LimitRange, NetworkPolicy, kube-prometheus-stack
2 Gateway, HTTPRoutes
3 Loki, OpenCost, Argo HTTPRoute

Argo Application syncOptions (CRD-bearing charts)

Every Application whose chart ships CRDs (external-secrets, kube-prometheus-stack, gateway-api, loki) carries:

syncPolicy:
  syncOptions:
    - ServerSideApply=true
    - Replace=true
    - RespectIgnoreDifferences=true

Avoids metadata.annotations: Too long errors common with kube-prom CRDs.

Secret Sync

  • Doppler is source of truth. ESO syncs into native K8s Secrets; pods consume via envFrom.
  • One ClusterSecretStore per Doppler config (oep-stg, mansety-prd, us-prd).
  • No SOPS, no AGE keys, no encrypted blobs in Git.
  • ESO chart: external-secrets 2.5.0 deployed via Argo Application directly from upstream chart repo (https://charts.external-secrets.io). ClusterSecretStores in separate cluster-secret-stores Application (wave 1, ensures CRDs exist first).
  • See ADR-0001 for full secret management rationale.
  • See external-secrets.md for usage + debugging.

TLS

  • 3 wildcard Cloudflare Origin CA certs (one per zone: *.unipuka.app, *.mansety.com, *.us-academy.net). 15-year validity. ECDSA/RSA-2048.
  • Stored in Doppler base as CF_ORIGIN_CERT_* / CF_ORIGIN_KEY_*.
  • ESO syncs into 3 K8s TLS Secrets in gateway ns via tls-secrets Application (wave 1). Uses tenant ClusterSecretStores (oep-stg/mansety-prd/us-prd) which inherit base keys.
  • Gateway listeners terminate TLS with these secrets. CF edge re-encrypts to origin.
  • Rotation: reissue in CF dashboard -> update Doppler -> ESO syncs within 1h (or force).
  • See tls-and-certs.md.

Gateway

  • Shared Gateway in gateway ns (Cilium Gateway API). One resource, multiple listeners:
  • :80 HTTP redirect to HTTPS
  • :443 per-tenant api hostname (x3) + argocd.unipuka.app + grafana.unipuka.app
  • :6001 per-tenant ws hostname (x3)
  • allowedRoutes.namespaces.selector with label gateway.unipuka.app/tenant: <tenant> enforces cross-tenant hostname isolation.
  • Cilium Gateway API: GatewayClass cilium (verified Accepted=True). CRDs v1.2.1 pre-installed on cluster (6 CRDs confirmed at BD-3.0).
  • DO LB provisioned automatically by Cilium.

NetworkPolicy

  • Default-deny ingress + egress per tenant ns.
  • Named allow rules: intra-ns, DNS (kube-dns selector), SaaS egress 443/tcp, managed DB ipBlock (MySQL + Redis private IPs), ingress from gateway ns.
  • DNS allow uses namespaceSelector + podSelector (kube-dns label) on UDP/TCP 53 - survives Pod IP churn.
  • Broad 443/tcp SaaS egress now; Cilium L7 FQDN allow deferred to later phase.

Quotas

  • Default per tenant ns: CPU req=4, mem req=8Gi, pods=30.
  • LimitRange container defaults: CPU 100m req / 500m limit, mem 128Mi req / 512Mi limit.
  • Per-tenant override via Helm values .Values.tenants.<name>.quota.

Observability

  • Monitoring node pool: dedicated monitoring-pool (s-2vcpu-4gb, min/max=1), taint dedicated=monitoring:NoSchedule. All monitoring Pods scheduled here.
  • kube-prometheus-stack: Prometheus + Grafana + kube-state-metrics + node-exporter. Prometheus PVC reclaim policy = Retain.
  • Loki: simple-scalable mode, DO Spaces backend (3 per-env buckets in ams3). Retention: 7d staging, 30d prod. Promtail/Alloy DaemonSet ships logs with X-Scope-OrgID tenant header.
  • OpenCost: port-forward only by default; wired to in-cluster Prometheus.
  • Grafana PV reclaim policy = Retain. Loki PV same.

Repo Layout

unipuka-infra-ops/
  bootstrap/
    root-application.yaml
    appprojects.yaml
    platform-apps/         # one Application YAML per concern
  platform/
    charts/
      external-secrets/
      tls-secrets/
      resource-quotas/
      network-policy/
      kube-prometheus-stack/
      gateway/
      loki/
      opencost/
unipuka-infra-do/oep-infra/
  argocd.tf
  dns_platform.tf
  cf_access_apps.tf
  monitoring_pool.tf
  spaces_logs.tf

Consequences

Positive:

  • Full GitOps: all platform state driven by Git commits, Argo auto-syncs.
  • Tenant isolation: namespace-scoped NetworkPolicy + Gateway listener selectors.
  • Zero-trust access: CF Access gates all operator UIs; no VPN required.
  • Observability: metrics (Prom), logs (Loki), cost (OpenCost) from day one.
  • Break-glass preserved for every auth-gated service.

Negative / Trade-offs:

  • CF Access adds browser-to-origin round-trip via Cloudflare edge for every UI request.
  • policy.default: role:admin is coarse - tightening deferred.
  • Loki Spaces keys pre-dated bucket TF management; keys scoped to buckets created separately from TF on first bootstrap.
  • server.insecure=true means argocd-server is HTTP internally; Gateway holds the only TLS boundary.

Alternatives Considered

  • Dex for Argo auth: rejected - adds operational complexity; CF Access reuses existing Google Workspace IdP already configured for docs.unipuka.app.
  • Per-tenant Gateway: rejected - one shared Gateway is simpler; listener selectors provide isolation without multiplying LBs.
  • Loki monolithic mode: rejected - simple-scalable gives independent read/write path; lower memory footprint vs distributed.
  • Sealed Secrets / SOPS: rejected per ADR-0001 (Doppler + ESO).

BD-4 reuse note

BD-4 (laravel-app Helm chart) reuses the BD-3 platform without modification. No new Argo CD Applications, namespaces, ClusterSecretStores, or Gateway listeners are added in BD-4. The chart is deployed into existing namespaces (oep-stg, mansety-prd, us-prd) by BD-6.2 ApplicationSet.

DOCR Basic tier: Registry upgraded from Starter to Basic to support image storage and tag retention requirements. See registry.tf and promote-image.md for operational details.