Monitoring¶

kube-prometheus-stack (Prometheus + Grafana) + Loki + OpenCost, all on the dedicated monitoring-pool node (taint dedicated=monitoring:NoSchedule).

Architecture¶

monitoring-pool node  (s-2vcpu-4gb, taint dedicated=monitoring:NoSchedule)
  ├── Prometheus       (50Gi DO Block Storage PV, 30d retention)
  ├── Grafana          (10Gi PV, CF Access SSO post-BD-3.14)
  ├── Alertmanager
  ├── kube-state-metrics
  ├── Loki             (DO Spaces backend, BD-3.12)
  └── OpenCost         (BD-3.13)

All nodes:
  └── Alloy DaemonSet  (Grafana Alloy, replaces EOL Promtail)

Prometheus access (port-forward)¶

kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090
# http://localhost:9090  ->  Status -> Targets to see all scrape targets

Grafana access¶

Port-forward (break-glass)¶

kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80
# http://localhost:3000
# Username: admin
# Password: $(doppler secrets get GRAFANA_ADMIN_PASSWORD --plain --project oep --config base)

CF Access SSO (after BD-3.14)¶

https://grafana.unipuka.app - CF Access challenge - GitHub OAuth login (unipuka org members only). Auth-proxy reads Cf-Access-Authenticated-User-Email header and auto-creates users with default role Admin. Built-in admin user retained for break-glass.

Debug: CF Access header not forwarded¶

# Check grafana.ini auth.proxy config
kubectl -n monitoring get cm kube-prometheus-stack-grafana -o yaml | grep -A5 auth.proxy
# Verify CF Access app is configured for grafana.unipuka.app (BD-3.14)

Dashboards¶

Grafana auto-provisions dashboards from the kube-prometheus-stack chart. To add custom dashboards: add a ConfigMap with label grafana_dashboard: "1" in monitoring ns. Grafana sidecar discovers and imports them automatically.

Retention + storage¶

Prometheus: 30d retention, 50Gi DO Block Storage, Retain reclaim policy.
Grafana: 10Gi PV for dashboard/user data.
Both PVs survive helm uninstall due to Retain policy.

ServiceMonitor conventions¶

All platform components expose metrics and are scraped via ServiceMonitor CRDs:

Argo CD: ServiceMonitor in argocd ns (added in BD-3.5 values).
Additional monitors: add ServiceMonitor resources in the relevant namespace with release: kube-prometheus-stack label so Prometheus discovers them.

Alerting¶

Alert routing placeholder - rules and receivers added in BD-6.6. See incident-response.md for current alert -> runbook mapping.

OpenCost¶

Deployed in BD-3.13.

kubectl --context do-ams3-oep-prd-cluster -n monitoring port-forward svc/opencost 9090:9090
# http://localhost:9090  -> per-namespace cost breakdown

Wired to in-cluster Prometheus (kube-prometheus-stack-prometheus:9090). Cost projections appear ~24h after install once enough data is collected.

Cross-check monthly estimate against DO billing dashboard for accuracy.

Loki¶

Deployed in BD-3.12. Simple-scalable mode, DO Spaces backend (unipuka-loki-oep-stg).

Query logs (port-forward to Grafana)¶

Add Loki datasource in Grafana: http://loki-gateway.monitoring.svc.cluster.local with tenant header X-Scope-OrgID: oep.

Or query directly:

kubectl --context do-ams3-oep-prd-cluster -n monitoring port-forward svc/loki-gateway 3100:80
curl 'http://localhost:3100/loki/api/v1/query_range?query={namespace="oep-stg"}' \
  -H 'X-Scope-OrgID: oep' | jq .

Retention¶

Default 7d (retention_period: 168h). Compactor enforces deletion. Per-tenant retention overrides added in BD-6.x via limits_config.

Spaces bucket layout¶

unipuka-loki-oep-stg (ams3) - chunks + ruler storage
unipuka-loki-mansety-prd, unipuka-loki-us-prd - reserved for future per-tenant isolation (BD-6.x)

Debug: no logs received¶

# Check Alloy DaemonSet (replaces EOL Promtail)
kubectl --context do-ams3-oep-prd-cluster -n monitoring get ds -l app.kubernetes.io/name=alloy
# Check Alloy logs
kubectl --context do-ams3-oep-prd-cluster -n monitoring logs -l app.kubernetes.io/name=alloy --tail=20

# Check Loki write component
kubectl --context do-ams3-oep-prd-cluster -n monitoring logs -l app.kubernetes.io/component=write --tail=20