laravel-app Runbook

Operational runbook for the laravel-app Helm chart deployed to DOKS tenant namespaces.

Chart architecture

One chart instance per tenant. 5 workload types per instance:

Workload	Kind	Inbound	Notes
api	Deployment	HTTP :80 via HTTPRoute	Nginx + php-fpm; liveness /status, readiness /health
worker	Deployment	none	Horizon queue consumer
websockets	Deployment	TCP :6001 via HTTPRoute	Reverb realtime; Redis scaling backend
scheduler	CronJob	none	schedule:run every minute; suspended by default
migrate	Job (PreSync)	none	migrate --force --isolated; runs before each Argo sync

Values layout

Secret/config boundary:

• Secrets: Doppler -> ESO ExternalSecret -> K8s Secret -> all pods envFrom. Never in values.
• Config: hostnames, replicas, resources, feature flags in values-<tenant>.yaml.

Per-tenant values files: apps/laravel/values-{oep-stg,mansety-prd,us-prd}.yaml.

Health probes

The /status vs /health split:

Probe	Path	Type	Semantics
liveness	GET /status	shallow	Is Nginx + php-fpm alive? No - restart pod.
readiness	GET /health	deep	Is DB + Redis reachable? No - pull from Service endpoints.

/status returns 200 Healthy! (no dependencies). health checks DB::connection()->getPdo() + Redis::connection()->ping(). Returns {"db":"ok","redis":"ok","app":"ok"} on 200, same shape with "fail" values on 503.

/health is registered in routes/status.php via mapStatusRoutes() (no prefix, no middleware). Do NOT register in routes/api.php - that path is prefixed api/v1/ by mapApiRoutes().

# Verify route at root (not api/v1 prefixed)
php artisan route:list | grep health
# expect: GET health  (not api/v1/health)

# Live check
curl https://<hostname>/health
# expect: {"db":"ok","redis":"ok","app":"ok"}

Secret flow

Doppler (oep.oep-stg) -> ClusterSecretStore (doppler-oep-stg)
  -> ExternalSecret (dataFrom: extract "/") -> K8s Secret (<release>-env)
    -> all Pods (envFrom: secretRef: <release>-env)

Refresh interval: 60s. ESO updates the K8s Secret in place.

Manual rollout after secret rotation (no Reloader installed yet):

kubectl rollout restart deployment/<release>-api -n <tenant>
kubectl rollout restart deployment/<release>-worker -n <tenant>
kubectl rollout restart deployment/<release>-websockets -n <tenant>

See rotate-secrets.md for the full rotation procedure.

api workload

• No command override - uses Dockerfile CMD (supervisord managing Nginx + php-fpm).
• /horizon dashboard served by api via catch-all HTTPRoute. Auth protected by auth.basic + user_type:admin in config/horizon.php.
• Storage volumes: emptyDir at /app/storage + /app/bootstrap/cache. Stateless (session/cache/queue on Redis).
• terminationGracePeriodSeconds: 60 - php-fpm drain + Nginx SIGQUIT.

worker (Horizon) workload

• Args: php artisan horizon (via args:, not command: - preserves entrypoint.sh ENTRYPOINT for storage dir creation).
• Graceful drain: preStop php artisan horizon:terminate. terminationGracePeriodSeconds: 300.
• fast_termination: false in config/horizon.php means Horizon waits for running jobs.
• Probes: exec [php, artisan, horizon:status] for both liveness + readiness.

Debugging Horizon not picking up jobs:

kubectl exec -n <tenant> deploy/<release>-worker -- php artisan horizon:status
kubectl logs -n <tenant> deploy/<release>-worker --tail=50

websockets (Reverb) workload

• Args: php artisan reverb:start --port=6001 (via args:, not command: - preserves entrypoint.sh ENTRYPOINT for storage dir creation).
• Env injected (beyond ExternalSecret): REVERB_SCALING_ENABLED=true, BROADCAST_DRIVER=reverb.
• Do NOT set REVERB_SCALING_REPLICATION - that env is not read by config/reverb.php (grill correction UNI-76).
• Reverb uses Redis (port 25061 TLS) as the scaling pubsub backend. NetworkPolicy allow-managed-db permits egress on 25061.
• Probes: tcpSocket :6001.
• WebSocket Upgrade end-to-end validation deferred to BD-6.4 (first cluster deploy).

scheduler workload

• CronJob */1 * * * *, concurrencyPolicy: Forbid, activeDeadlineSeconds: 300.
• All 3 tenants: suspend: false — cutover complete, scheduler running on all tenants.

Secrets (separate Application)

Each tenant's Doppler secrets and non-secret config are managed by a separate Argo CD Application (<tenant>-configs) defined in bootstrap/platform-apps/<tenant>-configs.yaml. That Application renders apps/configs/ with values-<tenant>.yaml to create:

• An ExternalSecret → ESO syncs → K8s Secret <tenant>-env (Doppler secrets)
• A ConfigMap <tenant>-config (non-secret env vars from the config: section of values-<tenant>.yaml)

The main <tenant> Application (this chart) references both via envFrom in all workloads.

Why separate app? Sync-wave ordering in the root App-of-Apps:

• Wave 1: <tenant>-configs Application → waits for ExternalSecret + ConfigMap Healthy = both exist
• Wave 2: laravel-app ApplicationSet → <tenant> Application → PreSync migrate runs (secret + config guaranteed to exist)

This breaks the PreSync chicken-and-egg: the K8s secret is always present by the time migrations run, even on the very first deploy of a new namespace.

# Force re-sync if ESO secret is stale
kubectl -n <tenant> annotate externalsecret <tenant>-env \
  force-sync=$(date +%s) --overwrite

Migrations

PreSync Job (templates/migrate/job.yaml): Argo CD runs migrate --force --isolated before rolling Deployments. Job failure aborts the sync; pods stay on the old image.

--isolated acquires an advisory cache lock via Redis (Valkey). It is safe in the Job because REDIS_HOST carries the tls:// prefix (e.g. tls://private-oep-stg-valkey-...) which makes phpredis negotiate TLS on port 25061. Without the prefix, the connection fails and the lock cannot be acquired.

The <tenant>-env secret and <tenant>-config ConfigMap are guaranteed to exist by the time the PreSync Job runs (provided by the <tenant>-configs Application in wave 1). See the Secrets section above.

Also: Entrypoint (soa/entrypoint.sh): migrate --force --isolated on every pod start as a secondary safety net - coordinates concurrent pod starts through the same Redis lock.

Migration BC convention: migrations must be backward-compatible to support rollback. Non-destructive: add columns/tables only; drop in a follow-up after old pods drain.

Troubleshoot failed PreSync Job:

kubectl -n <tenant> get job <tenant>-migrate
kubectl -n <tenant> logs job/<tenant>-migrate
# Fix migration, then:
argocd app sync <tenant> --force

Scaling + disruption

Workload	PDB	HPA
api	minAvailable 50% (0 on single-replica staging)	CPU 70% + mem 80%, staging max 4 / prod max 3
worker	minAvailable 0 (batch, ok to drain)	disabled
websockets	minAvailable 1	disabled (connection-based, future)

ResourceQuota cap: 4 CPU / 8Gi / 30 pods per namespace. HPA api max: 4 for oep-stg, 3 for mansety-prd/us-prd (bounded to keep prod under quota with 2x headroom). See quotas-and-limits.md.

Per-tenant values

Sizing (BD-4.9):

Workload	oep-stg	mansety-prd / us-prd
api	1 replica, 250m/512Mi req, 500m/1Gi lim	2 replicas, 500m/1Gi req, 1000m/2Gi lim
worker	1 replica, 250m/512Mi req, 500m/1Gi lim	1 replica, 500m/1Gi req, 1500m/2Gi lim
websockets	1 replica, 100m/256Mi req, 250m/512Mi lim	2 replicas, 200m/512Mi req, 500m/1Gi lim
scheduler	100m/256Mi req, 250m/512Mi lim	200m/512Mi req, 500m/1Gi lim

Image tag rollback

# Option A: git revert the deploy commit
git revert <deploy-sha>
# Argo prune + selfHeal re-syncs to previous values

# Option B: dispatch do-deploy.yml with old tag
gh workflow run do-deploy.yml -R unipuka/soa -f tag=v1.1.0 -f tenant=oep-stg

ApplicationSet

The laravel-app ApplicationSet (bootstrap/platform-apps/laravel-applicationset.yaml) is a List-generator ApplicationSet applied by the root App-of-Apps (sync-wave 2). It generates one Application per tenant named after the tenant key (e.g. oep-stg, mansety-prd, us-prd).

All Applications:

• project: tenants (namespace-scoped, no cluster-scoped resources)
• syncPolicy.automated.prune: true + selfHeal: true
• Helm source: apps/laravel/, value file: values-<tenant>.yaml
• Destination namespace: <tenant> (pre-created by Terraform)

# Verify Applications generated
argocd app list | grep -E 'oep-stg|mansety-prd|us-prd'
# expect: oep-stg, mansety-prd, us-prd (plus oep-stg-configs etc.)

# Check auto-sync + prune + selfHeal enabled on an Application
argocd app get oep-stg -o yaml | grep -A3 automated

Rollback: dispatch do-deploy.yml with an older tag, or git revert the deploy commit in unipuka-infra-ops - Argo prune + selfHeal re-syncs to previous values within ~3 min.

Tenant shutdown / decommission

Procedure used for us-prd (UNI-140), applicable to any tenant. Two Applications are involved per tenant (see Secrets (separate Application)), and both are automated: {prune: true, selfHeal: true} with no ignoreDifferences, so every change must land in git - a live kubectl scale is reverted within ~3 min.

Step A - drain (one PR, values-<tenant>.yaml):

api:
  replicas: 0
  hpa:
    enabled: false   # must be false, not just replicas: 0 - minReplicas would fight the scale-down
websockets:
  replicas: 0
worker:
  replicas: 1        # stays up until Horizon queues are drained
scheduler:
  suspend: true

After merging, before proceeding to Step B: confirm job/<tenant>-migrate (the PreSync hook this sync also re-runs) reports Complete, not Failed or still Active:

kubectl -n <tenant> get job <tenant>-migrate

If it failed, Argo CD aborts the sync and the Deployments stay on their old replica counts (Step A's drain did not take effect) - the existing Deployments keep running untouched. Treat Step A as incomplete: inspect kubectl -n <tenant> logs job/<tenant>-migrate, fix the migration, then re-sync (argocd app sync <tenant> --force) before touching anything else.

Then drain the queues. Enumerate every queue from config/horizon.php explicitly and check each one's depth with Queue::size($queue) - horizon:status reports supervisor health, not queue depth, and is not a substitute:

POD=$(kubectl get pod -n <tenant> -l app.kubernetes.io/component=worker -o name | head -1)
kubectl exec -n <tenant> $POD -- php artisan tinker --execute \
  "foreach(['default','transactions','posts','scout','notifications'] as \$q) echo \$q.'='.Queue::size(\$q).PHP_EOL;"

Wait for every queue to hit zero before proceeding. If this throws a Redis exception, treat the depth as unknown, not zero - do not proceed as if drained. See the Redis IP drift caveat below; if that's the cause, follow its abandonment path instead of blocking indefinitely.

Step B - remove the tenant (second PR):

• Delete the - tenant: <tenant> element from laravel-applicationset.yaml.
• Delete bootstrap/platform-apps/<tenant>-configs.yaml.

Per the ApplicationSet Application-Deletion semantics, because syncPolicy.preserveResourcesOnDeletion is unset on the ApplicationSet, the controller had added a resources-finalizer.argocd.argoproj.io finalizer to the generated Application - removing the list element deletes every resource managed by that generated <tenant> Application, with no further PreSync migrate run. In practice that's all of the tenant's workloads, but the mechanism is Application ownership, not a namespace-wide wipe - any resource in the namespace not tracked by this Application (there normally isn't one) would be left alone. The <tenant>-configs Application has no such finalizer, so pruning it orphans the <tenant>-env Secret/ExternalSecret and <tenant>-config ConfigMap in the namespace (harmless, cleaned up whenever the namespace itself is deleted).

Out of scope for both steps: the managed MySQL/Valkey clusters, namespace, ResourceQuota, NetworkPolicy, Gateway listeners, TLS ExternalSecret, and ClusterSecretStore all live outside these two Applications and are untouched.

Caveat - Redis IP drift can make the drain step unverifiable. DO Managed Valkey's private IP can change on HA failover; the allow-managed-db NetworkPolicy pins a static IP per tenant (platform/charts/network-policy/values.yaml). If that IP is stale, workers can't reach Redis at all - they crash-loop on the horizon:status probe (RedisException: Connection timed out) before ever pulling a job, so Horizon can't report queue size and failed_jobs does not grow (a job has to actually be attempted and exhaust its retries to land there; a worker that can't connect never attempts anything). A large pre-existing failed_jobs count is not evidence of this incident - check MAX(failed_at) against when connectivity broke before assuming any correlation. The real risk is the opposite: jobs pile up invisibly in Redis with no failure signal at all. Resolve the private hostname from inside the cluster to check:

kubectl run dnscheck --image=busybox:1.36 --rm -i --restart=Never -n <tenant> -- \
  nslookup private-<tenant>-valkey-do-user-<id>.i.db.ondigitalocean.com

If it doesn't match network-policy/values.yaml, that's a separate incident - fix it (update the IP there + cloudflared_tunnel.tf) independently of the shutdown, and treat anything already stuck in the queue as abandoned rather than blocking the decommission on it.

Chart versioning

ct lint enforces Chart.yaml version bump on every template change. BD-5.3 do-deploy.yml bumps atomically: Chart.yaml.version, Chart.yaml.appVersion, values-<tenant>.yaml.image.tag.

Debugging

ExternalSecret stuck (not syncing):

kubectl -n <tenant> describe externalsecret <release>-env
# Check conditions: Ready=False + message
kubectl -n external-secrets logs deploy/external-secrets -c manager --tail=50 | grep <tenant>

ExternalSecret-backed Secret not present (pod crashloops envFrom missing secret):

kubectl -n <tenant> get secret <release>-env
# If missing: ESO hasn't synced yet. Check ExtSecret status above.
# Force: kubectl -n <tenant> annotate externalsecret <release>-env force-sync=$(date +%s) --overwrite

Horizon not draining (worker pod stuck Terminating):

• terminationGracePeriodSeconds: 300 is intentional (5 min).
• If job stuck, check php artisan horizon:status logs for blocked queue worker.
• Kill stuck job manually: php artisan queue:forget <job-id>.

Reverb client can't connect:

• Check NetworkPolicy egress 25061 (Redis TLS): kubectl -n <tenant> get netpol allow-managed-db -o yaml.
• Check pod logs: kubectl -n <tenant> logs deploy/<release>-websockets.
• WebSocket Upgrade through Cilium Gateway - validate at BD-6.4.