Skip to content

laravel-app Runbook

Operational runbook for the laravel-app Helm chart deployed to DOKS tenant namespaces.

Chart architecture

One chart instance per tenant. 5 workload types per instance:

Workload Kind Inbound Notes
api Deployment HTTP :80 via HTTPRoute Nginx + php-fpm; liveness /status, readiness /health
worker Deployment none Horizon queue consumer
websockets Deployment TCP :6001 via HTTPRoute Reverb realtime; Redis scaling backend
scheduler CronJob none schedule:run every minute; suspended by default
migrate Job (PreSync) none migrate --force --isolated; runs before each Argo sync

Values layout

Secret/config boundary:

  • Secrets: Doppler -> ESO ExternalSecret -> K8s Secret -> all pods envFrom. Never in values.
  • Config: hostnames, replicas, resources, feature flags in values-<tenant>.yaml.

Per-tenant values files: apps/laravel/values-{oep-stg,mansety-prd,us-prd}.yaml.

Health probes

The /status vs /health split:

Probe Path Type Semantics
liveness GET /status shallow Is Nginx + php-fpm alive? No - restart pod.
readiness GET /health deep Is DB + Redis reachable? No - pull from Service endpoints.

/status returns 200 Healthy! (no dependencies). health checks DB::connection()->getPdo() + Redis::connection()->ping(). Returns {"db":"ok","redis":"ok","app":"ok"} on 200, same shape with "fail" values on 503.

/health is registered in routes/status.php via mapStatusRoutes() (no prefix, no middleware). Do NOT register in routes/api.php - that path is prefixed api/v1/ by mapApiRoutes().

# Verify route at root (not api/v1 prefixed)
php artisan route:list | grep health
# expect: GET health  (not api/v1/health)

# Live check
curl https://<hostname>/health
# expect: {"db":"ok","redis":"ok","app":"ok"}

Secret flow

Doppler (oep.oep-stg) -> ClusterSecretStore (doppler-oep-stg)
  -> ExternalSecret (dataFrom: extract "/") -> K8s Secret (<release>-env)
    -> all Pods (envFrom: secretRef: <release>-env)

Refresh interval: 60s. ESO updates the K8s Secret in place.

Manual rollout after secret rotation (no Reloader installed yet):

kubectl rollout restart deployment/<release>-api -n <tenant>
kubectl rollout restart deployment/<release>-worker -n <tenant>
kubectl rollout restart deployment/<release>-websockets -n <tenant>

See rotate-secrets.md for the full rotation procedure.

api workload

  • No command override - uses Dockerfile CMD (supervisord managing Nginx + php-fpm).
  • /horizon dashboard served by api via catch-all HTTPRoute. Auth protected by auth.basic + user_type:admin in config/horizon.php.
  • Storage volumes: emptyDir at /app/storage + /app/bootstrap/cache. Stateless (session/cache/queue on Redis).
  • terminationGracePeriodSeconds: 60 - php-fpm drain + Nginx SIGQUIT.

worker (Horizon) workload

  • Args: php artisan horizon (via args:, not command: - preserves entrypoint.sh ENTRYPOINT for storage dir creation).
  • Graceful drain: preStop php artisan horizon:terminate. terminationGracePeriodSeconds: 300.
  • fast_termination: false in config/horizon.php means Horizon waits for running jobs.
  • Probes: exec [php, artisan, horizon:status] for both liveness + readiness.

Debugging Horizon not picking up jobs:

kubectl exec -n <tenant> deploy/<release>-worker -- php artisan horizon:status
kubectl logs -n <tenant> deploy/<release>-worker --tail=50

websockets (Reverb) workload

  • Args: php artisan reverb:start --port=6001 (via args:, not command: - preserves entrypoint.sh ENTRYPOINT for storage dir creation).
  • Env injected (beyond ExternalSecret): REVERB_SCALING_ENABLED=true, BROADCAST_DRIVER=reverb.
  • Do NOT set REVERB_SCALING_REPLICATION - that env is not read by config/reverb.php (grill correction UNI-76).
  • Reverb uses Redis (port 25061 TLS) as the scaling pubsub backend. NetworkPolicy allow-managed-db permits egress on 25061.
  • Probes: tcpSocket :6001.
  • WebSocket Upgrade end-to-end validation deferred to BD-6.4 (first cluster deploy).

scheduler workload

  • CronJob */1 * * * *, concurrencyPolicy: Forbid, activeDeadlineSeconds: 300.
  • All 3 tenants: suspend: false — cutover complete, scheduler running on all tenants.

Secrets (separate Application)

Each tenant's Doppler secrets and non-secret config are managed by a separate Argo CD Application (<tenant>-configs) defined in bootstrap/platform-apps/<tenant>-configs.yaml. That Application renders apps/configs/ with values-<tenant>.yaml to create:

  • An ExternalSecret → ESO syncs → K8s Secret <tenant>-env (Doppler secrets)
  • A ConfigMap <tenant>-config (non-secret env vars from the config: section of values-<tenant>.yaml)

The main <tenant> Application (this chart) references both via envFrom in all workloads.

Why separate app? Sync-wave ordering in the root App-of-Apps:

  • Wave 1: <tenant>-configs Application → waits for ExternalSecret + ConfigMap Healthy = both exist
  • Wave 2: laravel-app ApplicationSet → <tenant> Application → PreSync migrate runs (secret + config guaranteed to exist)

This breaks the PreSync chicken-and-egg: the K8s secret is always present by the time migrations run, even on the very first deploy of a new namespace.

# Force re-sync if ESO secret is stale
kubectl -n <tenant> annotate externalsecret <tenant>-env \
  force-sync=$(date +%s) --overwrite

Migrations

PreSync Job (templates/migrate/job.yaml): Argo CD runs migrate --force --isolated before rolling Deployments. Job failure aborts the sync; pods stay on the old image.

--isolated acquires an advisory cache lock via Redis (Valkey). It is safe in the Job because REDIS_HOST carries the tls:// prefix (e.g. tls://private-oep-stg-valkey-...) which makes phpredis negotiate TLS on port 25061. Without the prefix, the connection fails and the lock cannot be acquired.

The <tenant>-env secret and <tenant>-config ConfigMap are guaranteed to exist by the time the PreSync Job runs (provided by the <tenant>-configs Application in wave 1). See the Secrets section above.

Also: Entrypoint (soa/entrypoint.sh): migrate --force --isolated on every pod start as a secondary safety net - coordinates concurrent pod starts through the same Redis lock.

Migration BC convention: migrations must be backward-compatible to support rollback. Non-destructive: add columns/tables only; drop in a follow-up after old pods drain.

Troubleshoot failed PreSync Job:

kubectl -n <tenant> get job <tenant>-migrate
kubectl -n <tenant> logs job/<tenant>-migrate
# Fix migration, then:
argocd app sync <tenant> --force

Scaling + disruption

Workload PDB HPA
api minAvailable 50% (0 on single-replica staging) CPU 70% + mem 80%, staging max 4 / prod max 3
worker minAvailable 0 (batch, ok to drain) disabled
websockets minAvailable 1 disabled (connection-based, future)

ResourceQuota cap: 4 CPU / 8Gi / 30 pods per namespace. HPA api max: 4 for oep-stg, 3 for mansety-prd/us-prd (bounded to keep prod under quota with 2x headroom). See quotas-and-limits.md.

Per-tenant values

Sizing (BD-4.9):

Workload oep-stg mansety-prd / us-prd
api 1 replica, 250m/512Mi req, 500m/1Gi lim 2 replicas, 500m/1Gi req, 1000m/2Gi lim
worker 1 replica, 250m/512Mi req, 500m/1Gi lim 1 replica, 500m/1Gi req, 1500m/2Gi lim
websockets 1 replica, 100m/256Mi req, 250m/512Mi lim 2 replicas, 200m/512Mi req, 500m/1Gi lim
scheduler 100m/256Mi req, 250m/512Mi lim 200m/512Mi req, 500m/1Gi lim

Image tag rollback

# Option A: git revert the deploy commit
git revert <deploy-sha>
# Argo prune + selfHeal re-syncs to previous values

# Option B: dispatch do-deploy.yml with old tag
gh workflow run do-deploy.yml -R unipuka/soa -f tag=v1.1.0 -f tenant=oep-stg

ApplicationSet

The laravel-app ApplicationSet (bootstrap/platform-apps/laravel-applicationset.yaml) is a List-generator ApplicationSet applied by the root App-of-Apps (sync-wave 2). It generates one Application per tenant named after the tenant key (e.g. oep-stg, mansety-prd, us-prd).

All Applications:

  • project: tenants (namespace-scoped, no cluster-scoped resources)
  • syncPolicy.automated.prune: true + selfHeal: true
  • Helm source: apps/laravel/, value file: values-<tenant>.yaml
  • Destination namespace: <tenant> (pre-created by Terraform)
# Verify Applications generated
argocd app list | grep -E 'oep-stg|mansety-prd|us-prd'
# expect: oep-stg, mansety-prd, us-prd (plus oep-stg-configs etc.)

# Check auto-sync + prune + selfHeal enabled on an Application
argocd app get oep-stg -o yaml | grep -A3 automated

Rollback: dispatch do-deploy.yml with an older tag, or git revert the deploy commit in unipuka-infra-ops - Argo prune + selfHeal re-syncs to previous values within ~3 min.

Chart versioning

ct lint enforces Chart.yaml version bump on every template change. BD-5.3 do-deploy.yml bumps atomically: Chart.yaml.version, Chart.yaml.appVersion, values-<tenant>.yaml.image.tag.

Debugging

ExternalSecret stuck (not syncing):

kubectl -n <tenant> describe externalsecret <release>-env
# Check conditions: Ready=False + message
kubectl -n external-secrets logs deploy/external-secrets -c manager --tail=50 | grep <tenant>

ExternalSecret-backed Secret not present (pod crashloops envFrom missing secret):

kubectl -n <tenant> get secret <release>-env
# If missing: ESO hasn't synced yet. Check ExtSecret status above.
# Force: kubectl -n <tenant> annotate externalsecret <release>-env force-sync=$(date +%s) --overwrite

Horizon not draining (worker pod stuck Terminating):

  • terminationGracePeriodSeconds: 300 is intentional (5 min).
  • If job stuck, check php artisan horizon:status logs for blocked queue worker.
  • Kill stuck job manually: php artisan queue:forget <job-id>.

Reverb client can't connect:

  • Check NetworkPolicy egress 25061 (Redis TLS): kubectl -n <tenant> get netpol allow-managed-db -o yaml.
  • Check pod logs: kubectl -n <tenant> logs deploy/<release>-websockets.
  • WebSocket Upgrade through Cilium Gateway - validate at BD-6.4.