Incident Response

Alert triage map. Match alert name to runbook section.

Alert -> runbook map

Alert	Severity	Runbook section
ApiHighErrorRate (5xx > 1% 5m)	P1	API errors
WorkerQueueStalled (> 10min)	P2	Queue stalled
WebsocketConnectionsDrop (> 50% 5m)	P2	WS drop
SchedulerMissed (> 2 consecutive)	P3	Scheduler
MigrateJobFailed	P1	Migrate failed
ArgoCDSyncFailed	P1	Argo sync

---

API high error rate

Purpose

Investigate elevated 5xx responses on the api workload.

When to use

Alert ApiHighErrorRate fires.

Steps

1. Check pod logs: kubectl logs -n <tenant> -l app=api --tail=100
2. Check recent deploy: argocd app history <tenant>-laravel
3. If bad deploy: rollback via do-deploy.yml dispatch with previous tag (see Promote Image)
4. If infra issue: check DO status page, managed DB health

Rollback

Re-dispatch do-deploy.yml with last known-good tag.

---

Worker queue stalled

Purpose

Horizon not processing jobs.

Steps

1. kubectl exec -n <tenant> deploy/worker -- php artisan horizon:status
2. Check Redis connectivity:

kubectl exec -n <tenant> deploy/<tenant>-worker -it -- php artisan tinker

Then in the tinker shell:

use Illuminate\Support\Facades\Redis; $redis = Redis::connection(); $redis->ping();

Expected: "PONG". Timeout or exception = Redis unreachable (check NetworkPolicy + DB firewall). 3. Restart worker pod if stuck: kubectl rollout restart deploy/worker -n <tenant>

---

Websocket connections drop

Purpose

Reverb pod(s) losing active WS connections.

Steps

1. Check Reverb pods: kubectl get pods -n <tenant> -l app=websockets
2. Check Redis broker connectivity (REVERB_SCALING_REPLICATION=redis)
3. Check Gateway listener :6001 health

---

Scheduler missed

Purpose

CronJob not firing or timing out.

Steps

1. kubectl get cronjobs -n <tenant>
2. kubectl get jobs -n <tenant> --sort-by=.metadata.creationTimestamp | tail -5
3. Check job logs for failures

---

Migrate Job failed

Purpose

PreSync migration failed, blocking Argo sync. Existing pods still running old code.

Steps

1. argocd app get <tenant>-laravel - note the failed Job name
2. kubectl logs job/<migrate-job-name> -n <tenant>
3. Fix migration or roll back image tag via do-deploy.yml dispatch
4. Once fixed: argocd app sync <tenant>-laravel

---

Managed DB unreachable (HA failover IP drift)

Symptoms

• API pods 0/1 Not Ready, readiness probe timing out on /health
• Worker crash-looping with RedisException: Connection timed out or MySQL equivalent
• /health returns {"redis":"error"} or hangs; /status still responds

Cause

DO Managed DB HA failover promoted the standby node. The private hostname still resolves, but now points to a different VPC IP. The allow-managed-db NetworkPolicy uses hardcoded ipBlock CIDRs - the old IP is now blocked.

Steps

# 1. Identify which DB is unreachable from pod logs
kubectl logs -n <tenant> deploy/<tenant>-worker --previous --tail=20

# 2. Confirm Redis is actually unreachable from a running pod
kubectl exec -n <tenant> deploy/<tenant>-api -it -- php artisan tinker
# In tinker:
#   use Illuminate\Support\Facades\Redis; $redis = Redis::connection(); $redis->ping();
# Expected: "PONG". Timeout = blocked by NetworkPolicy.

# 3. Resolve current private hostname from inside cluster
kubectl run dns-test --rm -i --restart=Never --image=busybox -n <tenant> \
  -- nslookup private-<tenant>-valkey-do-user-34192537-0.i.db.ondigitalocean.com
# (or mysql: private-<tenant>-mysql-do-user-34192537-0.a.db.ondigitalocean.com)

# 4. Compare with current NetworkPolicy
kubectl get networkpolicy allow-managed-db -n <tenant> -o yaml | grep cidr

# 5. If IPs differ: patch live NetworkPolicy immediately
kubectl patch networkpolicy allow-managed-db -n <tenant> --type='json' \
  -p='[{"op":"replace","path":"/spec/egress/1/to/0/ipBlock/cidr","value":"<new-ip>/32"}]'
# (index 0 = MySQL on port 25060, index 1 = Valkey on port 25061)

# 6. Update GitOps source to prevent Argo CD self-heal reverting the patch
# Edit platform/charts/network-policy/values.yaml + platform/charts/cloudflared/values.yaml
# Change redisIP (or mysqlIP) to the new IP, commit + push immediately

Note: Argo CD selfHeal: true will revert the live patch within minutes. Push the GitOps change before that happens.

Long-term fix

UNI-138 - replace ipBlock rules with CiliumNetworkPolicy toFQDNs.

---

Argo CD sync failed

Purpose

Argo CD cannot sync an Application.

Steps

1. argocd app get <app-name> - read the conditions
2. Common causes: PreSync hook failed (see above), network issue to cluster, invalid manifest
3. If network issue to infra-ops: check GitHub App credentials Secret in argocd ns