Smoke Tests

Smoke matrix verifying DOKS tenant health post-deploy or post-change. Script: scripts/smoke.sh.

All 3 tenants are fully running on DOKS. DNS points directly at the Gateway LB via Cloudflare.

Purpose

Run after deploys or config changes to confirm:

1. /health returns 200 with {db, redis, app} shape
2. Login (Sanctum) returns a token
3. List courses (authenticated) returns 200 + data key
4. WebSocket upgrade (Reverb) returns 101
5. Horizon queue workers are running
6. Scheduler CronJob can complete a one-off run

Prerequisites

Requirement	How to get
kubectl context set to do-ams3-oep-prd-cluster	kubectl config current-context
jq	brew install jq (macOS) / apt install jq (Debian/Ubuntu)
GATEWAY_LB_IP	kubectl -n gateway get svc -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}'
SMOKE_PHONE	Test user phone number on the target tenant DB
SMOKE_PASSWORD	Test user password
SMOKE_REVERB_APP_KEY	doppler secrets get REVERB_APP_KEY --project oep --config <tenant-config> --plain

The script uses curl --resolve to hit the Gateway LB IP directly via GATEWAY_LB_IP. The --insecure flag (-k) is used because Cloudflare Origin CA certs are not trusted by system roots (traffic goes CF edge → Gateway, not browser → Gateway).

Sensitive variables

SMOKE_PHONE, SMOKE_PASSWORD, SMOKE_REVERB_APP_KEY, and GATEWAY_LB_IP must not be committed to version control. In local shells, avoid history exposure by prefixing commands with a space (requires HISTCONTROL=ignorespace) or using read -s to enter values interactively. In CI/CD pipelines, retrieve from Doppler or a secret manager rather than hardcoding in workflow files.

Run

# oep-stg example
export GATEWAY_LB_IP=$(kubectl --context do-ams3-oep-prd-cluster \
  -n gateway get svc \
  -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

export SMOKE_PHONE="01012345678"
export SMOKE_PASSWORD="your-test-password"
export SMOKE_REVERB_APP_KEY=$(doppler secrets get REVERB_APP_KEY \
  --project oep --config oep-stg --plain)

scripts/smoke.sh oep-stg
# Exit 0 = all checks pass
# Exit 1 = one or more checks failed (details printed with FAIL label)

Parity method

The smoke script asserts status code + JSON key shape, not value equality.

Check	Assertion
/health	HTTP 200 + {db, redis, app} keys present
Login	HTTP 200 + data.token non-empty
List courses	HTTP 200 + data key present
WebSocket	HTTP 101 Switching Protocols
Horizon	php artisan horizon:status returns "running" or "paused"
Scheduler	One-off job from CronJob Completes within 120s

Baseline capture

Capture a /health snapshot after a known-good deploy for reference comparison:

export GATEWAY_LB_IP=$(kubectl --context do-ams3-oep-prd-cluster \
  -n gateway get svc \
  -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

curl -sk --resolve "api-staging.unipuka.app:443:$GATEWAY_LB_IP" \
  https://api-staging.unipuka.app/health \
  | jq --arg ts "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" '. + {captured_at: $ts}' \
  > docs/audit/smoke-baseline-oep.json

The smoke script does not diff against this file at runtime - it is a human-readable reference for shape comparison.

Optional env overrides

Variable	Default	Description
KUBECONTEXT	do-ams3-oep-prd-cluster	kubectl context
RELEASE_NAME	<tenant>-laravel	Helm release name (e.g. oep-stg-laravel)

Troubleshooting

Check 1 fails (/health not 200): Pods not Running. kubectl get pods -n <tenant>. Check ExternalSecret sync (kubectl describe externalsecret -n <tenant>).

Check 2 fails (login): Test account may not exist in DO DB. Run migrations: kubectl -n <tenant> exec deploy/<release>-api -- php artisan db:seed --class=TestUserSeeder (if seeder exists).

Check 4 fails (WebSocket 101): Gateway :6001 listener may not be Accepted. Check: kubectl -n gateway describe gateway shared | grep -A5 6001.

Check 5 fails (Horizon status): Worker deployment not Running. Check logs: kubectl -n <tenant> logs deploy/<release>-worker --tail=50.

Check 6 fails (Scheduler job): CronJob may be suspended (suspend: true). Verify kubectl -n <tenant> get cronjob shows SUSPEND: False for oep-stg.

Regenerate baseline

After a planned schema change, re-run the baseline capture and commit.