Argo CD Break Glass¶

Emergency access procedures when CF Access or GitHub OAuth is unavailable.

When to use¶

CF Access outage / misconfiguration → argocd.unipuka.app unreachable
GitHub OAuth app revoked / misconfigured → "Login via GitHub" fails
Need to reset Argo CD state or delete an application manually

Break glass: port-forward (bypasses CF Access)¶

This is the primary method for CLI access. The argocd CLI cannot connect directly through argocd.unipuka.app because:

gRPC-web POST calls return 404 through the Cloudflare proxy
--sso fails because Dex discovery fetches from the public CF-protected URL

# Terminal 1 - keep running
kubectl --context do-ams3-oep-prd-cluster \
  -n argocd port-forward svc/argo-cd-argocd-server 8080:80

# Terminal 2 - get admin password
ARGOCD_PASS=$(doppler secrets get ARGOCD_ADMIN_PASSWORD --plain --project oep --config base)
# Fallback if not yet in Doppler (initial install):
# ARGOCD_PASS=$(kubectl --context do-ams3-oep-prd-cluster -n argocd \
#   get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)

argocd login localhost:8080 \
  --username admin \
  --password "$ARGOCD_PASS" \
  --plaintext \
  --insecure

unset ARGOCD_PASS

# Verify access
argocd app list

The port-forward stays alive for the duration of the session. Subsequent argocd commands use the cached session in ~/.config/argocd/config and do not require re-login.

CLI via cloudflared (informational - currently non-functional)¶

This flow is documented for future reference once BD-3.14 (CF Access OIDC) is completed. At that point the ArgoCD server will validate the CF Access JWT directly (no Dex), and the CLI flow will be:

cloudflared access login https://argocd.unipuka.app
TOKEN=$(cloudflared access token --app argocd.unipuka.app)
argocd login argocd.unipuka.app \
  --header "cf-access-token: $TOKEN" \
  --grpc-web
argocd app list

Until then, use port-forward above.

Rotate admin password¶

# Generate new password
NEW_PWD=$(openssl rand -base64 32 | tr -d '/+=' | head -c 32)
NEW_BCRYPT=$(htpasswd -bnBC 10 "" "$NEW_PWD" | tr -d ':\n' | sed 's/\$2y/\$2a/')
TMPFILE=$(mktemp); echo "x:$NEW_BCRYPT" > "$TMPFILE"
htpasswd -bvB "$TMPFILE" x "$NEW_PWD" && rm -f "$TMPFILE"

# Store
doppler secrets set --project oep --config base \
  "ARGOCD_ADMIN_PASSWORD=$NEW_PWD" \
  "ARGOCD_ADMIN_PASSWORD_BCRYPT=$NEW_BCRYPT"
unset NEW_PWD NEW_BCRYPT

# Force ESO re-sync (if argocd-secret is managed by ESO)
kubectl --context do-ams3-oep-prd-cluster -n argocd \
  annotate externalsecret argocd-oidc-secret \
  force-sync=$(date +%s) --overwrite

Symptoms: clicking "Login via GitHub" redirects to GitHub with a malformed client_id.

Check Dex connector config:

kubectl --context do-ams3-oep-prd-cluster -n argocd get cm argocd-cm \
  -o jsonpath='{.data.dex\.config}'
# Expected: clientID: $dex.github.clientID (dot notation, not hyphenated)

Check argocd-secret has dex keys:

kubectl --context do-ams3-oep-prd-cluster -n argocd get secret argocd-secret \
  -o jsonpath='{.data}' | jq -r 'keys[]' | grep dex
# Expected: dex.github.clientID, dex.github.clientSecret

Force ESO re-sync if keys are missing:

kubectl --context do-ams3-oep-prd-cluster -n argocd \
  annotate externalsecret argocd-oidc-secret \
  force-sync=$(date +%s) --overwrite
sleep 15
kubectl --context do-ams3-oep-prd-cluster -n argocd rollout restart \
  deployment argo-cd-argocd-dex-server

Rotate GitHub OAuth App credentials:

GitHub → org settings → OAuth Apps → "Unipuka Argo CD Dex" → Generate new client secret
Store in Doppler: doppler secrets set --project oep --config base "ARGOCD_GITHUB_OAUTH_CLIENT_SECRET=<new>"
Force ESO sync + restart Dex (see above)

Troubleshoot: Argo CD unreachable at argocd.unipuka.app¶

Check CF Access app exists: Zero Trust → Access → Applications → "Argo CD"
Check DNS: dig +short argocd.unipuka.app → should return CF proxy IPs
Check Gateway listener: kubectl --context do-ams3-oep-prd-cluster -n gateway describe gateway shared | grep argocd
Check HTTPRoute: kubectl --context do-ams3-oep-prd-cluster -n gateway get httproute argocd
Fall back to port-forward (see above)