Skip to content

Argo CD

Argo CD is the GitOps controller for the platform. All platform components and tenant workloads are deployed and reconciled via Argo CD watching unipuka-infra-ops.

Architecture

oep-infra/argocd.tf
  └─ helm_release "argo-cd" (chart 9.5.15, namespace argocd)
       └─ [first-time bootstrap only]
            kubectl apply -f bootstrap/root-application.yaml
              └─ bootstrap/platform-apps/*.yaml (Applications, AppProjects, ApplicationSets)
                    └─ platform/charts/<concern>/  (Helm chart per concern)
                    └─ apps/laravel/ (tenant laravel-app chart via ApplicationSet)

Note: The root Application is NOT a kubernetes_manifest in Terraform. It is applied once manually after helm_release.argocd installs the Application CRD. Argo CD then self-manages it via auto-sync. See bootstrap.md Step 5 for the full sequence.

  • Install: TF helm_release in oep-infra/argocd.tf. Pinned chart version. TF is source of truth for chart version + values. No bootstrap.sh.
  • App-of-Apps: root Application (bootstrap/root-application.yaml) points at bootstrap/platform-apps/. Each YAML in that directory is a child Application managing one platform concern (ESO, TLS secrets, quotas, kube-prom, gateway, loki, opencost).
  • ApplicationSet: reserved for tenant workloads (BD-6.2). Not used for platform.
  • server.insecure=true: argocd-server listens on plain HTTP; Gateway terminates TLS once at the front (BD-3.7). Avoids TLS-in-TLS.
  • Dex: enabled. Provides GitHub OAuth as the app-level login layer.

CLI access (port-forward)

The argocd CLI cannot connect directly through argocd.unipuka.app - gRPC-web calls return 404 through the Cloudflare proxy, and --sso fails because Dex discovery fetches from the public CF-protected URL. Use port-forward instead:

# Terminal 1 - keep running
kubectl --context do-ams3-oep-prd-cluster \
  -n argocd port-forward svc/argo-cd-argocd-server 8080:80

# Terminal 2 - login
ARGOCD_PASS=$(doppler secrets get ARGOCD_ADMIN_PASSWORD --plain --project oep --config base)
argocd login localhost:8080 \
  --username admin \
  --password "$ARGOCD_PASS" \
  --plaintext \
  --insecure
unset ARGOCD_PASS

# Verify
argocd app list

The port-forward must stay alive for the session. The argocd session is cached in ~/.config/argocd/config - subsequent commands do not require re-login while the port-forward is running.

Full procedure, troubleshooting, and password rotation: argocd-break-glass.md.

Admin password

Stored in Doppler oep/base as ARGOCD_ADMIN_PASSWORD. On fresh install, also available from the K8s secret (until rotated):

kubectl --context do-ams3-oep-prd-cluster -n argocd \
  get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo

Browser access

https://argocd.unipuka.app → CF Access challenge (GitHub login, unipuka org members only) → CF Access JWT issued → Gateway forwards request → Argo CD shows "Log in via GitHub" → Dex GitHub OAuth → Argo UI loads.

Sync waves overview

Wave Applications
-1 root (this app)
0 external-secrets, cluster-secret-stores
1 tls-secrets, resource-quotas, network-policy, kube-prometheus-stack
2 gateway
3 loki, opencost, argocd-httproute

Child Applications declare their wave via argocd.argoproj.io/sync-wave annotation.

AppProjects + Applications

Both AppProjects are committed in bootstrap/platform-apps/appprojects.yaml (sync-wave -1) and applied by the root App-of-Apps.

  • platform AppProject: cluster-scoped. All 12 platform Applications use this project. Permits all cluster-scoped resources (CRDs, ClusterRoles, GatewayClasses) and all namespace-scoped resources. Allowed destinations: argocd, external-secrets, monitoring, gateway, cloudflared.

  • tenants AppProject: namespace-scoped to oep-stg, mansety-prd, us-prd. No cluster-scoped resources permitted (no ClusterRole, no namespace creation - namespaces are pre-created by Terraform). Source repo restricted to unipuka-infra-ops. The laravel-app ApplicationSet generates one Application per tenant using this project.

# Verify both projects exist
argocd proj list
# expect: platform, tenants (plus default)

# Confirm tenants project denies cluster-scoped resources
argocd proj get tenants
# clusterResourceWhitelist: [] (empty = deny all cluster-scoped)

Log locations

# Application controller (reconciliation loop, sync errors)
kubectl -n argocd logs -l app.kubernetes.io/name=argocd-application-controller --tail=100

# Repo server (git fetch, manifest generation errors)
kubectl -n argocd logs -l app.kubernetes.io/name=argocd-repo-server --tail=100

# API server (UI/CLI auth, API errors)
kubectl -n argocd logs -l app.kubernetes.io/name=argocd-server --tail=100

Rollback

Rollback = git revert the offending commit in unipuka-infra-ops and push. Argo CD auto-syncs within the reconciliation interval (default 3 min) or force via:

argocd app sync root --force

For Argo CD itself (the helm_release): update version in argocd.tf and terraform apply -target=helm_release.argocd.

Auth via CF Access + GitHub OAuth (BD-3.14)

Two-layer authentication:

  1. CF Access (argocd.unipuka.app) - edge gate. Only @unipuka.com Google accounts or unipuka GitHub org members can reach the URL. Configured via TF cf_access_apps.tf.

  2. GitHub OAuth via Dex - app-level login. After CF Access passes the request, Argo CD shows a login form with "Log in via GitHub". Dex (built-in Argo CD component) handles the OAuth flow, restricts to unipuka org members.

GitHub OAuth App ("Unipuka Argo CD Dex")

  • Created at: github.com/organizations/unipuka/settings/applications
  • Callback URL: https://argocd.unipuka.app/api/dex/callback
  • Credentials in Doppler base: ARGOCD_GITHUB_OAUTH_CLIENT_ID + ARGOCD_GITHUB_OAUTH_CLIENT_SECRET
  • Synced into: argocd-secret keys dex.github.clientID + dex.github.clientSecret via ESO ExternalSecret argocd-oidc-secret in argocd ns

Dex config reference

# In argocd-cm ConfigMap (managed by argocd.tf helm values):
dex.config: |
  connectors:
    - type: github
      id: github
      name: GitHub
      config:
        clientID: $dex.github.clientID      # reads from argocd-secret
        clientSecret: $dex.github.clientSecret
        redirectURI: https://argocd.unipuka.app/api/dex/callback
        orgs:
          - name: unipuka

Note: use dot notation ($dex.github.*) not hyphenated names - Argo CD's CM parser stops at hyphens in $secret-name:key format.

Break-glass

See argocd-break-glass.md for port-forward access, admin password rotation, and GitHub OAuth troubleshooting.

Troubleshooting

Application stuck Syncing / OutOfSync

argocd app get <name> --show-operation
argocd app sync <name> --force
# Check for resource-level errors:
kubectl -n argocd describe application <name>

Repo credentials not working (clone fails)

# Verify Secret present with correct label
kubectl -n argocd get secret argocd-repo-creds-github \
  -o jsonpath='{.metadata.labels.argocd\.argoproj\.io/secret-type}'
# expect: repo-creds

# Check repo-server logs for authentication errors
kubectl -n argocd logs -l app.kubernetes.io/name=argocd-repo-server | grep -i "auth\|error\|fail"

# Trigger re-fetch of credentials (restart repo-server)
kubectl -n argocd rollout restart deployment argocd-repo-server

Application controller OOM

Increase memory limit in argocd.tf values:

controller = {
  resources = {
    limits = { memory = "2Gi" }
    requests = { memory = "1Gi" }
  }
}

Then terraform apply -target=helm_release.argocd.

laravel-app chart integration

Two-tier ApplicationSet pattern for tenant workloads:

Tier 1 - Secrets (sync-wave 1)

Direct Application manifests in bootstrap/platform-apps/<tenant>-secrets.yaml (one per tenant). Renders apps/secrets/ chart → ExternalSecret → ESO creates <tenant>-env K8s Secret. Root App waits for each *-secrets Application to be Synced/Healthy before wave 2.

Tier 2 - App (sync-wave 2)

bootstrap/platform-apps/laravel-applicationset.yaml generates one Application per tenant (oep-stg, mansety-prd, us-prd). Renders apps/laravel/ chart with per-tenant values.

Key patterns:

  • PreSync hook: templates/migrate/job.yaml runs migrate --force --isolated before each sync. The <tenant>-env secret is guaranteed to exist (created by wave 1). Job failure aborts the sync - pods stay on previous image.
  • Chart version: every deploy bumps Chart.yaml.version + Chart.yaml.appVersion + values-<tenant>.yaml.image.tag atomically (BD-5.3 lockstep contract).

See laravel-app.md for full chart runbook.