Bootstrap

First-time setup: provision namespaces, secrets, Argo CD, and trigger initial sync.

Purpose

Run once per cluster lifecycle (or after full teardown + rebuild). Gets the cluster from empty DOKS to a state where Argo CD is running and all Applications are Synced/Healthy.

When to use

• First deploy to a new cluster
• Full cluster rebuild after teardown
• After cluster upgrade that requires re-bootstrapping

---

Step 0: Prerequisites (BD-3.0)

Run this before any other step. Provisions secrets and verifies tooling assumed by all BD-3.X tickets.

0.1 Cloudflare wildcard Origin certs (3 zones)

Create in CF dashboard: SSL/TLS -> Origin Server -> Create Certificate for each zone:

Zone	Hostnames
unipuka.app	*.unipuka.app, unipuka.app
mansety.com	*.mansety.com, mansety.com
us-academy.net	*.us-academy.net, us-academy.net

Validity: 15 years. Upload to Doppler oep.base as CF_ORIGIN_CERT_<ZONE> / CF_ORIGIN_KEY_<ZONE> (see populate-doppler.md for the upload recipe). Shred local PEM files after upload.

0.2 DO Spaces Loki buckets + access keys

Buckets managed in oep-infra/spaces_logs.tf (BD-3.12). Create via TF:

cd unipuka-infra-do
direnv exec oep-infra doppler run --project oep --config base -- \
  terraform -chdir=oep-infra apply \
  -target=digitalocean_spaces_bucket.loki_oep_stg \
  -target=digitalocean_spaces_bucket.loki_mansety_prd \
  -target=digitalocean_spaces_bucket.loki_us_prd \
  -auto-approve

Then create scoped access keys per bucket and store in Doppler base: LOKI_SPACES_KEY_<ENV> + LOKI_SPACES_SECRET_<ENV> for OEP_STG, MANSETY_PRD, US_PRD.

0.3 Admin passwords

Generate and store in Doppler oep.base:

# Argo CD (plain + bcrypt)
ARGOCD_PWD=$(openssl rand -base64 32 | tr -d '/+=' | head -c 32)
ARGOCD_BCRYPT=$(htpasswd -bnBC 10 "" "$ARGOCD_PWD" | tr -d ':\n' | sed 's/\$2y/\$2a/')
TMPFILE=$(mktemp); echo "x:$ARGOCD_BCRYPT" > "$TMPFILE"
htpasswd -bvB "$TMPFILE" x "$ARGOCD_PWD" && rm -f "$TMPFILE"
doppler secrets set --project oep --config base \
  "ARGOCD_ADMIN_PASSWORD=$ARGOCD_PWD" "ARGOCD_ADMIN_PASSWORD_BCRYPT=$ARGOCD_BCRYPT"
unset ARGOCD_PWD ARGOCD_BCRYPT

# Grafana
GRAFANA_PWD=$(openssl rand -base64 32 | tr -d '/+=' | head -c 32)
doppler secrets set --project oep --config base "GRAFANA_ADMIN_PASSWORD=$GRAFANA_PWD"
unset GRAFANA_PWD

0.4 CF team name + Google Workspace IdP UUID

CF_TOKEN=$(doppler secrets get CLOUDFLARE_API_TOKEN --plain --project oep --config base)
CF_ACCT="445333d135773573989007b17be102b5"
TEAM=$(curl -s -H "Authorization: Bearer $CF_TOKEN" \
  "https://api.cloudflare.com/client/v4/accounts/$CF_ACCT/access/organizations" \
  | jq -r '.result.auth_domain' | sed 's/\.cloudflareaccess\.com$//')
IDP_UUID=$(curl -s -H "Authorization: Bearer $CF_TOKEN" \
  "https://api.cloudflare.com/client/v4/accounts/$CF_ACCT/access/identity_providers" \
  | jq -r '.result[] | select(.type=="google-apps") | .id')
doppler secrets set --project oep --config base "CF_TEAM_NAME=$TEAM" "CF_ACCESS_GOOGLE_IDP_UUID=$IDP_UUID"
unset CF_TOKEN TEAM IDP_UUID

0.5 Verify CF API token scope

Token must include Account.Access: Apps and Policies:Edit. Audit in CF dashboard -> My Profile -> API Tokens -> existing TF token -> Edit -> add scope if missing.

0.6 Verify Gateway API CRDs

kubectl get crd | grep gateway.networking.k8s.io | wc -l   # expect >= 5
kubectl get gatewayclass cilium -o jsonpath='{.status.conditions[?(@.type=="Accepted")].status}'
# expect: True

0.7 Toolchain

which kubectl direnv doctl doppler cloudflared htpasswd
doctl kubernetes cluster kubeconfig save oep-prd-cluster
direnv allow unipuka-infra-do/oep-infra

BD-4 additions (required for Helm chart work):

# Install
brew install helm chart-testing kubeconform yq composer php@8.2

# Pin helm via asdf (workspace-level)
asdf install helm 3.16.0
asdf local helm 3.16.0  # run from unipuka-infra-ops/

# Verify
helm version --short        # >= v3.16.0
ct version                  # >= 3.10.x
kubeconform -v
yq --version                # >= 4.x
php -v | head -1            # PHP 8.2.x
composer -V                 # 2.x

Step 0 exit gate

# All 17 keys present
doppler secrets --project oep --config base --only-names 2>/dev/null | grep -c "CF_ORIGIN\|LOKI_SPACES\|ARGOCD_ADMIN\|GRAFANA_ADMIN\|CF_TEAM\|CF_ACCESS_GOOGLE"
# expect: 17

# Certs valid
for Z in UNIPUKA_APP MANSETY_COM US_ACADEMY_NET; do
  doppler secrets get CF_ORIGIN_CERT_$Z --plain --project oep --config base \
    | openssl x509 -noout -dates 2>&1 | grep notAfter
done

# Cluster running
doctl kubernetes cluster get oep-prd-cluster -o json | jq -r '.[0].status.state'
# expect: running

---

Step 1: Generate Doppler admin token

The Doppler Terraform provider needs a Personal Token (or Service Account token on Team/Enterprise plan) to create and manage the oep project structure. Service Tokens are read-only and tied to a single config - they cannot create projects or environments.

Generate a Personal Token (all plans)

1. Log in at dashboard.doppler.com
2. Click your avatar (top-right) → Personal Settings
3. Select Tokens in the left sidebar
4. Click Generate token → name it terraform-oep-infra → click Generate

Copy the token value. It is shown only once.

export TF_VAR_doppler_admin_token="dp.pt.xxxx..."

Generate a Service Account token (Team/Enterprise plans - preferred)

Service Accounts are dedicated machine identities - not tied to any person.

1. Go to Workplace Settings (cog icon, top-left) → Service Accounts
2. Click Add Service Account → name it terraform-oep-infra
3. Set Workplace Role to Workplace Admin
4. Generate a token under the service account → copy the value

export TF_VAR_doppler_admin_token="dp.sa.xxxx..."

For CI: add the token value as a GitHub org secret named DOPPLER_ADMIN_TOKEN.

---

Step 2: Generate Doppler service tokens

Three read-only Service Tokens, one per tenant environment. Used by ESO ClusterSecretStore to sync secrets from Doppler into Kubernetes Secrets.

1. In dashboard.doppler.com, open project oep
2. For each config (oep-stg, mansety-prd, us-prd):
3. Click the config name → Access tab → Service Tokens → Generate
4. Name: k8s-<env>-<YYYY-MM> (e.g. k8s-oep-stg-2026-05)
5. Scope: Read-Only
6. Copy the token (shown once)

Store each token in three places:

Location	Name	Purpose
GH repo secret (unipuka/soa)	DOPPLER_SERVICE_TOKEN_OEP_STG	Used by soa CI workflows (doppler run)
GH org secret	DOPPLER_SERVICE_TOKEN_OEP_STG	Passed to TF plan/apply as TF_VAR_doppler_service_token_oep_stg
Doppler dashboard	(source of truth)	Rotation reference

export TF_VAR_doppler_service_token_oep_stg="dp.st.xxxx..."
export TF_VAR_doppler_service_token_mansety_prd="dp.st.xxxx..."
export TF_VAR_doppler_service_token_us_prd="dp.st.xxxx..."

For CI: add DOPPLER_ADMIN_TOKEN, DOPPLER_SERVICE_TOKEN_OEP_STG, DOPPLER_SERVICE_TOKEN_MANSETY_PRD, DOPPLER_SERVICE_TOKEN_US_PRD as GitHub org secrets. These are mapped to TF_VAR_* in the reusable plan/apply workflows.

Rotation runbook: rotate-secrets.md

---

Step 3: Set up Cloudflare Access IdPs (first-time only)

Before running terraform apply for the first time, configure the Cloudflare Zero Trust Identity Providers and extend the Cloudflare API token scope. This is required for docs_site.tf to apply without errors.

See Cloudflare Access IdP Setup for full instructions.

---

Step 4: Create Argo CD GitHub App

Argo CD uses a GitHub App (not a PAT) to clone unipuka-infra-ops. GitHub Apps have no expiry and support fine-grained repo permissions.

4.1 Create the app

1. Go to github.com/organizations/unipuka/settings/apps/new
2. Fill in:
3. GitHub App name: unipuka-argocd
4. Homepage URL: https://argocd.unipuka.app
5. Webhook: uncheck "Active" (Argo CD uses polling, not webhooks)
6. Repository permissions: Contents = Read-only, Metadata = Read-only
7. Where can this be installed: Only on this account
8. Click Create GitHub App

4.2 Capture App ID and generate private key

After creation:

1. Copy the App ID from the app settings page (numeric, e.g. 12345678)
2. Scroll to Private keys → Generate a private key
3. Download the .pem file

4.3 Install the app

1. On the app settings page → Install App → install on the unipuka org
2. Select Only select repositories → choose unipuka-infra-ops
3. Click Install
4. After install, the URL ends with .../installations/<installationId> - copy the number

4.4 Export variables

export TF_VAR_argocd_github_app_id="12345678"
export TF_VAR_argocd_github_app_installation_id="987654321"
export TF_VAR_argocd_github_app_private_key="$(cat /path/to/unipuka-argocd.pem)"

For CI: add ARGOCD_GITHUB_APP_ID, ARGOCD_GITHUB_APP_INSTALLATION_ID, ARGOCD_GITHUB_APP_PRIVATE_KEY as GitHub org secrets.

---

Step 5: Run terraform apply

Prerequisites:

• [ ] doctl kubernetes cluster kubeconfig save oep-prd-cluster - kubeconfig set
• [ ] doppler_admin_token exported (Step 1)
• [ ] 3x Doppler service tokens exported (Step 2)
• [ ] Cloudflare Access IdPs configured + UUIDs in Doppler (Step 3)
• [ ] cf_account_id filled in tenants.tf (from Step 3)
• [ ] Argo CD GitHub App vars exported (Step 4)
• [ ] terraform.tfvars populated (copy from terraform.tfvars.example)
• [ ] .envrc populated with Spaces backend creds (direnv allow)

First-time bootstrap (Argo CD two-step apply)

The kubernetes_manifest resource is intentionally not used for the root Application - the Application CRD doesn't exist at plan time on a fresh cluster. Bootstrap sequence:

cd unipuka-infra-do

# Step A: install Argo CD helm release (installs Application CRD + all pods)
doppler run --project oep --config base -- \
  terraform -chdir=oep-infra apply -target=helm_release.argocd -auto-approve

# Wait for all argocd pods to be Running
kubectl --context do-ams3-oep-prd-cluster -n argocd get pods -w

# Step B: apply root Application (one-time manual bootstrap)
kubectl --context do-ams3-oep-prd-cluster apply \
  -f ../unipuka-infra-ops/bootstrap/root-application.yaml

# Step C: apply remaining TF resources
doppler run --project oep --config base -- \
  terraform -chdir=oep-infra apply

Subsequent applies: single terraform apply.

Subsequent applies are a single terraform apply.

# Normal (post-bootstrap) apply
doppler run --project oep --config base -- terraform apply

---

Step 6: Verify

# Namespaces with labels
kubectl get ns --show-labels

# ESO ClusterSecretStores valid
kubectl get clustersecretstores

# DOCR pull secret present in all tenant namespaces
for ns in oep-stg mansety-prd us-prd external-secrets monitoring; do
  kubectl get secret do-registry -n $ns -o name
done

# Argo CD pods running
kubectl -n argocd get pods

# Argo CD root Application Synced/Healthy
kubectl -n argocd get application root

# Argo CD reachable via port-forward (service name from helm_release name "argo-cd")
kubectl --context do-ams3-oep-prd-cluster \
  -n argocd port-forward svc/argo-cd-argocd-server 8080:80
# Login: argocd login localhost:8080 --username admin \
#   --password "$(doppler secrets get ARGOCD_ADMIN_PASSWORD --plain --project oep --config base)" \
#   --plaintext --insecure

# Repo creds picked up
kubectl -n argocd get cm argocd-cm -o jsonpath='{.data.repositories}'

# All applications healthy (after platform apps land)
argocd app list

---

Rollback

TF bootstrap is additive - existing cluster is unchanged.

• Remove specific resources: terraform destroy -target=kubernetes_namespace.tenants
• Argo CD removal: helm uninstall argocd -n argocd

V&V

• argocd app list shows root, platform - both Synced/Healthy
• Subsequent terraform apply produces an empty diff
• kubectl get ns lists all 6 namespaces with labels
• kubectl get clustersecretstores shows 3 stores all Valid