Network Policy

Default-deny NetworkPolicy per tenant namespace. Each namespace gets 5 standard NetworkPolicy resources + 1 CiliumNetworkPolicy covering all required traffic patterns.

Policy set per tenant namespace

Policy	Type	Rule
default-deny	Ingress + Egress	Deny everything
allow-intra-ns	Ingress + Egress	Pod-to-pod within same namespace
allow-dns	Egress	kube-dns on UDP/TCP 53 (namespace+pod selector)
allow-saas-https	Egress	0.0.0.0/0 on TCP 443, excluding RFC1918
allow-managed-db	Egress	Managed MySQL (25060) + Redis (25061) private IPs
allow-from-gateway	Ingress	CiliumNetworkPolicy - fromEntities: ingress (identity 8)

DNS allow design

Uses namespaceSelector + podSelector targeting k8s-app: kube-dns in kube-system. This survives kube-dns Pod IP churn (no ipBlock for DNS - Pod IPs rotate on restart). Canonical 2026 pattern for Kubernetes NetworkPolicy + CoreDNS.

SaaS egress design

Broad 0.0.0.0/0 TCP 443 excluding RFC1918. Covers Doppler, Sentry, Bugsnag, Cloudflare CDN, S3/Spaces, and any SaaS the app calls. Tighter FQDN-based egress via Cilium L7 network policies is a future phase.

Gateway ingress design

allow-from-gateway is a CiliumNetworkPolicy (not a standard K8s NetworkPolicy) because Cilium Gateway API embeds Envoy inside cilium-agent (no pods in the gateway namespace). Traffic from the Envoy proxy to tenant pods carries the reserved ingress Cilium identity (identity 8, reserved:ingress), not any namespace/pod label. This entity only exists in Cilium 1.14+ and can only be referenced via CiliumNetworkPolicy.fromEntities: [ingress].

A standard K8s namespaceSelector: gateway matches nothing (no pods there) and silently blocks all Gateway → pod traffic, causing 503 connection timeouts from the LB.

Verify with: cilium-dbg identity list | grep ingress → should show 8 reserved:ingress.

# Verify the CiliumNetworkPolicy is deployed
kubectl -n oep-stg get ciliumnetworkpolicy allow-from-gateway -o yaml

# Debug dropped packets via Hubble
hubble observe --namespace oep-stg --verdict DROPPED --type drop

Managed DB IPs

Private IPs from DO Managed MySQL + Valkey, pinned in platform/charts/network-policy/values.yaml and platform/charts/cloudflared/values.yaml.

These IPs are NOT permanently stable. DO Managed DB HA clusters promote the standby node on failover, which assigns the primary role to a different VPC IP. The private hostname continues to resolve correctly, but the underlying IP changes.

Update IPs when:

• A new tenant is added
• A managed DB is replaced
• A managed DB HA failover occurs (check by resolving the private hostname from inside the cluster - see incident response below)

Detect IP drift fast

# Resolve current IP from inside the cluster
kubectl run dns-test --rm -i --restart=Never --image=busybox -n <tenant> \
  -- nslookup private-<tenant>-valkey-do-user-34192537-0.i.db.ondigitalocean.com

# Compare against current NetworkPolicy
kubectl get networkpolicy allow-managed-db -n <tenant> -o jsonpath='{.spec.egress[1].to[0].ipBlock.cidr}'

If they differ, update values.yaml, push, and Argo CD will apply the fix within the sync interval.

Future: UNI-138 tracks replacing allow-managed-db with CiliumNetworkPolicy using toFQDNs so Cilium resolves hostnames dynamically and this manual update is never needed.

Cross-tenant deny verification

# Pod in oep-stg CANNOT reach pod in mansety-prd
kubectl -n mansety-prd run sink --image=traefik/whoami --restart=Never --port 80
kubectl -n oep-stg run probe --image=curlimages/curl --restart=Never -- sleep 600
SINK_IP=$(kubectl -n mansety-prd get pod sink -o jsonpath='{.status.podIP}')
kubectl -n oep-stg exec probe -- curl -sI --max-time 3 http://$SINK_IP || echo "blocked OK"
kubectl -n oep-stg delete pod probe; kubectl -n mansety-prd delete pod sink

Debug recipes

Pod can't reach external SaaS

# Check egress policies
kubectl -n oep-stg get networkpolicy allow-saas-https -o yaml

# Test from a probe pod
kubectl -n oep-stg run probe --image=curlimages/curl --restart=Never -- sleep 600
kubectl -n oep-stg exec probe -- curl -sI --max-time 5 https://api.doppler.com
kubectl -n oep-stg delete pod probe

Pod can't reach managed DB

# Verify DB IP in values matches actual
kubectl -n oep-stg get networkpolicy allow-managed-db -o yaml | grep cidr
# Compare with: dig +short private-oep-stg-mysql-do-user-34192537-0.a.db.ondigitalocean.com

# Test connectivity
kubectl -n oep-stg run probe --image=busybox --restart=Never -- sleep 600
kubectl -n oep-stg exec probe -- nc -vz 10.10.0.5 25060
kubectl -n oep-stg delete pod probe

cilium hubble (if installed)

hubble observe --namespace oep-stg --verdict DROPPED

Adding a new egress allow rule

Edit platform/charts/network-policy/templates/policies.yaml, add a new NetworkPolicy resource in the range $tenants block, then commit + push.

Known limits

• No FQDN-based egress allow without Cilium L7 (future phase).
• The broad allow-saas-https prevents L4 blocking of specific SaaS endpoints.
• Webhook traffic from the api (443/tcp to external) is covered by allow-saas-https.