# gcloud-lab GitOps + Terraform source of truth for a **GKE lab** I designed, ran, and then **shut down** once the GPU bill stopped being worth it. This is not a live cluster. It is the manifests, node-pool definitions, and GitOps wiring from a real environment that served vLLM inference, CNPG databases, and a handful of in-cluster apps. The same cost model that made the GPU pools scale to zero is why the whole footprint went to zero. **Canonical copy:** [forgejo.siriusdevops.com/sirius/gcloud-lab](https://forgejo.siriusdevops.com/sirius/gcloud-lab) --- ## What this is evidence of If you are reading this as a hiring screen, start here. Every claim below maps to a file in this repo. | Claim | Where to look | | --- | --- | | GKE cluster, custom VPC, dual-stack, Cilium datapath | [`modules/gke.tf`](modules/gke.tf), [`modules/vpc.tf`](modules/vpc.tf) | | CPU + three GPU node pools (L4, RTX PRO 6000, A100 80GB), all SPOT except CPU | [`modules/nodepool.tf`](modules/nodepool.tf), [`modules/nodepool-gpu.tf`](modules/nodepool-gpu.tf), [`modules/pro6000-nodepool.tf`](modules/pro6000-nodepool.tf), [`modules/a100-nodepool.tf`](modules/a100-nodepool.tf) | | Flux CD applies the tree; SOPS decrypts secrets in-cluster | [`clusters/devops-lab/`](clusters/devops-lab/), [`.sops.yaml`](.sops.yaml) | | Production-style **vLLM** OpenAI-compatible servers (not Ollama) | [`infrastructure/gpus/base/vllm-servers/`](infrastructure/gpus/base/vllm-servers/) | | KEDA HTTP scale-to-zero on the expensive GPUs | [`infrastructure/gpus/base/keda-gpu-scaling/`](infrastructure/gpus/base/keda-gpu-scaling/) | | CloudNative-PG operator + per-app Postgres | [`infrastructure/controllers/base/cnpg/`](infrastructure/controllers/base/cnpg/), `apps/base/customer1/*-db/` | | Gateway API (not legacy Ingress) + Tailscale for internals | [`infrastructure/gatewayapi/`](infrastructure/gatewayapi/), [`infrastructure/tailnet/`](infrastructure/tailnet/) | | Workload NetworkPolicies (trading stack) | [`apps/base/customer1/trading-platform/network-policies/`](apps/base/customer1/trading-platform/network-policies/) | I run 24/7 infrastructure now on a Raspberry Pi (Forgejo, Cloudflare tunnel, containerized sites). This repo is the cloud chapter that came before that. --- ## Architecture ``` Git (this repo, branch master) │ Flux source-controller (1m) │ ┌─────────────────┼──────────────────┐ ▼ ▼ ▼ infra-controllers infra-gpus apps/staging CNPG / KEDA vLLM + KEDA customer1 overlay cert-manager HTTPScaledObject (kustomize) Tailscale L4 / RTX6000 / A100 kube-prometheus │ │ │ └──────────── GKE us-central1-a ─────┘ │ ┌───────────────┬────────────┼────────────┬──────────────┐ ▼ ▼ ▼ ▼ ▼ e2-standard-2 g2-standard-8 g4-standard-48 a2-ultragpu-1g CPU pool L4 SPOT RTX PRO 6000 A100 80GB SPOT 1–5 nodes 1 node SPOT 0–1 SPOT 0–1 (always on) KEDA 0↔1 KEDA 0↔1 ``` Datapath: GKE `ADVANCED_DATAPATH` + `enable_cilium_clusterwide_network_policy`. Dual-stack VPC (`10.0.0.0/16`, pods `192.168.32.0/20`, services `192.168.16.0/24`). GPU nodes are tainted (`nvidia.com/gpu…=present:NoSchedule`) so only inference pods land on them. --- ## GPU inference (vLLM) Three independent OpenAI-compatible servers, each pinned to a pool via `nodeSelector` + matching taint/toleration. Model weights cached on PVC so a scale-up does not re-pull 20–40 GB from Hugging Face. ### L4 dispatcher — always on [`infrastructure/gpus/base/vllm-servers/vllm-l4.yaml`](infrastructure/gpus/base/vllm-servers/vllm-l4.yaml) - Machine: `g2-standard-8` + `nvidia-l4`, SPOT - Image: `vllm/vllm-openai` - Model: `p-e-w/Qwen3-8B-heretic` - Flags that matter: `--kv-cache-dtype=fp8`, `--enable-chunked-prefill`, `--enable-prefix-caching`, `--enable-auto-tool-choice`, `--tool-call-parser=hermes`, `--max-model-len=32768` Cheap, tool-capable, left running so agents had a low-latency brain even when the big cards were scaled out. ### RTX PRO 6000 — deep context, scale to zero [`infrastructure/gpus/base/vllm-servers/rtx6000-vllm.yaml`](infrastructure/gpus/base/vllm-servers/rtx6000-vllm.yaml) [`infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml`](infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml) - Machine: `g4-standard-48` + `nvidia-rtx-pro-6000`, SPOT, **min 0 / max 1** - Image: `vllm/vllm-openai:latest-cu129-ubuntu2404` - Model: `edp1096/Huihui-Qwen3.6-27B-abliterated-FP8` - `--max-model-len=262144`, `--enable-chunked-prefill`, `--tool-call-parser=qwen3_xml`, `--reasoning-parser=qwen3`, MTP speculative decoding (`num_speculative_tokens: 2`) - Startup / liveness / readiness probes on `/health` and `/v1/models` (model load is slow; `failureThreshold: 60` on startup) - KEDA `HTTPScaledObject`: `replicas.min: 0`, `max: 1`, host `rtx6000-brain-service.customer1.svc.cluster.local` Idle deep-thinker capacity cost nothing. A request woke the Deployment; GKE then scaled the node pool off zero. ### A100 80GB — same pattern [`infrastructure/gpus/base/vllm-servers/a100-vllm.yaml`](infrastructure/gpus/base/vllm-servers/a100-vllm.yaml) [`modules/a100-nodepool.tf`](modules/a100-nodepool.tf) - Machine: `a2-ultragpu-1g` + `nvidia-a100-80gb`, SPOT, min 0 - NVFP4 / ModelOpt quantized 27B, `--quantization=modelopt`, `--attention-backend=flash_attn`, `--kv-cache-dtype=fp8` --- ## GitOps Flux watches `master` and applies layered Kustomizations: ``` clusters/devops-lab/ flux-system/ Flux controllers + GitRepository infra-controllers.yaml → infrastructure/controllers/staging CNPG 0.26.1, KEDA ≥2.14, cert-manager, Tailscale, kube-prometheus infra-gpus.yaml → infrastructure/gpus/staging infra-gatewayapi.yaml → Gateway API + HTTPRoutes customer1-strimzi.yaml → Strimzi (Kafka) first customer1.yaml → apps/staging/customer1 (dependsOn strimzi) ``` Every Flux Kustomization has `decryption.provider: sops` and `secretRef: sops-age`. Encrypted `data`/`stringData` in Git; Flux decrypts at apply time. Age recipient is in [`.sops.yaml`](.sops.yaml). The private key is **not** in this repo. Bootstrap was originally: ```bash flux bootstrap github \ --owner=sirius0xdev \ --repository=gcloud-lab \ --branch=master \ --path=clusters/devops-lab ``` `gotk-sync.yaml` still records that GitHub URL. This Forgejo copy is the archive; the cluster is gone, so the GitRepository object was never re-pointed. --- ## Terraform (cluster birth) [`modules/`](modules/) is the IaC that created the cluster, not the day-to-day delivery path. | File | Resource | | --- | --- | | `gke.tf` | `devops-lab-cluster` in `us-central1-a`, default pool removed, Cilium datapath, dual-stack IP policy | | `vpc.tf` | `devops-lab-network` / `devops-lab-subnetwork`, ULA IPv6, secondary ranges | | `nodepool.tf` | CPU `e2-standard-2`, 1–5, pd-standard 100 Gi | | `nodepool-gpu.tf` | L4 SPOT `g2-standard-8`, autoscaling 1–1, GPU taint | | `pro6000-nodepool.tf` | RTX PRO 6000 SPOT `g4-standard-48`, **0–1**, hyperdisk-balanced | | `a100-nodepool.tf` | A100 80GB SPOT `a2-ultragpu-1g`, **0–1**, pd-ssd 200 Gi | | `db-bucket.tf` | GCS bucket for CNPG backups | | `providers.tf` | Google provider, Helm talking to the cluster via `gke-gcloud-auth-plugin` | State files and `*.tfvars` are gitignored. Do not `terraform apply` this against a live billing account unless you intend to recreate it. --- ## Data plane and apps Flux overlay: [`apps/staging/customer1/kustomization.yaml`](apps/staging/customer1/kustomization.yaml) | Piece | Role | | --- | --- | | CloudNative-PG | Operator `0.26.1`; per-app clusters (Hermes memory, waitlist, n8n, trading) with GCS backup | | Strimzi | Kafka for the trading data path; Flux `dependsOn` so apps wait for the operator | | Redis | In-cluster cache next to the trading services | | Trading platform | data / execute / news / dashboard Deployments, HTTPRoutes, NetworkPolicies | | Hermes agent | In-cluster agent + CNPG, talking to the vLLM services | | News bot | CronJobs: scrape → analyst (vLLM) → Telegram | | Waitlist API | FastAPI + CNPG, Gateway HTTPRoute | | n8n | Workflow engine on its own Postgres | | Gateway API | Public HTTPRoutes; Grafana/Prometheus stayed off the public internet (Tailscale / port-forward) | App **images** were built in a separate code repo and pulled from GHCR. This repo is manifests only (after cleanup: no Dockerfiles, no Helm-of-the-app, no planning markdown). --- ## Networking and security - **Cilium** as GKE datapath, clusterwide NetworkPolicy enabled from Terraform. - **Gateway API** `HTTPRoute` for public paths; internals not published. - **Tailscale operator** for operator access to Grafana and cluster services without a public LB. - **NetworkPolicies** on the trading namespace: DNS, Postgres, Redis, Kafka, in-namespace HTTP, egress HTTPS to market APIs. Everything else denied. - **SOPS + age** for Secrets in Git. Flux `sops-age` Secret in `flux-system` held the private key. - GPU nodes tainted so a random Deployment cannot schedule onto a $2+/hr card. --- ## Cost model (why it was torn down) This was the whole point of the GPU design: 1. L4 SPOT stayed at 1 node — cheap enough to keep a dispatcher warm. 2. RTX 6000 and A100 pools **autoscaled 0–1**. KEDA HTTPScaledObject set the Deployment to 0 when there was no traffic; the node pool followed. 3. Weights on PVC, long startup probes — first request after idle paid a cold-start, not a 40 GB pull. 4. CronJobs for batch work instead of idle inference pods. 5. When even the L4 + control-plane bill stopped making sense, the cluster was destroyed. Scale-to-zero was the rehearsal for scale-to-nothing. --- ## Repository layout ``` gcloud-lab/ ├── modules/ Terraform: VPC, GKE, node pools, GCS ├── clusters/devops-lab/ Flux entry (GitRepository + Kustomizations) ├── infrastructure/ │ ├── controllers/ CNPG, KEDA, cert-manager, Tailscale, Prometheus │ ├── gpus/ vLLM Deployments + KEDA HTTPScaledObjects │ ├── gatewayapi/ Gateway + HTTPRoutes │ └── tailnet/ Tailscale ProxyGroup ├── apps/ │ ├── base/customer1/ Namespaced workloads (kustomize) │ ├── base/osint-dashboard/ Helm chart used on this cluster │ └── staging/ Overlays Flux actually syncs ├── monitoring/ Extra Grafana/Prometheus config ├── .sops.yaml Age recipient for secret encryption ├── .github/workflows/ Historical GH Actions (pgvector image, etc.) └── mise.toml Local CLI pin (gcloud, kubectl, helm, sops, terraform, k9s) ``` --- ## Reading the code (suggested order) 1. [`modules/gke.tf`](modules/gke.tf) + [`modules/vpc.tf`](modules/vpc.tf) — cluster shape. 2. GPU pools, then the matching vLLM YAML — taint keys must match or the pod never schedules. 3. [`clusters/devops-lab/customer1.yaml`](clusters/devops-lab/customer1.yaml) — Flux `dependsOn`, SOPS, prune/force. 4. [`apps/staging/customer1/kustomization.yaml`](apps/staging/customer1/kustomization.yaml) — what actually shipped in `customer1`. 5. One NetworkPolicy under `apps/base/customer1/trading-platform/network-policies/` — default-deny thinking. --- ## Local tooling ```bash mise trust && mise install # kubectl, helm, sops, terraform, k9s, gcloud ``` Decrypt a secret locally (needs the age key, which is not in Git): ```bash sops -d apps/base/customer1/waitlist-api/waitlist-telegram-secret.yaml ``` --- ## Status | | | | --- | --- | | Cluster | **Destroyed** (GCP project `devops-lab-cluster`, GKE `devops-lab-cluster`, `us-central1-a`) | | This repo | Archive of what ran | | Live infra today | Self-hosted on a Pi — see [siriusdevops.com/lab](https://siriusdevops.com/lab) | Lance Walters — [siriusdevops.com](https://siriusdevops.com)