4f235b52f4
Drop duplicate app/Helm trees, planning notes, and the vwap-monitor source tree so this repo is GitOps + Terraform only. Document the GKE cluster as shut down, with file-level pointers for vLLM, KEDA scale-to-zero, Flux, and SOPS.
12 KiB
gcloud-lab
GitOps + Terraform source of truth for a GKE lab I designed, ran, and then shut down once the GPU bill stopped being worth it.
This is not a live cluster. It is the manifests, node-pool definitions, and GitOps wiring from a real environment that served vLLM inference, CNPG databases, and a handful of in-cluster apps. The same cost model that made the GPU pools scale to zero is why the whole footprint went to zero.
Canonical copy: forgejo.siriusdevops.com/sirius/gcloud-lab
What this is evidence of
If you are reading this as a hiring screen, start here. Every claim below maps to a file in this repo.
| Claim | Where to look |
|---|---|
| GKE cluster, custom VPC, dual-stack, Cilium datapath | modules/gke.tf, modules/vpc.tf |
| CPU + three GPU node pools (L4, RTX PRO 6000, A100 80GB), all SPOT except CPU | modules/nodepool.tf, modules/nodepool-gpu.tf, modules/pro6000-nodepool.tf, modules/a100-nodepool.tf |
| Flux CD applies the tree; SOPS decrypts secrets in-cluster | clusters/devops-lab/, .sops.yaml |
| Production-style vLLM OpenAI-compatible servers (not Ollama) | infrastructure/gpus/base/vllm-servers/ |
| KEDA HTTP scale-to-zero on the expensive GPUs | infrastructure/gpus/base/keda-gpu-scaling/ |
| CloudNative-PG operator + per-app Postgres | infrastructure/controllers/base/cnpg/, apps/base/customer1/*-db/ |
| Gateway API (not legacy Ingress) + Tailscale for internals | infrastructure/gatewayapi/, infrastructure/tailnet/ |
| Workload NetworkPolicies (trading stack) | apps/base/customer1/trading-platform/network-policies/ |
I run 24/7 infrastructure now on a Raspberry Pi (Forgejo, Cloudflare tunnel, containerized sites). This repo is the cloud chapter that came before that.
Architecture
Git (this repo, branch master)
│
Flux source-controller (1m)
│
┌─────────────────┼──────────────────┐
▼ ▼ ▼
infra-controllers infra-gpus apps/staging
CNPG / KEDA vLLM + KEDA customer1 overlay
cert-manager HTTPScaledObject (kustomize)
Tailscale L4 / RTX6000 / A100
kube-prometheus
│ │ │
└──────────── GKE us-central1-a ─────┘
│
┌───────────────┬────────────┼────────────┬──────────────┐
▼ ▼ ▼ ▼ ▼
e2-standard-2 g2-standard-8 g4-standard-48 a2-ultragpu-1g
CPU pool L4 SPOT RTX PRO 6000 A100 80GB SPOT
1–5 nodes 1 node SPOT 0–1 SPOT 0–1
(always on) KEDA 0↔1 KEDA 0↔1
Datapath: GKE ADVANCED_DATAPATH + enable_cilium_clusterwide_network_policy. Dual-stack VPC (10.0.0.0/16, pods 192.168.32.0/20, services 192.168.16.0/24). GPU nodes are tainted (nvidia.com/gpu…=present:NoSchedule) so only inference pods land on them.
GPU inference (vLLM)
Three independent OpenAI-compatible servers, each pinned to a pool via nodeSelector + matching taint/toleration. Model weights cached on PVC so a scale-up does not re-pull 20–40 GB from Hugging Face.
L4 dispatcher — always on
infrastructure/gpus/base/vllm-servers/vllm-l4.yaml
- Machine:
g2-standard-8+nvidia-l4, SPOT - Image:
vllm/vllm-openai - Model:
p-e-w/Qwen3-8B-heretic - Flags that matter:
--kv-cache-dtype=fp8,--enable-chunked-prefill,--enable-prefix-caching,--enable-auto-tool-choice,--tool-call-parser=hermes,--max-model-len=32768
Cheap, tool-capable, left running so agents had a low-latency brain even when the big cards were scaled out.
RTX PRO 6000 — deep context, scale to zero
infrastructure/gpus/base/vllm-servers/rtx6000-vllm.yaml
infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml
- Machine:
g4-standard-48+nvidia-rtx-pro-6000, SPOT, min 0 / max 1 - Image:
vllm/vllm-openai:latest-cu129-ubuntu2404 - Model:
edp1096/Huihui-Qwen3.6-27B-abliterated-FP8 --max-model-len=262144,--enable-chunked-prefill,--tool-call-parser=qwen3_xml,--reasoning-parser=qwen3, MTP speculative decoding (num_speculative_tokens: 2)- Startup / liveness / readiness probes on
/healthand/v1/models(model load is slow;failureThreshold: 60on startup) - KEDA
HTTPScaledObject:replicas.min: 0,max: 1, hostrtx6000-brain-service.customer1.svc.cluster.local
Idle deep-thinker capacity cost nothing. A request woke the Deployment; GKE then scaled the node pool off zero.
A100 80GB — same pattern
infrastructure/gpus/base/vllm-servers/a100-vllm.yaml
modules/a100-nodepool.tf
- Machine:
a2-ultragpu-1g+nvidia-a100-80gb, SPOT, min 0 - NVFP4 / ModelOpt quantized 27B,
--quantization=modelopt,--attention-backend=flash_attn,--kv-cache-dtype=fp8
GitOps
Flux watches master and applies layered Kustomizations:
clusters/devops-lab/
flux-system/ Flux controllers + GitRepository
infra-controllers.yaml → infrastructure/controllers/staging
CNPG 0.26.1, KEDA ≥2.14, cert-manager, Tailscale, kube-prometheus
infra-gpus.yaml → infrastructure/gpus/staging
infra-gatewayapi.yaml → Gateway API + HTTPRoutes
customer1-strimzi.yaml → Strimzi (Kafka) first
customer1.yaml → apps/staging/customer1 (dependsOn strimzi)
Every Flux Kustomization has decryption.provider: sops and secretRef: sops-age. Encrypted data/stringData in Git; Flux decrypts at apply time. Age recipient is in .sops.yaml. The private key is not in this repo.
Bootstrap was originally:
flux bootstrap github \
--owner=sirius0xdev \
--repository=gcloud-lab \
--branch=master \
--path=clusters/devops-lab
gotk-sync.yaml still records that GitHub URL. This Forgejo copy is the archive; the cluster is gone, so the GitRepository object was never re-pointed.
Terraform (cluster birth)
modules/ is the IaC that created the cluster, not the day-to-day delivery path.
| File | Resource |
|---|---|
gke.tf |
devops-lab-cluster in us-central1-a, default pool removed, Cilium datapath, dual-stack IP policy |
vpc.tf |
devops-lab-network / devops-lab-subnetwork, ULA IPv6, secondary ranges |
nodepool.tf |
CPU e2-standard-2, 1–5, pd-standard 100 Gi |
nodepool-gpu.tf |
L4 SPOT g2-standard-8, autoscaling 1–1, GPU taint |
pro6000-nodepool.tf |
RTX PRO 6000 SPOT g4-standard-48, 0–1, hyperdisk-balanced |
a100-nodepool.tf |
A100 80GB SPOT a2-ultragpu-1g, 0–1, pd-ssd 200 Gi |
db-bucket.tf |
GCS bucket for CNPG backups |
providers.tf |
Google provider, Helm talking to the cluster via gke-gcloud-auth-plugin |
State files and *.tfvars are gitignored. Do not terraform apply this against a live billing account unless you intend to recreate it.
Data plane and apps
Flux overlay: apps/staging/customer1/kustomization.yaml
| Piece | Role |
|---|---|
| CloudNative-PG | Operator 0.26.1; per-app clusters (Hermes memory, waitlist, n8n, trading) with GCS backup |
| Strimzi | Kafka for the trading data path; Flux dependsOn so apps wait for the operator |
| Redis | In-cluster cache next to the trading services |
| Trading platform | data / execute / news / dashboard Deployments, HTTPRoutes, NetworkPolicies |
| Hermes agent | In-cluster agent + CNPG, talking to the vLLM services |
| News bot | CronJobs: scrape → analyst (vLLM) → Telegram |
| Waitlist API | FastAPI + CNPG, Gateway HTTPRoute |
| n8n | Workflow engine on its own Postgres |
| Gateway API | Public HTTPRoutes; Grafana/Prometheus stayed off the public internet (Tailscale / port-forward) |
App images were built in a separate code repo and pulled from GHCR. This repo is manifests only (after cleanup: no Dockerfiles, no Helm-of-the-app, no planning markdown).
Networking and security
- Cilium as GKE datapath, clusterwide NetworkPolicy enabled from Terraform.
- Gateway API
HTTPRoutefor public paths; internals not published. - Tailscale operator for operator access to Grafana and cluster services without a public LB.
- NetworkPolicies on the trading namespace: DNS, Postgres, Redis, Kafka, in-namespace HTTP, egress HTTPS to market APIs. Everything else denied.
- SOPS + age for Secrets in Git. Flux
sops-ageSecret influx-systemheld the private key. - GPU nodes tainted so a random Deployment cannot schedule onto a $2+/hr card.
Cost model (why it was torn down)
This was the whole point of the GPU design:
- L4 SPOT stayed at 1 node — cheap enough to keep a dispatcher warm.
- RTX 6000 and A100 pools autoscaled 0–1. KEDA HTTPScaledObject set the Deployment to 0 when there was no traffic; the node pool followed.
- Weights on PVC, long startup probes — first request after idle paid a cold-start, not a 40 GB pull.
- CronJobs for batch work instead of idle inference pods.
- When even the L4 + control-plane bill stopped making sense, the cluster was destroyed. Scale-to-zero was the rehearsal for scale-to-nothing.
Repository layout
gcloud-lab/
├── modules/ Terraform: VPC, GKE, node pools, GCS
├── clusters/devops-lab/ Flux entry (GitRepository + Kustomizations)
├── infrastructure/
│ ├── controllers/ CNPG, KEDA, cert-manager, Tailscale, Prometheus
│ ├── gpus/ vLLM Deployments + KEDA HTTPScaledObjects
│ ├── gatewayapi/ Gateway + HTTPRoutes
│ └── tailnet/ Tailscale ProxyGroup
├── apps/
│ ├── base/customer1/ Namespaced workloads (kustomize)
│ ├── base/osint-dashboard/ Helm chart used on this cluster
│ └── staging/ Overlays Flux actually syncs
├── monitoring/ Extra Grafana/Prometheus config
├── .sops.yaml Age recipient for secret encryption
├── .github/workflows/ Historical GH Actions (pgvector image, etc.)
└── mise.toml Local CLI pin (gcloud, kubectl, helm, sops, terraform, k9s)
Reading the code (suggested order)
modules/gke.tf+modules/vpc.tf— cluster shape.- GPU pools, then the matching vLLM YAML — taint keys must match or the pod never schedules.
clusters/devops-lab/customer1.yaml— FluxdependsOn, SOPS, prune/force.apps/staging/customer1/kustomization.yaml— what actually shipped incustomer1.- One NetworkPolicy under
apps/base/customer1/trading-platform/network-policies/— default-deny thinking.
Local tooling
mise trust && mise install # kubectl, helm, sops, terraform, k9s, gcloud
Decrypt a secret locally (needs the age key, which is not in Git):
sops -d apps/base/customer1/waitlist-api/waitlist-telegram-secret.yaml
Status
| Cluster | Destroyed (GCP project devops-lab-cluster, GKE devops-lab-cluster, us-central1-a) |
| This repo | Archive of what ran |
| Live infra today | Self-hosted on a Pi — see siriusdevops.com/lab |
Lance Walters — siriusdevops.com