4f235b52f4
Drop duplicate app/Helm trees, planning notes, and the vwap-monitor source tree so this repo is GitOps + Terraform only. Document the GKE cluster as shut down, with file-level pointers for vLLM, KEDA scale-to-zero, Flux, and SOPS.
246 lines
12 KiB
Markdown
246 lines
12 KiB
Markdown
# gcloud-lab
|
||
|
||
GitOps + Terraform source of truth for a **GKE lab** I designed, ran, and then **shut down** once the GPU bill stopped being worth it.
|
||
|
||
This is not a live cluster. It is the manifests, node-pool definitions, and GitOps wiring from a real environment that served vLLM inference, CNPG databases, and a handful of in-cluster apps. The same cost model that made the GPU pools scale to zero is why the whole footprint went to zero.
|
||
|
||
**Canonical copy:** [forgejo.siriusdevops.com/sirius/gcloud-lab](https://forgejo.siriusdevops.com/sirius/gcloud-lab)
|
||
|
||
---
|
||
|
||
## What this is evidence of
|
||
|
||
If you are reading this as a hiring screen, start here. Every claim below maps to a file in this repo.
|
||
|
||
| Claim | Where to look |
|
||
| --- | --- |
|
||
| GKE cluster, custom VPC, dual-stack, Cilium datapath | [`modules/gke.tf`](modules/gke.tf), [`modules/vpc.tf`](modules/vpc.tf) |
|
||
| CPU + three GPU node pools (L4, RTX PRO 6000, A100 80GB), all SPOT except CPU | [`modules/nodepool.tf`](modules/nodepool.tf), [`modules/nodepool-gpu.tf`](modules/nodepool-gpu.tf), [`modules/pro6000-nodepool.tf`](modules/pro6000-nodepool.tf), [`modules/a100-nodepool.tf`](modules/a100-nodepool.tf) |
|
||
| Flux CD applies the tree; SOPS decrypts secrets in-cluster | [`clusters/devops-lab/`](clusters/devops-lab/), [`.sops.yaml`](.sops.yaml) |
|
||
| Production-style **vLLM** OpenAI-compatible servers (not Ollama) | [`infrastructure/gpus/base/vllm-servers/`](infrastructure/gpus/base/vllm-servers/) |
|
||
| KEDA HTTP scale-to-zero on the expensive GPUs | [`infrastructure/gpus/base/keda-gpu-scaling/`](infrastructure/gpus/base/keda-gpu-scaling/) |
|
||
| CloudNative-PG operator + per-app Postgres | [`infrastructure/controllers/base/cnpg/`](infrastructure/controllers/base/cnpg/), `apps/base/customer1/*-db/` |
|
||
| Gateway API (not legacy Ingress) + Tailscale for internals | [`infrastructure/gatewayapi/`](infrastructure/gatewayapi/), [`infrastructure/tailnet/`](infrastructure/tailnet/) |
|
||
| Workload NetworkPolicies (trading stack) | [`apps/base/customer1/trading-platform/network-policies/`](apps/base/customer1/trading-platform/network-policies/) |
|
||
|
||
I run 24/7 infrastructure now on a Raspberry Pi (Forgejo, Cloudflare tunnel, containerized sites). This repo is the cloud chapter that came before that.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```
|
||
Git (this repo, branch master)
|
||
│
|
||
Flux source-controller (1m)
|
||
│
|
||
┌─────────────────┼──────────────────┐
|
||
▼ ▼ ▼
|
||
infra-controllers infra-gpus apps/staging
|
||
CNPG / KEDA vLLM + KEDA customer1 overlay
|
||
cert-manager HTTPScaledObject (kustomize)
|
||
Tailscale L4 / RTX6000 / A100
|
||
kube-prometheus
|
||
│ │ │
|
||
└──────────── GKE us-central1-a ─────┘
|
||
│
|
||
┌───────────────┬────────────┼────────────┬──────────────┐
|
||
▼ ▼ ▼ ▼ ▼
|
||
e2-standard-2 g2-standard-8 g4-standard-48 a2-ultragpu-1g
|
||
CPU pool L4 SPOT RTX PRO 6000 A100 80GB SPOT
|
||
1–5 nodes 1 node SPOT 0–1 SPOT 0–1
|
||
(always on) KEDA 0↔1 KEDA 0↔1
|
||
```
|
||
|
||
Datapath: GKE `ADVANCED_DATAPATH` + `enable_cilium_clusterwide_network_policy`. Dual-stack VPC (`10.0.0.0/16`, pods `192.168.32.0/20`, services `192.168.16.0/24`). GPU nodes are tainted (`nvidia.com/gpu…=present:NoSchedule`) so only inference pods land on them.
|
||
|
||
---
|
||
|
||
## GPU inference (vLLM)
|
||
|
||
Three independent OpenAI-compatible servers, each pinned to a pool via `nodeSelector` + matching taint/toleration. Model weights cached on PVC so a scale-up does not re-pull 20–40 GB from Hugging Face.
|
||
|
||
### L4 dispatcher — always on
|
||
|
||
[`infrastructure/gpus/base/vllm-servers/vllm-l4.yaml`](infrastructure/gpus/base/vllm-servers/vllm-l4.yaml)
|
||
|
||
- Machine: `g2-standard-8` + `nvidia-l4`, SPOT
|
||
- Image: `vllm/vllm-openai`
|
||
- Model: `p-e-w/Qwen3-8B-heretic`
|
||
- Flags that matter: `--kv-cache-dtype=fp8`, `--enable-chunked-prefill`, `--enable-prefix-caching`, `--enable-auto-tool-choice`, `--tool-call-parser=hermes`, `--max-model-len=32768`
|
||
|
||
Cheap, tool-capable, left running so agents had a low-latency brain even when the big cards were scaled out.
|
||
|
||
### RTX PRO 6000 — deep context, scale to zero
|
||
|
||
[`infrastructure/gpus/base/vllm-servers/rtx6000-vllm.yaml`](infrastructure/gpus/base/vllm-servers/rtx6000-vllm.yaml)
|
||
[`infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml`](infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml)
|
||
|
||
- Machine: `g4-standard-48` + `nvidia-rtx-pro-6000`, SPOT, **min 0 / max 1**
|
||
- Image: `vllm/vllm-openai:latest-cu129-ubuntu2404`
|
||
- Model: `edp1096/Huihui-Qwen3.6-27B-abliterated-FP8`
|
||
- `--max-model-len=262144`, `--enable-chunked-prefill`, `--tool-call-parser=qwen3_xml`, `--reasoning-parser=qwen3`, MTP speculative decoding (`num_speculative_tokens: 2`)
|
||
- Startup / liveness / readiness probes on `/health` and `/v1/models` (model load is slow; `failureThreshold: 60` on startup)
|
||
- KEDA `HTTPScaledObject`: `replicas.min: 0`, `max: 1`, host `rtx6000-brain-service.customer1.svc.cluster.local`
|
||
|
||
Idle deep-thinker capacity cost nothing. A request woke the Deployment; GKE then scaled the node pool off zero.
|
||
|
||
### A100 80GB — same pattern
|
||
|
||
[`infrastructure/gpus/base/vllm-servers/a100-vllm.yaml`](infrastructure/gpus/base/vllm-servers/a100-vllm.yaml)
|
||
[`modules/a100-nodepool.tf`](modules/a100-nodepool.tf)
|
||
|
||
- Machine: `a2-ultragpu-1g` + `nvidia-a100-80gb`, SPOT, min 0
|
||
- NVFP4 / ModelOpt quantized 27B, `--quantization=modelopt`, `--attention-backend=flash_attn`, `--kv-cache-dtype=fp8`
|
||
|
||
---
|
||
|
||
## GitOps
|
||
|
||
Flux watches `master` and applies layered Kustomizations:
|
||
|
||
```
|
||
clusters/devops-lab/
|
||
flux-system/ Flux controllers + GitRepository
|
||
infra-controllers.yaml → infrastructure/controllers/staging
|
||
CNPG 0.26.1, KEDA ≥2.14, cert-manager, Tailscale, kube-prometheus
|
||
infra-gpus.yaml → infrastructure/gpus/staging
|
||
infra-gatewayapi.yaml → Gateway API + HTTPRoutes
|
||
customer1-strimzi.yaml → Strimzi (Kafka) first
|
||
customer1.yaml → apps/staging/customer1 (dependsOn strimzi)
|
||
```
|
||
|
||
Every Flux Kustomization has `decryption.provider: sops` and `secretRef: sops-age`. Encrypted `data`/`stringData` in Git; Flux decrypts at apply time. Age recipient is in [`.sops.yaml`](.sops.yaml). The private key is **not** in this repo.
|
||
|
||
Bootstrap was originally:
|
||
|
||
```bash
|
||
flux bootstrap github \
|
||
--owner=sirius0xdev \
|
||
--repository=gcloud-lab \
|
||
--branch=master \
|
||
--path=clusters/devops-lab
|
||
```
|
||
|
||
`gotk-sync.yaml` still records that GitHub URL. This Forgejo copy is the archive; the cluster is gone, so the GitRepository object was never re-pointed.
|
||
|
||
---
|
||
|
||
## Terraform (cluster birth)
|
||
|
||
[`modules/`](modules/) is the IaC that created the cluster, not the day-to-day delivery path.
|
||
|
||
| File | Resource |
|
||
| --- | --- |
|
||
| `gke.tf` | `devops-lab-cluster` in `us-central1-a`, default pool removed, Cilium datapath, dual-stack IP policy |
|
||
| `vpc.tf` | `devops-lab-network` / `devops-lab-subnetwork`, ULA IPv6, secondary ranges |
|
||
| `nodepool.tf` | CPU `e2-standard-2`, 1–5, pd-standard 100 Gi |
|
||
| `nodepool-gpu.tf` | L4 SPOT `g2-standard-8`, autoscaling 1–1, GPU taint |
|
||
| `pro6000-nodepool.tf` | RTX PRO 6000 SPOT `g4-standard-48`, **0–1**, hyperdisk-balanced |
|
||
| `a100-nodepool.tf` | A100 80GB SPOT `a2-ultragpu-1g`, **0–1**, pd-ssd 200 Gi |
|
||
| `db-bucket.tf` | GCS bucket for CNPG backups |
|
||
| `providers.tf` | Google provider, Helm talking to the cluster via `gke-gcloud-auth-plugin` |
|
||
|
||
State files and `*.tfvars` are gitignored. Do not `terraform apply` this against a live billing account unless you intend to recreate it.
|
||
|
||
---
|
||
|
||
## Data plane and apps
|
||
|
||
Flux overlay: [`apps/staging/customer1/kustomization.yaml`](apps/staging/customer1/kustomization.yaml)
|
||
|
||
| Piece | Role |
|
||
| --- | --- |
|
||
| CloudNative-PG | Operator `0.26.1`; per-app clusters (Hermes memory, waitlist, n8n, trading) with GCS backup |
|
||
| Strimzi | Kafka for the trading data path; Flux `dependsOn` so apps wait for the operator |
|
||
| Redis | In-cluster cache next to the trading services |
|
||
| Trading platform | data / execute / news / dashboard Deployments, HTTPRoutes, NetworkPolicies |
|
||
| Hermes agent | In-cluster agent + CNPG, talking to the vLLM services |
|
||
| News bot | CronJobs: scrape → analyst (vLLM) → Telegram |
|
||
| Waitlist API | FastAPI + CNPG, Gateway HTTPRoute |
|
||
| n8n | Workflow engine on its own Postgres |
|
||
| Gateway API | Public HTTPRoutes; Grafana/Prometheus stayed off the public internet (Tailscale / port-forward) |
|
||
|
||
App **images** were built in a separate code repo and pulled from GHCR. This repo is manifests only (after cleanup: no Dockerfiles, no Helm-of-the-app, no planning markdown).
|
||
|
||
---
|
||
|
||
## Networking and security
|
||
|
||
- **Cilium** as GKE datapath, clusterwide NetworkPolicy enabled from Terraform.
|
||
- **Gateway API** `HTTPRoute` for public paths; internals not published.
|
||
- **Tailscale operator** for operator access to Grafana and cluster services without a public LB.
|
||
- **NetworkPolicies** on the trading namespace: DNS, Postgres, Redis, Kafka, in-namespace HTTP, egress HTTPS to market APIs. Everything else denied.
|
||
- **SOPS + age** for Secrets in Git. Flux `sops-age` Secret in `flux-system` held the private key.
|
||
- GPU nodes tainted so a random Deployment cannot schedule onto a $2+/hr card.
|
||
|
||
---
|
||
|
||
## Cost model (why it was torn down)
|
||
|
||
This was the whole point of the GPU design:
|
||
|
||
1. L4 SPOT stayed at 1 node — cheap enough to keep a dispatcher warm.
|
||
2. RTX 6000 and A100 pools **autoscaled 0–1**. KEDA HTTPScaledObject set the Deployment to 0 when there was no traffic; the node pool followed.
|
||
3. Weights on PVC, long startup probes — first request after idle paid a cold-start, not a 40 GB pull.
|
||
4. CronJobs for batch work instead of idle inference pods.
|
||
5. When even the L4 + control-plane bill stopped making sense, the cluster was destroyed. Scale-to-zero was the rehearsal for scale-to-nothing.
|
||
|
||
---
|
||
|
||
## Repository layout
|
||
|
||
```
|
||
gcloud-lab/
|
||
├── modules/ Terraform: VPC, GKE, node pools, GCS
|
||
├── clusters/devops-lab/ Flux entry (GitRepository + Kustomizations)
|
||
├── infrastructure/
|
||
│ ├── controllers/ CNPG, KEDA, cert-manager, Tailscale, Prometheus
|
||
│ ├── gpus/ vLLM Deployments + KEDA HTTPScaledObjects
|
||
│ ├── gatewayapi/ Gateway + HTTPRoutes
|
||
│ └── tailnet/ Tailscale ProxyGroup
|
||
├── apps/
|
||
│ ├── base/customer1/ Namespaced workloads (kustomize)
|
||
│ ├── base/osint-dashboard/ Helm chart used on this cluster
|
||
│ └── staging/ Overlays Flux actually syncs
|
||
├── monitoring/ Extra Grafana/Prometheus config
|
||
├── .sops.yaml Age recipient for secret encryption
|
||
├── .github/workflows/ Historical GH Actions (pgvector image, etc.)
|
||
└── mise.toml Local CLI pin (gcloud, kubectl, helm, sops, terraform, k9s)
|
||
```
|
||
|
||
---
|
||
|
||
## Reading the code (suggested order)
|
||
|
||
1. [`modules/gke.tf`](modules/gke.tf) + [`modules/vpc.tf`](modules/vpc.tf) — cluster shape.
|
||
2. GPU pools, then the matching vLLM YAML — taint keys must match or the pod never schedules.
|
||
3. [`clusters/devops-lab/customer1.yaml`](clusters/devops-lab/customer1.yaml) — Flux `dependsOn`, SOPS, prune/force.
|
||
4. [`apps/staging/customer1/kustomization.yaml`](apps/staging/customer1/kustomization.yaml) — what actually shipped in `customer1`.
|
||
5. One NetworkPolicy under `apps/base/customer1/trading-platform/network-policies/` — default-deny thinking.
|
||
|
||
---
|
||
|
||
## Local tooling
|
||
|
||
```bash
|
||
mise trust && mise install # kubectl, helm, sops, terraform, k9s, gcloud
|
||
```
|
||
|
||
Decrypt a secret locally (needs the age key, which is not in Git):
|
||
|
||
```bash
|
||
sops -d apps/base/customer1/waitlist-api/waitlist-telegram-secret.yaml
|
||
```
|
||
|
||
---
|
||
|
||
## Status
|
||
|
||
| | |
|
||
| --- | --- |
|
||
| Cluster | **Destroyed** (GCP project `devops-lab-cluster`, GKE `devops-lab-cluster`, `us-central1-a`) |
|
||
| This repo | Archive of what ran |
|
||
| Live infra today | Self-hosted on a Pi — see [siriusdevops.com/lab](https://siriusdevops.com/lab) |
|
||
|
||
Lance Walters — [siriusdevops.com](https://siriusdevops.com)
|