gcloud-lab/README.md

# gcloud-lab

GitOps + Terraform source of truth for a **GKE lab** I designed, ran, and then **shut down** once the GPU bill stopped being worth it.

This is not a live cluster. It is the manifests, node-pool definitions, and GitOps wiring from a real environment that served vLLM inference, CNPG databases, and a handful of in-cluster apps. The same cost model that made the GPU pools scale to zero is why the whole footprint went to zero.

**Canonical copy:** [forgejo.siriusdevops.com/sirius/gcloud-lab](https://forgejo.siriusdevops.com/sirius/gcloud-lab)

---

## What this is evidence of

If you are reading this as a hiring screen, start here. Every claim below maps to a file in this repo.

| Claim | Where to look |
| --- | --- |
| GKE cluster, custom VPC, dual-stack, Cilium datapath | [`modules/gke.tf`](modules/gke.tf), [`modules/vpc.tf`](modules/vpc.tf) |
| CPU + three GPU node pools (L4, RTX PRO 6000, A100 80GB), all SPOT except CPU | [`modules/nodepool.tf`](modules/nodepool.tf), [`modules/nodepool-gpu.tf`](modules/nodepool-gpu.tf), [`modules/pro6000-nodepool.tf`](modules/pro6000-nodepool.tf), [`modules/a100-nodepool.tf`](modules/a100-nodepool.tf) |
| Flux CD applies the tree; SOPS decrypts secrets in-cluster | [`clusters/devops-lab/`](clusters/devops-lab/), [`.sops.yaml`](.sops.yaml) |
| Production-style **vLLM** OpenAI-compatible servers (not Ollama) | [`infrastructure/gpus/base/vllm-servers/`](infrastructure/gpus/base/vllm-servers/) |
| KEDA HTTP scale-to-zero on the expensive GPUs | [`infrastructure/gpus/base/keda-gpu-scaling/`](infrastructure/gpus/base/keda-gpu-scaling/) |
| CloudNative-PG operator + per-app Postgres | [`infrastructure/controllers/base/cnpg/`](infrastructure/controllers/base/cnpg/), `apps/base/customer1/*-db/` |
| Gateway API (not legacy Ingress) + Tailscale for internals | [`infrastructure/gatewayapi/`](infrastructure/gatewayapi/), [`infrastructure/tailnet/`](infrastructure/tailnet/) |
| Workload NetworkPolicies (trading stack) | [`apps/base/customer1/trading-platform/network-policies/`](apps/base/customer1/trading-platform/network-policies/) |

I run 24/7 infrastructure now on a Raspberry Pi (Forgejo, Cloudflare tunnel, containerized sites). This repo is the cloud chapter that came before that.

---

## Architecture

```
                         Git (this repo, branch master)
                                      │
                         Flux source-controller (1m)
                                      │
                    ┌─────────────────┼──────────────────┐
                    ▼                 ▼                  ▼
           infra-controllers    infra-gpus         apps/staging
           CNPG / KEDA          vLLM + KEDA        customer1 overlay
           cert-manager         HTTPScaledObject   (kustomize)
           Tailscale            L4 / RTX6000 / A100
           kube-prometheus
                    │                 │                  │
                    └──────────── GKE us-central1-a ─────┘
                                      │
         ┌───────────────┬────────────┼────────────┬──────────────┐
         ▼               ▼            ▼            ▼              ▼
   e2-standard-2    g2-standard-8  g4-standard-48  a2-ultragpu-1g
   CPU pool         L4 SPOT        RTX PRO 6000    A100 80GB SPOT
   1–5 nodes        1 node         SPOT 0–1        SPOT 0–1
                   (always on)     KEDA 0↔1        KEDA 0↔1
```

Datapath: GKE `ADVANCED_DATAPATH` + `enable_cilium_clusterwide_network_policy`. Dual-stack VPC (`10.0.0.0/16`, pods `192.168.32.0/20`, services `192.168.16.0/24`). GPU nodes are tainted (`nvidia.com/gpu…=present:NoSchedule`) so only inference pods land on them.

---

## GPU inference (vLLM)

Three independent OpenAI-compatible servers, each pinned to a pool via `nodeSelector` + matching taint/toleration. Model weights cached on PVC so a scale-up does not re-pull 20–40 GB from Hugging Face.

### L4 dispatcher — always on

[`infrastructure/gpus/base/vllm-servers/vllm-l4.yaml`](infrastructure/gpus/base/vllm-servers/vllm-l4.yaml)

- Machine: `g2-standard-8` + `nvidia-l4`, SPOT
- Image: `vllm/vllm-openai`
- Model: `p-e-w/Qwen3-8B-heretic`
- Flags that matter: `--kv-cache-dtype=fp8`, `--enable-chunked-prefill`, `--enable-prefix-caching`, `--enable-auto-tool-choice`, `--tool-call-parser=hermes`, `--max-model-len=32768`

Cheap, tool-capable, left running so agents had a low-latency brain even when the big cards were scaled out.

### RTX PRO 6000 — deep context, scale to zero

[`infrastructure/gpus/base/vllm-servers/rtx6000-vllm.yaml`](infrastructure/gpus/base/vllm-servers/rtx6000-vllm.yaml)  
[`infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml`](infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml)

- Machine: `g4-standard-48` + `nvidia-rtx-pro-6000`, SPOT, **min 0 / max 1**
- Image: `vllm/vllm-openai:latest-cu129-ubuntu2404`
- Model: `edp1096/Huihui-Qwen3.6-27B-abliterated-FP8`
- `--max-model-len=262144`, `--enable-chunked-prefill`, `--tool-call-parser=qwen3_xml`, `--reasoning-parser=qwen3`, MTP speculative decoding (`num_speculative_tokens: 2`)
- Startup / liveness / readiness probes on `/health` and `/v1/models` (model load is slow; `failureThreshold: 60` on startup)
- KEDA `HTTPScaledObject`: `replicas.min: 0`, `max: 1`, host `rtx6000-brain-service.customer1.svc.cluster.local`

Idle deep-thinker capacity cost nothing. A request woke the Deployment; GKE then scaled the node pool off zero.

### A100 80GB — same pattern

[`infrastructure/gpus/base/vllm-servers/a100-vllm.yaml`](infrastructure/gpus/base/vllm-servers/a100-vllm.yaml)  
[`modules/a100-nodepool.tf`](modules/a100-nodepool.tf)

- Machine: `a2-ultragpu-1g` + `nvidia-a100-80gb`, SPOT, min 0
- NVFP4 / ModelOpt quantized 27B, `--quantization=modelopt`, `--attention-backend=flash_attn`, `--kv-cache-dtype=fp8`

---

## GitOps

Flux watches `master` and applies layered Kustomizations:

```
clusters/devops-lab/
  flux-system/          Flux controllers + GitRepository
  infra-controllers.yaml  → infrastructure/controllers/staging
                            CNPG 0.26.1, KEDA ≥2.14, cert-manager, Tailscale, kube-prometheus
  infra-gpus.yaml         → infrastructure/gpus/staging
  infra-gatewayapi.yaml   → Gateway API + HTTPRoutes
  customer1-strimzi.yaml  → Strimzi (Kafka) first
  customer1.yaml          → apps/staging/customer1  (dependsOn strimzi)
```

Every Flux Kustomization has `decryption.provider: sops` and `secretRef: sops-age`. Encrypted `data`/`stringData` in Git; Flux decrypts at apply time. Age recipient is in [`.sops.yaml`](.sops.yaml). The private key is **not** in this repo.

Bootstrap was originally:

```bash
flux bootstrap github \
  --owner=sirius0xdev \
  --repository=gcloud-lab \
  --branch=master \
  --path=clusters/devops-lab
```

`gotk-sync.yaml` still records that GitHub URL. This Forgejo copy is the archive; the cluster is gone, so the GitRepository object was never re-pointed.

---

## Terraform (cluster birth)

[`modules/`](modules/) is the IaC that created the cluster, not the day-to-day delivery path.

| File | Resource |
| --- | --- |
| `gke.tf` | `devops-lab-cluster` in `us-central1-a`, default pool removed, Cilium datapath, dual-stack IP policy |
| `vpc.tf` | `devops-lab-network` / `devops-lab-subnetwork`, ULA IPv6, secondary ranges |
| `nodepool.tf` | CPU `e2-standard-2`, 1–5, pd-standard 100 Gi |
| `nodepool-gpu.tf` | L4 SPOT `g2-standard-8`, autoscaling 1–1, GPU taint |
| `pro6000-nodepool.tf` | RTX PRO 6000 SPOT `g4-standard-48`, **0–1**, hyperdisk-balanced |
| `a100-nodepool.tf` | A100 80GB SPOT `a2-ultragpu-1g`, **0–1**, pd-ssd 200 Gi |
| `db-bucket.tf` | GCS bucket for CNPG backups |
| `providers.tf` | Google provider, Helm talking to the cluster via `gke-gcloud-auth-plugin` |

State files and `*.tfvars` are gitignored. Do not `terraform apply` this against a live billing account unless you intend to recreate it.

---

## Data plane and apps

Flux overlay: [`apps/staging/customer1/kustomization.yaml`](apps/staging/customer1/kustomization.yaml)

| Piece | Role |
| --- | --- |
| CloudNative-PG | Operator `0.26.1`; per-app clusters (Hermes memory, waitlist, n8n, trading) with GCS backup |
| Strimzi | Kafka for the trading data path; Flux `dependsOn` so apps wait for the operator |
| Redis | In-cluster cache next to the trading services |
| Trading platform | data / execute / news / dashboard Deployments, HTTPRoutes, NetworkPolicies |
| Hermes agent | In-cluster agent + CNPG, talking to the vLLM services |
| News bot | CronJobs: scrape → analyst (vLLM) → Telegram |
| Waitlist API | FastAPI + CNPG, Gateway HTTPRoute |
| n8n | Workflow engine on its own Postgres |
| Gateway API | Public HTTPRoutes; Grafana/Prometheus stayed off the public internet (Tailscale / port-forward) |

App **images** were built in a separate code repo and pulled from GHCR. This repo is manifests only (after cleanup: no Dockerfiles, no Helm-of-the-app, no planning markdown).

---

## Networking and security

- **Cilium** as GKE datapath, clusterwide NetworkPolicy enabled from Terraform.
- **Gateway API** `HTTPRoute` for public paths; internals not published.
- **Tailscale operator** for operator access to Grafana and cluster services without a public LB.
- **NetworkPolicies** on the trading namespace: DNS, Postgres, Redis, Kafka, in-namespace HTTP, egress HTTPS to market APIs. Everything else denied.
- **SOPS + age** for Secrets in Git. Flux `sops-age` Secret in `flux-system` held the private key.
- GPU nodes tainted so a random Deployment cannot schedule onto a $2+/hr card.

---

## Cost model (why it was torn down)

This was the whole point of the GPU design:

1. L4 SPOT stayed at 1 node — cheap enough to keep a dispatcher warm.
2. RTX 6000 and A100 pools **autoscaled 0–1**. KEDA HTTPScaledObject set the Deployment to 0 when there was no traffic; the node pool followed.
3. Weights on PVC, long startup probes — first request after idle paid a cold-start, not a 40 GB pull.
4. CronJobs for batch work instead of idle inference pods.
5. When even the L4 + control-plane bill stopped making sense, the cluster was destroyed. Scale-to-zero was the rehearsal for scale-to-nothing.

---

## Repository layout

```
gcloud-lab/
├── modules/                         Terraform: VPC, GKE, node pools, GCS
├── clusters/devops-lab/             Flux entry (GitRepository + Kustomizations)
├── infrastructure/
│   ├── controllers/                 CNPG, KEDA, cert-manager, Tailscale, Prometheus
│   ├── gpus/                        vLLM Deployments + KEDA HTTPScaledObjects
│   ├── gatewayapi/                  Gateway + HTTPRoutes
│   └── tailnet/                     Tailscale ProxyGroup
├── apps/
│   ├── base/customer1/              Namespaced workloads (kustomize)
│   ├── base/osint-dashboard/        Helm chart used on this cluster
│   └── staging/                     Overlays Flux actually syncs
├── monitoring/                      Extra Grafana/Prometheus config
├── .sops.yaml                       Age recipient for secret encryption
├── .github/workflows/               Historical GH Actions (pgvector image, etc.)
└── mise.toml                        Local CLI pin (gcloud, kubectl, helm, sops, terraform, k9s)
```

---

## Reading the code (suggested order)

1. [`modules/gke.tf`](modules/gke.tf) + [`modules/vpc.tf`](modules/vpc.tf) — cluster shape.
2. GPU pools, then the matching vLLM YAML — taint keys must match or the pod never schedules.
3. [`clusters/devops-lab/customer1.yaml`](clusters/devops-lab/customer1.yaml) — Flux `dependsOn`, SOPS, prune/force.
4. [`apps/staging/customer1/kustomization.yaml`](apps/staging/customer1/kustomization.yaml) — what actually shipped in `customer1`.
5. One NetworkPolicy under `apps/base/customer1/trading-platform/network-policies/` — default-deny thinking.

---

## Local tooling

```bash
mise trust && mise install   # kubectl, helm, sops, terraform, k9s, gcloud
```

Decrypt a secret locally (needs the age key, which is not in Git):

```bash
sops -d apps/base/customer1/waitlist-api/waitlist-telegram-secret.yaml
```

---

## Status

| | |
| --- | --- |
| Cluster | **Destroyed** (GCP project `devops-lab-cluster`, GKE `devops-lab-cluster`, `us-central1-a`) |
| This repo | Archive of what ran |
| Live infra today | Self-hosted on a Pi — see [siriusdevops.com/lab](https://siriusdevops.com/lab) |

Lance Walters — [siriusdevops.com](https://siriusdevops.com)