sirius0xdev
53fde83544
fix(rtx6000-scaling): set minReplicas to 1 to prevent transient scale-up/down thrashing from periodic health checks
...
Prevents the watcher cron (every 5m) from causing unnecessary spot pod churn. Keeps 1 ready pod always (KEDA overrides deployment replicas).
2026-04-29 11:20:45 +00:00
sirius0xdev
dd34e21f76
Update rtx6000-vllm.yaml
2026-04-29 01:22:41 -04:00
sirius0xdev
bfb1b3d435
Update keda-vllm.yaml
2026-04-29 01:10:18 -04:00
sirius0xdev
b9e8448956
Merge pull request #67 from sirius0xdev/feat/optimize-rtx6000-vllm-keda-scaling
...
fix(keda): simplify HTTPScaledObject to valid schema fields only
2026-04-29 00:56:13 -04:00
sirius0xdev
5c61d600b0
fix(keda): simplify HTTPScaledObject to valid schema fields only
...
- Removed cooldownPeriod, scaledownPeriod, paths (all undeclared in http.keda.sh/v1alpha1 CRD)
- Kept pollingInterval:10 for responsiveness + min:0 replicas for true scale-to-zero
- Cleaned both rtx6000 and a100 configs for consistency
Dry-run should now succeed. Scale behavior preserved via polling + low targetPendingRequests (defaults work well for vLLM).
2026-04-29 04:54:49 +00:00
sirius0xdev
2de02d75c9
Merge pull request #66 from sirius0xdev/feat/optimize-rtx6000-vllm-keda-scaling
...
fix(keda): remove invalid spec.paths from HTTPScaledObject
2026-04-29 00:50:53 -04:00
sirius0xdev
a4b500a3ab
fix(keda): remove invalid spec.paths from HTTPScaledObject
...
HTTPScaledObject CRD (http.keda.sh/v1alpha1) does not support .spec.paths field.
Scaling now triggers on all HTTP requests to the host (health/liveness ignored via targetPendingRequests:1 + pollingInterval:10s).
Refs https://keda.sh/docs/scalers/http-addon/
2026-04-29 04:50:10 +00:00
sirius0xdev
f268a83663
Update kustomization.yaml
2026-04-29 00:35:46 -04:00
sirius0xdev
7e7b7d414d
fix: restrict KEDA scaling to /v1/(chat/)?completions paths only
...
- Switch to HTTPScaledObject (matches a100 pattern)
- Probes (/health, /v1/models) now ignored for scaling
- Keeps fast polling (10s), quick cooldown (30s), 15min idle scale-to-0
Health/readiness/startup probes hit pod IP directly (bypass service), so never triggered scaling anyway — but paths ensure only inference traffic scales.
2026-04-29 04:14:46 +00:00
sirius0xdev
d08bf5844a
feat: optimize RTX6000 vLLM KEDA scaling
...
- Rename deployment to rtx6000-brain-vllm to match scaler target ref
- Set deployment replicas to 0 (KEDA controlled)
- Add pollingInterval: 10s and cooldownPeriod: 30s for faster response
This enables scale-to-zero when idle and quick scale-up on first request.
See https://github.com/sirius0xdev/gcloud-lab/tree/master/infrastructure%2Fgpus%2Fbase
2026-04-29 04:12:36 +00:00
sirius0xdev
64eaeef931
Update keda-vllm.yaml
2026-04-29 00:05:09 -04:00
sirius0xdev
d3bea19f41
Update keda-vllm.yaml
2026-04-28 23:58:26 -04:00
sirius0xdev
5622e6c49c
Update rtx6000-vllm.yaml
2026-04-28 23:55:29 -04:00
sirius0xdev
12acba27d7
Update kustomization.yaml
2026-04-28 23:51:19 -04:00
sirius0xdev
7b33f57cab
add storage class
2026-04-29 02:07:21 +00:00
sirius0xdev
79942eeb5e
fix disk type
2026-04-29 00:04:47 +00:00
sirius0xdev
c2f42f067d
Merge branch 'master' of github.com:sirius0xdev/gcloud-lab
...
C
jk#
2026-04-28 23:17:01 +00:00
sirius0xdev
bb7dcd5651
fix node selector for 6000pro
2026-04-28 23:16:51 +00:00
sirius0xdev
94811c019d
Update route.yaml
2026-04-28 13:18:34 -04:00
sirius0xdev
d9048a7014
remove hostname from gateway
2026-04-28 05:00:57 +00:00
sirius0xdev
aca1298f64
fix route
2026-04-28 04:30:14 +00:00
sirius0xdev
3c233b0916
fix hostname
2026-04-28 04:18:33 +00:00
sirius0xdev
e9e2e02a34
fix certs apigateway
2026-04-28 04:16:38 +00:00
sirius0xdev
f2d1badae4
fix typo
2026-04-28 04:07:36 +00:00
sirius0xdev
fc44f9128c
fix gateway
2026-04-28 04:00:33 +00:00
sirius0xdev
dc285d26a8
fix kustomization
2026-04-28 03:38:08 +00:00
sirius0xdev
b680a95d1f
fix typos
2026-04-28 03:33:52 +00:00
sirius0xdev
9f5d1fb723
clean up and add routes to paas site
2026-04-28 03:31:52 +00:00
Hermes Agent
6183159974
feat: add Prometheus, Grafana, and Tailscale monitoring stack
...
- Install Prometheus + Grafana via kube-prometheus-stack (ClusterIP only, no public ingress)
- Deploy Tailscale Operator for secure VPN access to internal services
- Add CNPG/PostgreSQL monitoring dashboards
- Add vLLM inference monitoring dashboards (tokens, latency, GPU)
- Add Cilium networking dashboards (policy, traffic, drops)
- Update infra-controllers staging kustomization to include all controllers
- Add monitoring-configs Flux sync for dashboard deployment
- Update README with monitoring architecture and access instructions
- Remove broken stale monitoring files (Azure Key Vault refs, wrong domains)
Access: kubectl port-forward or Tailscale VPN (replace auth key before deploy)
2026-04-26 02:31:53 +00:00
sirius0xdev
7a3126caff
working on certmanager and promethus stack
2026-04-23 01:19:56 +00:00
Sirius Claw
4fb80d3157
fix: update KEDA Helm controllers to Flux v2 APIs
2026-04-20 04:00:10 +00:00
Sirius Claw
e736d5f874
feat: Add KEDA and KEDA HTTP Add-on controllers
2026-04-20 03:36:34 +00:00
sirius0xdev
b6d93886bf
fix infra kustomization.yaml
2026-02-02 23:28:51 +00:00
sirius0xdev
995baa64cb
added service account for db backups and changed terraform to include bucket and service accounts
2026-02-02 23:20:46 +00:00
sirius0xdev
b0eb3c2e8c
fix release.yaml
2026-01-06 01:54:18 +00:00
sirius0xdev
87b2dae39f
fix cnpg kustomization
2026-01-06 01:46:55 +00:00
sirius0xdev
521ea092af
fix dir structure
2026-01-06 01:30:10 +00:00
sirius0xdev
d458420cbf
added cnpg controller
2026-01-05 23:03:09 +00:00
