sirius0xdev
61165c2b00
fix: separate monitoring from controllers so broken prometheus-stack doesn't block tailnet chain
2026-05-04 00:30:02 +00:00
sirius0xdev
5e3b34646c
fix(tailnet): move tsproxy-rtx6000.yaml into tailnet dir for Flux build
2026-05-04 00:17:17 +00:00
sirius0xdev
83b70951c6
fix(tailnet): move TsProxy to separate Kustomization that depends on operator
...
- Remove TsProxy from infrastructure-controllers to avoid CRD timing issue
- Create infrastructure/tailnet/ Kustomization for TsProxy resources
- Add infrastructure-tailnet Flux Kustomization with dependsOn
- Add dependsOn to customer1 Kustomization for trade-dashboard TsProxy
2026-05-04 00:06:51 +00:00
sirius0xdev
5f9b55af55
feat(tailscale): add operator authkey secret and rtx6000-brain TsProxy
...
- Add SOPS-encrypted tailscale-operator-authkey secret for operator auth
- Add TsProxy to expose rtx6000-brain-service on tailnet (port 8000)
- Enable trade-dashboard TsProxy (was waiting for operator install)
2026-05-03 23:30:28 +00:00
sirius0xdev
7e7794d3f5
start vllm server
2026-05-03 22:29:58 +00:00
sirius0xdev
698d4aec3d
Update kustomization.yaml
2026-05-02 19:44:14 -04:00
sirius0xdev
5bc7f60f35
Update kustomization.yaml
2026-05-02 19:43:59 -04:00
sirius0xdev
31e5af495a
Update rtx6000-vllm.yaml
2026-05-02 11:06:22 -04:00
sirius0xdev
324c69faf9
Update rtx6000-vllm.yaml
2026-05-02 09:11:46 -04:00
sirius0xdev
9a7904c219
Update rtx6000-vllm.yaml
2026-05-02 08:36:04 -04:00
sirius0xdev
1aceefbedd
Update kustomization.yaml
2026-05-02 08:26:36 -04:00
sirius0xdev
864cfd2cc9
Update rtx6000-vllm.yaml
2026-05-02 08:26:09 -04:00
sirius0xdev
08e9998019
Update rtx6000-vllm.yaml
2026-05-02 08:24:21 -04:00
sirius0xdev
96d7af2886
Update kustomization.yaml
2026-04-30 01:21:28 -04:00
sirius0xdev
d9fef8cb5d
Update kustomization.yaml
2026-04-29 21:43:02 -04:00
sirius0xdev
0b5867a501
Update kustomization.yaml
2026-04-29 21:41:52 -04:00
sirius0xdev
1366590721
Update kustomization.yaml
2026-04-29 13:49:49 -04:00
sirius0xdev
ae8e51cc79
Update kustomization.yaml
2026-04-29 12:58:32 -04:00
sirius0xdev
de39eb1053
Update kustomization.yaml
2026-04-29 12:58:05 -04:00
sirius0xdev
bd760287a8
fix(rtx6000): enable scale-to-zero with 20min cooldown + wake-up route
...
- Set replicas.min: 0 for scale-to-zero when idle
- Add config.cooldownPeriod: 1200 to KEDA HTTP add-on HelmRelease
(20 min buffer before scaling down from 1 replica)
- Add external HTTPRoute at brain.siriusdevops.com for manual wake-up
via curl from Hermes/Telegram
2026-04-29 16:01:03 +00:00
sirius0xdev
17f46261e0
Update rtx6000-vllm.yaml
2026-04-29 11:21:02 -04:00
sirius0xdev
24fee8f4a2
Update rtx6000-vllm.yaml
2026-04-29 09:41:48 -04:00
sirius0xdev
53fde83544
fix(rtx6000-scaling): set minReplicas to 1 to prevent transient scale-up/down thrashing from periodic health checks
...
Prevents the watcher cron (every 5m) from causing unnecessary spot pod churn. Keeps 1 ready pod always (KEDA overrides deployment replicas).
2026-04-29 11:20:45 +00:00
sirius0xdev
dd34e21f76
Update rtx6000-vllm.yaml
2026-04-29 01:22:41 -04:00
sirius0xdev
bfb1b3d435
Update keda-vllm.yaml
2026-04-29 01:10:18 -04:00
sirius0xdev
b9e8448956
Merge pull request #67 from sirius0xdev/feat/optimize-rtx6000-vllm-keda-scaling
...
fix(keda): simplify HTTPScaledObject to valid schema fields only
2026-04-29 00:56:13 -04:00
sirius0xdev
5c61d600b0
fix(keda): simplify HTTPScaledObject to valid schema fields only
...
- Removed cooldownPeriod, scaledownPeriod, paths (all undeclared in http.keda.sh/v1alpha1 CRD)
- Kept pollingInterval:10 for responsiveness + min:0 replicas for true scale-to-zero
- Cleaned both rtx6000 and a100 configs for consistency
Dry-run should now succeed. Scale behavior preserved via polling + low targetPendingRequests (defaults work well for vLLM).
2026-04-29 04:54:49 +00:00
sirius0xdev
2de02d75c9
Merge pull request #66 from sirius0xdev/feat/optimize-rtx6000-vllm-keda-scaling
...
fix(keda): remove invalid spec.paths from HTTPScaledObject
2026-04-29 00:50:53 -04:00
sirius0xdev
a4b500a3ab
fix(keda): remove invalid spec.paths from HTTPScaledObject
...
HTTPScaledObject CRD (http.keda.sh/v1alpha1) does not support .spec.paths field.
Scaling now triggers on all HTTP requests to the host (health/liveness ignored via targetPendingRequests:1 + pollingInterval:10s).
Refs https://keda.sh/docs/scalers/http-addon/
2026-04-29 04:50:10 +00:00
sirius0xdev
f268a83663
Update kustomization.yaml
2026-04-29 00:35:46 -04:00
sirius0xdev
7e7b7d414d
fix: restrict KEDA scaling to /v1/(chat/)?completions paths only
...
- Switch to HTTPScaledObject (matches a100 pattern)
- Probes (/health, /v1/models) now ignored for scaling
- Keeps fast polling (10s), quick cooldown (30s), 15min idle scale-to-0
Health/readiness/startup probes hit pod IP directly (bypass service), so never triggered scaling anyway — but paths ensure only inference traffic scales.
2026-04-29 04:14:46 +00:00
sirius0xdev
d08bf5844a
feat: optimize RTX6000 vLLM KEDA scaling
...
- Rename deployment to rtx6000-brain-vllm to match scaler target ref
- Set deployment replicas to 0 (KEDA controlled)
- Add pollingInterval: 10s and cooldownPeriod: 30s for faster response
This enables scale-to-zero when idle and quick scale-up on first request.
See https://github.com/sirius0xdev/gcloud-lab/tree/master/infrastructure%2Fgpus%2Fbase
2026-04-29 04:12:36 +00:00
sirius0xdev
64eaeef931
Update keda-vllm.yaml
2026-04-29 00:05:09 -04:00
sirius0xdev
d3bea19f41
Update keda-vllm.yaml
2026-04-28 23:58:26 -04:00
sirius0xdev
5622e6c49c
Update rtx6000-vllm.yaml
2026-04-28 23:55:29 -04:00
sirius0xdev
12acba27d7
Update kustomization.yaml
2026-04-28 23:51:19 -04:00
sirius0xdev
7b33f57cab
add storage class
2026-04-29 02:07:21 +00:00
sirius0xdev
79942eeb5e
fix disk type
2026-04-29 00:04:47 +00:00
sirius0xdev
c2f42f067d
Merge branch 'master' of github.com:sirius0xdev/gcloud-lab
...
C
jk#
2026-04-28 23:17:01 +00:00
sirius0xdev
bb7dcd5651
fix node selector for 6000pro
2026-04-28 23:16:51 +00:00
sirius0xdev
94811c019d
Update route.yaml
2026-04-28 13:18:34 -04:00
sirius0xdev
d9048a7014
remove hostname from gateway
2026-04-28 05:00:57 +00:00
sirius0xdev
aca1298f64
fix route
2026-04-28 04:30:14 +00:00
sirius0xdev
3c233b0916
fix hostname
2026-04-28 04:18:33 +00:00
sirius0xdev
e9e2e02a34
fix certs apigateway
2026-04-28 04:16:38 +00:00
sirius0xdev
f2d1badae4
fix typo
2026-04-28 04:07:36 +00:00
sirius0xdev
fc44f9128c
fix gateway
2026-04-28 04:00:33 +00:00
sirius0xdev
dc285d26a8
fix kustomization
2026-04-28 03:38:08 +00:00
sirius0xdev
b680a95d1f
fix typos
2026-04-28 03:33:52 +00:00
sirius0xdev
9f5d1fb723
clean up and add routes to paas site
2026-04-28 03:31:52 +00:00
