sirius/gcloud-lab
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: Add ROADMAP.md for API gateway and multi-tenant SaaS transition

Browse source
This commit is contained in:
Sirius Claw 2026-04-20 02:36:06 +00:00
parent 439ba8ab23
commit 7075b6d79d
1 changed files with 24 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

24
ROADMAP.md Normal file
View file

@ -0,0 +1,24 @@
# GKE AI Inference Platform - Project Roadmap
This roadmap tracks the tasks required to transition our dual-tier vLLM architecture (L4 Dispatcher + A100 Thinker) into a secure, multi-tenant SaaS offering for a limited group of premium users.
## Phase 1: Security & API Gateway
- [ ] **Choose API Gateway:** Select an ingress/gateway solution capable of API key auth and rate limiting (e.g., Kong, Traefik, or GCP API Gateway).
- [ ] **Implement API Key Auth:** Require a valid token/key to hit the vLLM endpoints.
- [ ] **Configure Rate Limiting:** Prevent a single user from spamming requests and hogging the L4 queue or unnecessarily waking the A100.
- [ ] **Network Isolation:** Ensure vLLM services are not publicly exposed directly; all traffic must flow through the gateway.
## Phase 2: User Access & Quotas
- [ ] **User Tiering:** Define what "access" means.
- *Example:* X number of L4 fast-tokens per month, Y number of A100 deep-thinking hours per month.
- [ ] **Usage Tracking:** Implement a lightweight logging/metrics system (Prometheus/Grafana or a custom DB) to track token usage per API key.
- [ ] **Onboarding Process:** Create a secure way to generate and distribute API keys to the limited user cohort.
## Phase 3: Infrastructure Tuning & Observability
- [ ] **A100 Sleep Tuning:** Monitor KEDA scale-down metrics. If users trigger the A100 too frequently, adjust the 15-minute timeout or implement a queuing system for heavy tasks.
- [ ] **Alerting:** Set up Slack/Telegram alerts for GPU OOM (Out of Memory) errors, KEDA scaling failures, and gateway 429 (Rate Limit Exceeded) spikes.
- [ ] **Cost Monitoring:** Set up strict GCP billing alerts to ensure the A100 node doesn't accidentally run 24/7 due to a stuck scale-to-zero metric.
## Phase 4: Billing (Optional)
- [ ] **Stripe Integration:** Hook API key generation to Stripe subscriptions.
- [ ] **Usage-based Billing:** Bill users automatically based on the tokens generated at the Gateway level.