diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..e9c9d54 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,24 @@ +# GKE AI Inference Platform - Project Roadmap + +This roadmap tracks the tasks required to transition our dual-tier vLLM architecture (L4 Dispatcher + A100 Thinker) into a secure, multi-tenant SaaS offering for a limited group of premium users. + +## Phase 1: Security & API Gateway +- [ ] **Choose API Gateway:** Select an ingress/gateway solution capable of API key auth and rate limiting (e.g., Kong, Traefik, or GCP API Gateway). +- [ ] **Implement API Key Auth:** Require a valid token/key to hit the vLLM endpoints. +- [ ] **Configure Rate Limiting:** Prevent a single user from spamming requests and hogging the L4 queue or unnecessarily waking the A100. +- [ ] **Network Isolation:** Ensure vLLM services are not publicly exposed directly; all traffic must flow through the gateway. + +## Phase 2: User Access & Quotas +- [ ] **User Tiering:** Define what "access" means. + - *Example:* X number of L4 fast-tokens per month, Y number of A100 deep-thinking hours per month. +- [ ] **Usage Tracking:** Implement a lightweight logging/metrics system (Prometheus/Grafana or a custom DB) to track token usage per API key. +- [ ] **Onboarding Process:** Create a secure way to generate and distribute API keys to the limited user cohort. + +## Phase 3: Infrastructure Tuning & Observability +- [ ] **A100 Sleep Tuning:** Monitor KEDA scale-down metrics. If users trigger the A100 too frequently, adjust the 15-minute timeout or implement a queuing system for heavy tasks. +- [ ] **Alerting:** Set up Slack/Telegram alerts for GPU OOM (Out of Memory) errors, KEDA scaling failures, and gateway 429 (Rate Limit Exceeded) spikes. +- [ ] **Cost Monitoring:** Set up strict GCP billing alerts to ensure the A100 node doesn't accidentally run 24/7 due to a stuck scale-to-zero metric. + +## Phase 4: Billing (Optional) +- [ ] **Stripe Integration:** Hook API key generation to Stripe subscriptions. +- [ ] **Usage-based Billing:** Bill users automatically based on the tokens generated at the Gateway level.