Compare commits

..

14 commits

Author SHA1 Message Date
Sirius DevOps
4f235b52f4 Archive cleanup and rewrite README as a lab write-up.
Drop duplicate app/Helm trees, planning notes, and the vwap-monitor
source tree so this repo is GitOps + Terraform only. Document the
GKE cluster as shut down, with file-level pointers for vLLM, KEDA
scale-to-zero, Flux, and SOPS.
2026-09-04 22:38:00 -04:00
Sirius DevOps
fe4fa1f34d
clean up 2026-08-28 20:02:01 -04:00
sirius0xdev
eb27a51523
clean up 2026-06-04 20:18:23 -04:00
sirius0xdev
88db49b8b5
Delete dummy0auth.yaml 2026-05-31 21:48:58 -04:00
sirius0xdev
87632b2247 clean up unused files 2026-05-29 22:55:22 +00:00
sirius0xdev
d7beb9f6ab Merge branch 'master' of github.com:sirius0xdev/gcloud-lab 2026-05-29 22:30:11 +00:00
sirius0xdev
2b54a18724
Update kustomization.yaml 2026-05-29 11:41:16 -04:00
Hermes DevOps
12766f293c fix(dashboard): add :8000 port to service URLs in ConfigMap
Next.js rewrites now use env vars (DATA_SERVICE_URL etc.) which point
to in-cluster services. Missing port caused rewrites to hit port 80
instead of 8000, breaking /api/data/* proxy (candles, orderbook).
2026-05-29 13:39:05 +00:00
Hermes DevOps
18764cf6b5 fix: allow port 80 in inter-service egress for ClusterIP traffic 2026-05-29 13:16:35 +00:00
sirius0xdev
79e0d98fbe
Update kustomization.yaml 2026-05-29 08:42:41 -04:00
sirius0xdev
f49e3c9037 Merge branch 'master' of github.com:sirius0xdev/gcloud-lab
fuck me#
2026-05-28 03:17:03 +00:00
sirius0xdev
37cf8d1276 Merge branch 'master' of github.com:sirius0xdev/gcloud-lab
please wprl#
2026-05-27 02:19:52 +00:00
sirius0xdev
789903c835 Merge branch 'master' of github.com:sirius0xdev/gcloud-lab 2026-05-26 19:02:01 +00:00
sirius0xdev
9b949ed0bd move reddis 2026-05-26 18:28:52 +00:00
180 changed files with 213 additions and 11150 deletions

View file

@ -1,203 +0,0 @@
# Repository Reorganization Plan
## Goal
Enforce the rule: **gcloud-lab = Kubernetes manifests only. No application code.**
Application code (source, Dockerfiles, CI/CD workflows, Helm charts, deployment scripts) belongs in `hermes-projects/`. gcloud-lab should contain only K8s manifests, Terraform infrastructure modules, cluster configs, and infra controller configs.
---
## Current State
### gcloud-lab/ — Full Directory Audit
```
gcloud-lab/
├── apps/
│ ├── base/
│ │ ├── customer1/ [KEEP] K8s manifests (deployments, services, configmaps, secrets)
│ │ ├── monitoring/ [KEEP] K8s manifests (dashboards)
│ │ └── osint-dashboard/ [KEEP] Helm chart (templates, values.yaml, Chart.yaml)
│ ├── staging/
│ │ ├── customer1/ [KEEP] K8s overlay (kustomization.yaml)
│ │ └── osint-dashboard/ [KEEP] K8s overlay (kustomization.yaml)
│ └── vwap-monitor/ [MOVE] App code (Dockerfile, app/, deploy/)
├── clusters/ [KEEP] Cluster configs (devops-lab/*.yaml, flux-system/)
├── infrastructure/ [KEEP] Infra controllers, gatewayapi, gpus, tailnet
├── misc/ [KEEP] Terraform snippets + K8s YAML
├── modules/ [KEEP] Terraform modules (gke.tf, nodepool.tf, etc.)
├── scripts/ [KEEP] Setup scripts
├── .github/workflows/
│ ├── osint-dashboard-infra.yml [KEEP] Infra deployment workflow
│ └── trade-dashboard.yml [MOVE] CI for trade-dashboard app → hermes-projects/
├── .devcontainer.json [KEEP] Dev environment config
├── .sops.yaml [KEEP] SOPS encryption config
├── .terraform.lock.hcl [KEEP] Terraform lock
├── .gitignore [KEEP] Git ignore rules
├── mise.toml [KEEP] Tool version management
├── README.md [KEEP] (will be updated)
├── infra-tailnet.yaml [KEEP] Tailscale infra config
├── tailscale-0auth.yaml [KEEP] Tailscale config
├── rays-new-deployment.yaml [REVIEW] Orphan K8s deployment YAML — move to apps/base/
├── trade-dashboard/ [MOVE] Full FastAPI app → hermes-projects/trade-dashboard/
├── trading-platform/ [MOVE] Duplicate/deploy configs → hermes-projects/trading-platform/
├── trading-scripts/ [MOVE] Application code → hermes-projects/trading-scripts/
├── analyses/ [REMOVE] Research artifacts (not code, not infra)
└── plans/ [REMOVE] Planning docs (not code, not infra)
```
---
## Items to Move / Remove
### 1. `trade-dashboard/` → hermes-projects/trade-dashboard/
**What it is:** Full FastAPI application (not K8s manifests)
- `Dockerfile` — build config for the app
- `app/` — Python source code (main.py, models.py, schemas.py, database.py, requirements.txt, static/)
- `alembic/` — database migration scripts (env.py, versions/)
- `alembic.ini` — alembic config
**Also move:** `.github/workflows/trade-dashboard.yml` (CI workflow for this app)
**Already in gcloud-lab:** `apps/base/customer1/trade-dashboard/` — these are the K8s manifests for trade-dashboard (deployment.yaml, service.yaml, configmap.yaml, kustomization.yaml). **KEEP these** — they belong here.
### 2. `trading-platform/` → hermes-projects/trading-platform/
**What it is:** Duplicate/alternative deployment configs that overlap with hermes-projects/trading-platform/
Contents:
- `.github/workflows/` — 3 CI/CD workflows (build-push.yml, build-test.yml, deploy.yml)
- `README.md` — project readme
- `deploy/` — deployment configs:
- `ci-cd/` — additional CI workflows
- `docker-compose/` — docker-compose.dev.yml
- `dockerfiles/` — Dockerfiles (api-gateway, dashboard, data-service, execute-service, news-service)
- `helm/` — Helm charts (api-gateway, dashboard, data-service, execute-service, news-service)
- `k8s/` — raw K8s manifests (deployments, services, hpa, cert-manager, ingress)
- `mtls/` — mTLS README
- `scripts/` — deploy.sh, generate-mtls-certs.sh
- `dockerfiles/` — Dockerfiles (dashboard, data-service, execute-service, news-service)
- `helm/` — Helm chart with templates (trading-platform chart, values.yaml, secrets)
**Already in gcloud-lab:** `apps/base/customer1/trading-platform/` — these are the K8s manifests. **KEEP these** — they belong here.
**Already in hermes-projects:** `hermes-projects/trading-platform/` — source code exists here (dashboard, data-service, execute-service, news-service, data_infrastructure). The gcloud-lab trading-platform/ deploy/dockerfiles/helm content should be MERGED into hermes-projects/trading-platform/.
**Decision needed:** The `trading-platform/` in gcloud-lab has BOTH deploy configs (dockerfiles, helm, k8s manifests) AND CI workflows. The K8s manifests in `deploy/k8s/base/` are similar but NOT identical to what's in `apps/base/customer1/trading-platform/`. Need to decide which is authoritative.
### 3. `trading-scripts/` → hermes-projects/trading-scripts/
**What it is:** Application code (Python trading scripts)
- `market_data.py` — market data script
- `orb-monitor/` — monitoring tool (monitor.py, config.yaml)
- `README.md`, `ROADMAP.md` — documentation
### 4. `apps/vwap-monitor/` → hermes-projects/vwap-monitor/
**What it is:** Application code with a Dockerfile
- `Dockerfile` — build config
- `app/` — source code (scanner.py, requirements.txt)
- `deploy/` — deployment config (deployment.yaml, kustomization.yaml, secret.yaml, config.env)
**Note:** The `deploy/` subdirectory contains K8s manifests. These should be moved BACK into gcloud-lab as `apps/base/customer1/vwap-monitor/`. The app code (Dockerfile + app/) goes to hermes-projects.
### 5. `analyses/` → REMOVE from gcloud-lab
**What it is:** Research/analysis markdown documents
- `telegram-webhook-container-analysis.md`
- `telegram-webhook-failure-analysis.md`
These are one-time research artifacts, not infrastructure config. Remove from gcloud-lab entirely.
### 6. `plans/` → REMOVE from gcloud-lab
**What it is:** Planning/strategy markdown documents
- `2026-04-25-openclaw-brain-v1.1.md`
- `AI_ARCHITECTURE.md`
- `models-to-try.md`
These are planning docs, not infrastructure config. Remove from gcloud-lab entirely.
### 7. `rays-new-deployment.yaml` → REVIEW
**What it is:** A standalone K8s deployment YAML at repo root.
**Action:** Move to `apps/base/customer1/hermes-agent/` (appears related to hermes-agent/rays deployment based on filename). Already similar files exist in that directory.
---
## Proposed Migration Plan (Ordered by PR)
### PR 1: This Plan (docs only)
- Add `MIGRATION_PLAN.md` (this file)
- Update `README.md` to document the new structure
### PR 2: Remove planning/research docs
- Delete `analyses/` directory
- Delete `plans/` directory
- Low risk, no dependencies
### PR 3: Move trade-dashboard app to hermes-projects
- Move `trade-dashboard/` → hermes-projects/trade-dashboard/
- Move `.github/workflows/trade-dashboard.yml` → hermes-projects/.github/workflows/
- K8s manifests in `apps/base/customer1/trade-dashboard/` stay in place
- Verify image references in K8s manifests still point to correct registry
### PR 4: Move trading-platform deploy configs to hermes-projects
- Move `trading-platform/` → merge with hermes-projects/trading-platform/
- CI workflows → hermes-projects/trading-platform/.github/workflows/
- Dockerfiles → hermes-projects/trading-platform/dockerfiles/
- Helm charts → hermes-projects/trading-platform/helm/
- K8s manifests from `trading-platform/deploy/k8s/` → reconcile with `apps/base/customer1/trading-platform/`
- **Decision needed:** Which K8s manifests are authoritative? The ones in gcloud-lab/apps/ or trading-platform/deploy/k8s/?
### PR 5: Move trading-scripts to hermes-projects
- Move `trading-scripts/` → hermes-projects/trading-scripts/
- Simple move, no K8s manifest reconciliation needed
### PR 6: Split vwap-monitor (app → hermes-projects, K8s → gcloud-lab)
- Move `apps/vwap-monitor/app/` + `apps/vwap-monitor/Dockerfile` → hermes-projects/vwap-monitor/
- Move `apps/vwap-monitor/deploy/` K8s manifests → `apps/base/customer1/vwap-monitor/`
- Update image references in K8s manifests
---
## Items That Stay in gcloud-lab (No Changes)
| Path | Reason |
|------|--------|
| `apps/base/customer1/` | K8s manifests (kustomize structure) |
| `apps/base/monitoring/` | K8s manifests (dashboards) |
| `apps/base/osint-dashboard/` | Helm chart for infra |
| `apps/staging/` | K8s overlays |
| `clusters/` | Cluster configs, flux-system |
| `infrastructure/` | Controllers, gatewayapi, gpus, tailnet |
| `misc/` | Terraform snippets + K8s YAML |
| `modules/` | Terraform modules |
| `scripts/` | Setup scripts |
| Root config files | .sops.yaml, .devcontainer.json, mise.toml, .gitignore, .terraform.lock.hcl |
| `infra-tailnet.yaml` | Tailscale infra config |
| `tailscale-0auth.yaml` | Tailscale config |
---
## K8s Manifest Reference Check
After moves, verify these image references still resolve:
| K8s Manifest | Image Reference |
|--------------|----------------|
| `apps/base/customer1/trade-dashboard/deployment.yaml` | Check image tag matches hermes-projects build |
| `apps/base/customer1/trading-platform/*/deployment.yaml` | Check image tags match hermes-projects build |
| `apps/base/customer1/hermes-agent/deployment.yaml` | N/A (already correct) |
| `apps/base/customer1/siriusdevops-site/deployment.yaml` | N/A (already correct) |
---
## Decisions Needed Before Proceeding
1. **trading-platform K8s manifest authority:** `trading-platform/deploy/k8s/base/` vs `apps/base/customer1/trading-platform/` — which is the source of truth?
2. **analyses/ and plans/:** Delete entirely, or archive somewhere else?
3. **rays-new-deployment.yaml:** Move to `apps/base/customer1/hermes-agent/` or delete?

688
README.md
View file

@ -1,524 +1,246 @@
# GCloud-Lab DevOps Infrastructure # gcloud-lab
A production-grade cloud-native infrastructure laboratory demonstrating GitOps, multi-tenant AI agent hosting, and automated security pipelines — all run by a single DevOps engineer on Google Cloud Platform. Trusted by builders who ship. GitOps + Terraform source of truth for a **GKE lab** I designed, ran, and then **shut down** once the GPU bill stopped being worth it.
## Table of Contents This is not a live cluster. It is the manifests, node-pool definitions, and GitOps wiring from a real environment that served vLLM inference, CNPG databases, and a handful of in-cluster apps. The same cost model that made the GPU pools scale to zero is why the whole footprint went to zero.
- [Project Overview](#project-overview) **Canonical copy:** [forgejo.siriusdevops.com/sirius/gcloud-lab](https://forgejo.siriusdevops.com/sirius/gcloud-lab)
- [Architecture](#architecture)
- [DevOps Tools & Technologies](#devops-tools--technologies)
- [Monitoring](#monitoring)
- [Project Structure](#project-structure)
- [Infrastructure Components](#infrastructure-components)
- [Applications](#applications)
- [Getting Started](#getting-started)
- [Security](#security)
- [Cost Optimization](#cost-optimization)
- [License](#license)
--- ---
## Project Overview ## What this is evidence of
This repository is the single source of truth for a multi-application cloud platform running on GKE. Every deployment, database, and network policy flows through Git via Flux CD. What lives here: If you are reading this as a hiring screen, start here. Every claim below maps to a file in this repo.
1. **AgentForge** — Private multi-tenant AI agent workspace with dual-tier vLLM inference (L4 dispatcher + RTX 6000 deep thinker) and isolated CNPG databases per tenant. | Claim | Where to look |
2. **Multi-Profile AI Agent Team** — Six specialist AI profiles (backend-dev, frontend-dev, researcher, outreach, quant, sec-ops) orchestrated through a shared Kanban board with automated audit-to-fix pipelines. | --- | --- |
3. **Waitlist API** — FastAPI landing page backend with idempotent signups, async PostgreSQL, and Telegram fire-and-forget notifications. | GKE cluster, custom VPC, dual-stack, Cilium datapath | [`modules/gke.tf`](modules/gke.tf), [`modules/vpc.tf`](modules/vpc.tf) |
4. **Autonomous News Quant Pipeline** — 371 global feed scraper with DeepSeek-R1 analysis generating actionable futures trading signals. | CPU + three GPU node pools (L4, RTX PRO 6000, A100 80GB), all SPOT except CPU | [`modules/nodepool.tf`](modules/nodepool.tf), [`modules/nodepool-gpu.tf`](modules/nodepool-gpu.tf), [`modules/pro6000-nodepool.tf`](modules/pro6000-nodepool.tf), [`modules/a100-nodepool.tf`](modules/a100-nodepool.tf) |
5. **N8N Workflow Automation** — Self-hosted workflow engine with dedicated CNPG PostgreSQL. | Flux CD applies the tree; SOPS decrypts secrets in-cluster | [`clusters/devops-lab/`](clusters/devops-lab/), [`.sops.yaml`](.sops.yaml) |
6. **Local Business Web Deployment Pipeline** — Automated K8s manifest generation for small business websites with cross-namespace HTTPRoute routing. | Production-style **vLLM** OpenAI-compatible servers (not Ollama) | [`infrastructure/gpus/base/vllm-servers/`](infrastructure/gpus/base/vllm-servers/) |
| KEDA HTTP scale-to-zero on the expensive GPUs | [`infrastructure/gpus/base/keda-gpu-scaling/`](infrastructure/gpus/base/keda-gpu-scaling/) |
| CloudNative-PG operator + per-app Postgres | [`infrastructure/controllers/base/cnpg/`](infrastructure/controllers/base/cnpg/), `apps/base/customer1/*-db/` |
| Gateway API (not legacy Ingress) + Tailscale for internals | [`infrastructure/gatewayapi/`](infrastructure/gatewayapi/), [`infrastructure/tailnet/`](infrastructure/tailnet/) |
| Workload NetworkPolicies (trading stack) | [`apps/base/customer1/trading-platform/network-policies/`](apps/base/customer1/trading-platform/network-policies/) |
I run 24/7 infrastructure now on a Raspberry Pi (Forgejo, Cloudflare tunnel, containerized sites). This repo is the cloud chapter that came before that.
--- ---
## Architecture ## Architecture
```text ```
┌──────────────────────────────────────────────────────────────────────────────┐ Git (this repo, branch master)
│ Google Cloud Platform │
│ ┌────────────────────────────────────────────────────────────────────────┐ │ Flux source-controller (1m)
│ │ GKE Cluster (devops-lab-cluster) │ │
│ │ │ │ ┌─────────────────┼──────────────────┐
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ ▼ ▼ ▼
│ │ │ Standard │ │ L4 GPU Pool │ │ RTX 6000 GPU │ │ │ infra-controllers infra-gpus apps/staging
│ │ │ Node Pool │ │ (SPOT L4) │ │ (SPOT RTX6K) │ │ │ CNPG / KEDA vLLM + KEDA customer1 overlay
│ │ │ e2-std-2 │ │ 1 node (24/7)│ │ 0-1 nodes │ │ │ cert-manager HTTPScaledObject (kustomize)
│ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │ Tailscale L4 / RTX6000 / A100
│ │ │ │ kube-prometheus
│ │ ┌──────────────────────────────────────────────────────────────────┐ │ │ │ │ │
│ │ │ Cilium CNI + Hubble + NetworkPolicy │ │ │ └──────────── GKE us-central1-a ─────┘
│ │ └──────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │ ┌───────────────┬────────────┼────────────┬──────────────┐
│ │ ┌──────────────────────────────────────────────────────────────────┐ │ │ ▼ ▼ ▼ ▼ ▼
│ │ │ Kubernetes Gateway API — external-http-gateway │ │ │ e2-standard-2 g2-standard-8 g4-standard-48 a2-ultragpu-1g
│ │ │ HTTPRoute PathPrefix → AgentForge / Waitlist / Apps │ │ │ CPU pool L4 SPOT RTX PRO 6000 A100 80GB SPOT
│ │ └──────────────────────────────────────────────────────────────────┘ │ │ 15 nodes 1 node SPOT 01 SPOT 01
│ │ │ │ (always on) KEDA 0↔1 KEDA 0↔1
│ │ ┌───────────────────────────────┐ ┌───────────────────────────────┐ │ │
│ │ │ customer1 namespace │ │ agent-forge namespace │ │ │
│ │ │ - AgentForge (PAaaS) │ │ - Tenant-specific Hermes Agent │ │ │
│ │ │ - Dual-tier vLLM │ │ - Isolated CNPG databases │ │ │
│ │ │ L4 Dispatcher (24/7) │ │ - Qwen 3.6 27B Abliterated │ │ │
│ │ │ RTX 6000 Deep Thinker (KEDA) │ │ - KEDA scale-to-zero │ │ │
│ │ │ - Waitlist API (FastAPI) │ │ │ │ │
│ │ │ - News Bot Pipeline │ │ ┌───────────────────────────┐ │ │ │
│ │ │ - Landing Page │ │ │ sec-ops audit agent │ │ │
│ │ │ - CNPG PostgreSQL Cluster │ │ │ Automated vuln scanning │ │ │
│ │ └───────────────────────────────┘ │ │ → backend-dev auto-fix │ │ │
│ │ │ └───────────────────────────┘ │ │ │
│ │ ┌───────────────────────────────┐ └───────────────────────────────┘ │ │
│ │ │ local-business namespaces │ │ │
│ │ │ - nginx + ConfigMap per biz │ ┌───────────────────────────────┐ │ │
│ │ │ - Cross-ns HTTPRoute refs │ │ monitoring namespace │ │ │
│ │ └───────────────────────────────┘ │ - Prometheus + Grafana │ │ │
│ │ │ - Tailscale-only access │ │ │
│ │ ┌───────────────────────────────┐ │ - No public ingress │ │ │
│ │ │ kanban namespace │ └───────────────────────────────┘ │ │
│ │ │ - Hermes Agent Orchestrator │ │ │
│ │ │ - 6 Specialist Profiles │ ┌───────────────────────────────┐ │ │
│ │ │ - Isolated hermes-pgdb │ │ n8n namespace │ │ │
│ │ └───────────────────────────────┘ │ - Workflow automation │ │ │
│ │ │ - Dedicated PostgreSQL │ │ │
│ │ └───────────────────────────────┘ │ │
│ └────────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────────┘
``` ```
--- Datapath: GKE `ADVANCED_DATAPATH` + `enable_cilium_clusterwide_network_policy`. Dual-stack VPC (`10.0.0.0/16`, pods `192.168.32.0/20`, services `192.168.16.0/24`). GPU nodes are tainted (`nvidia.com/gpu…=present:NoSchedule`) so only inference pods land on them.
## DevOps Tools & Technologies
### Infrastructure as Code (IaC)
| Tool | Version | Purpose |
|------|---------|---------|
| **Terraform** | 1.7+ | Infrastructure provisioning for GCP resources |
| **Google Provider** | 7.14.1 | Terraform provider for GCP |
| **Helm Provider** | Latest | Terraform provider for Helm charts |
| **Flux Provider** | 1.7.6 | Terraform provider for Flux bootstrap |
### Container Orchestration & Networking
| Tool | Version | Purpose |
|------|---------|---------|
| **Google Kubernetes Engine (GKE)** | Latest | Managed Kubernetes cluster |
| **Cilium** | 1.18.5 | CNI plugin with eBPF-based networking |
| **Hubble** | 1.18.5 | Network observability and monitoring |
| **Kubernetes Gateway API** | v1 | Ingress routing and traffic management |
### GitOps & Configuration Management
| Tool | Version | Purpose |
|------|---------|---------|
| **Flux CD** | 1.7.6 | GitOps continuous delivery |
| **Kustomize** | v1beta1 | Kubernetes manifest customization |
| **Helm** | 3+ | Kubernetes package manager |
| **SOPS** | Latest | Secrets encryption in Git |
| **Age** | Latest | Modern encryption for SOPS |
### Database
| Tool | Version | Purpose |
|------|---------|---------|
| **CloudNative PG** | 0.26.1 | PostgreSQL Kubernetes operator |
| **PostgreSQL** | 15.2 | Relational database (multi-cluster fleet) |
### AI/ML Infrastructure
| Tool | Version | Purpose |
|------|---------|---------|
| **vLLM** | v0.9.1 | High-throughput LLM inference server |
| **Qwen 3.6 27B Abliterated** | Latest | Uncensored reasoning model (RTX 6000 deep thinker tier) |
| **Qwen 2.5 Coder 7B Abliterated** | Latest | Fast tool-calling dispatcher (L4 24/7 tier) |
| **NVIDIA L4 GPU** | - | 24/7 GPU for fast triage and dispatch |
| **NVIDIA RTX 6000 Pro** | - | SPOT GPU for deep reasoning and multi-file context |
### Development Environment
| Tool | Version | Purpose |
|------|---------|---------|
| **Mise** | Latest | Development tool version manager |
| **Dev Containers** | Latest | Consistent development environment |
| **k9s** | Latest | Kubernetes CLI dashboard |
### Monitoring & Observability
| Tool | Version | Purpose |
|------|---------|---------|
| **Prometheus** | Latest | Metrics collection via kube-prometheus-stack |
| **Grafana** | Latest | Dashboards & visualizations |
| **Tailscale** | Latest | Secure VPN access to internal services |
--- ---
## Project Structure ## GPU inference (vLLM)
Three independent OpenAI-compatible servers, each pinned to a pool via `nodeSelector` + matching taint/toleration. Model weights cached on PVC so a scale-up does not re-pull 2040 GB from Hugging Face.
### L4 dispatcher — always on
[`infrastructure/gpus/base/vllm-servers/vllm-l4.yaml`](infrastructure/gpus/base/vllm-servers/vllm-l4.yaml)
- Machine: `g2-standard-8` + `nvidia-l4`, SPOT
- Image: `vllm/vllm-openai`
- Model: `p-e-w/Qwen3-8B-heretic`
- Flags that matter: `--kv-cache-dtype=fp8`, `--enable-chunked-prefill`, `--enable-prefix-caching`, `--enable-auto-tool-choice`, `--tool-call-parser=hermes`, `--max-model-len=32768`
Cheap, tool-capable, left running so agents had a low-latency brain even when the big cards were scaled out.
### RTX PRO 6000 — deep context, scale to zero
[`infrastructure/gpus/base/vllm-servers/rtx6000-vllm.yaml`](infrastructure/gpus/base/vllm-servers/rtx6000-vllm.yaml)
[`infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml`](infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml)
- Machine: `g4-standard-48` + `nvidia-rtx-pro-6000`, SPOT, **min 0 / max 1**
- Image: `vllm/vllm-openai:latest-cu129-ubuntu2404`
- Model: `edp1096/Huihui-Qwen3.6-27B-abliterated-FP8`
- `--max-model-len=262144`, `--enable-chunked-prefill`, `--tool-call-parser=qwen3_xml`, `--reasoning-parser=qwen3`, MTP speculative decoding (`num_speculative_tokens: 2`)
- Startup / liveness / readiness probes on `/health` and `/v1/models` (model load is slow; `failureThreshold: 60` on startup)
- KEDA `HTTPScaledObject`: `replicas.min: 0`, `max: 1`, host `rtx6000-brain-service.customer1.svc.cluster.local`
Idle deep-thinker capacity cost nothing. A request woke the Deployment; GKE then scaled the node pool off zero.
### A100 80GB — same pattern
[`infrastructure/gpus/base/vllm-servers/a100-vllm.yaml`](infrastructure/gpus/base/vllm-servers/a100-vllm.yaml)
[`modules/a100-nodepool.tf`](modules/a100-nodepool.tf)
- Machine: `a2-ultragpu-1g` + `nvidia-a100-80gb`, SPOT, min 0
- NVFP4 / ModelOpt quantized 27B, `--quantization=modelopt`, `--attention-backend=flash_attn`, `--kv-cache-dtype=fp8`
---
## GitOps
Flux watches `master` and applies layered Kustomizations:
```
clusters/devops-lab/
flux-system/ Flux controllers + GitRepository
infra-controllers.yaml → infrastructure/controllers/staging
CNPG 0.26.1, KEDA ≥2.14, cert-manager, Tailscale, kube-prometheus
infra-gpus.yaml → infrastructure/gpus/staging
infra-gatewayapi.yaml → Gateway API + HTTPRoutes
customer1-strimzi.yaml → Strimzi (Kafka) first
customer1.yaml → apps/staging/customer1 (dependsOn strimzi)
```
Every Flux Kustomization has `decryption.provider: sops` and `secretRef: sops-age`. Encrypted `data`/`stringData` in Git; Flux decrypts at apply time. Age recipient is in [`.sops.yaml`](.sops.yaml). The private key is **not** in this repo.
Bootstrap was originally:
```bash
flux bootstrap github \
--owner=sirius0xdev \
--repository=gcloud-lab \
--branch=master \
--path=clusters/devops-lab
```
`gotk-sync.yaml` still records that GitHub URL. This Forgejo copy is the archive; the cluster is gone, so the GitRepository object was never re-pointed.
---
## Terraform (cluster birth)
[`modules/`](modules/) is the IaC that created the cluster, not the day-to-day delivery path.
| File | Resource |
| --- | --- |
| `gke.tf` | `devops-lab-cluster` in `us-central1-a`, default pool removed, Cilium datapath, dual-stack IP policy |
| `vpc.tf` | `devops-lab-network` / `devops-lab-subnetwork`, ULA IPv6, secondary ranges |
| `nodepool.tf` | CPU `e2-standard-2`, 15, pd-standard 100 Gi |
| `nodepool-gpu.tf` | L4 SPOT `g2-standard-8`, autoscaling 11, GPU taint |
| `pro6000-nodepool.tf` | RTX PRO 6000 SPOT `g4-standard-48`, **01**, hyperdisk-balanced |
| `a100-nodepool.tf` | A100 80GB SPOT `a2-ultragpu-1g`, **01**, pd-ssd 200 Gi |
| `db-bucket.tf` | GCS bucket for CNPG backups |
| `providers.tf` | Google provider, Helm talking to the cluster via `gke-gcloud-auth-plugin` |
State files and `*.tfvars` are gitignored. Do not `terraform apply` this against a live billing account unless you intend to recreate it.
---
## Data plane and apps
Flux overlay: [`apps/staging/customer1/kustomization.yaml`](apps/staging/customer1/kustomization.yaml)
| Piece | Role |
| --- | --- |
| CloudNative-PG | Operator `0.26.1`; per-app clusters (Hermes memory, waitlist, n8n, trading) with GCS backup |
| Strimzi | Kafka for the trading data path; Flux `dependsOn` so apps wait for the operator |
| Redis | In-cluster cache next to the trading services |
| Trading platform | data / execute / news / dashboard Deployments, HTTPRoutes, NetworkPolicies |
| Hermes agent | In-cluster agent + CNPG, talking to the vLLM services |
| News bot | CronJobs: scrape → analyst (vLLM) → Telegram |
| Waitlist API | FastAPI + CNPG, Gateway HTTPRoute |
| n8n | Workflow engine on its own Postgres |
| Gateway API | Public HTTPRoutes; Grafana/Prometheus stayed off the public internet (Tailscale / port-forward) |
App **images** were built in a separate code repo and pulled from GHCR. This repo is manifests only (after cleanup: no Dockerfiles, no Helm-of-the-app, no planning markdown).
---
## Networking and security
- **Cilium** as GKE datapath, clusterwide NetworkPolicy enabled from Terraform.
- **Gateway API** `HTTPRoute` for public paths; internals not published.
- **Tailscale operator** for operator access to Grafana and cluster services without a public LB.
- **NetworkPolicies** on the trading namespace: DNS, Postgres, Redis, Kafka, in-namespace HTTP, egress HTTPS to market APIs. Everything else denied.
- **SOPS + age** for Secrets in Git. Flux `sops-age` Secret in `flux-system` held the private key.
- GPU nodes tainted so a random Deployment cannot schedule onto a $2+/hr card.
---
## Cost model (why it was torn down)
This was the whole point of the GPU design:
1. L4 SPOT stayed at 1 node — cheap enough to keep a dispatcher warm.
2. RTX 6000 and A100 pools **autoscaled 01**. KEDA HTTPScaledObject set the Deployment to 0 when there was no traffic; the node pool followed.
3. Weights on PVC, long startup probes — first request after idle paid a cold-start, not a 40 GB pull.
4. CronJobs for batch work instead of idle inference pods.
5. When even the L4 + control-plane bill stopped making sense, the cluster was destroyed. Scale-to-zero was the rehearsal for scale-to-nothing.
---
## Repository layout
``` ```
gcloud-lab/ gcloud-lab/
├── modules/ # Terraform IaC modules ├── modules/ Terraform: VPC, GKE, node pools, GCS
│ ├── providers.tf # Provider configurations ├── clusters/devops-lab/ Flux entry (GitRepository + Kustomizations)
│ ├── gke.tf # GKE cluster definition ├── infrastructure/
│ ├── vpc.tf # VPC and subnet configuration │ ├── controllers/ CNPG, KEDA, cert-manager, Tailscale, Prometheus
│ ├── nodepool.tf # Standard node pool │ ├── gpus/ vLLM Deployments + KEDA HTTPScaledObjects
│ ├── nodepool-gpu.tf # GPU node pools (L4 + RTX 6000 SPOT) │ ├── gatewayapi/ Gateway + HTTPRoutes
│ ├── flux.tf # Flux GitOps bootstrap │ └── tailnet/ Tailscale ProxyGroup
│ ├── helm.tf # Helm chart deployments (Cilium) ├── apps/
│ └── variables.tf # Input variables │ ├── base/customer1/ Namespaced workloads (kustomize)
│ ├── base/osint-dashboard/ Helm chart used on this cluster
├── clusters/ # Cluster configurations │ └── staging/ Overlays Flux actually syncs
│ └── devops-lab/ ├── monitoring/ Extra Grafana/Prometheus config
│ ├── flux-system/ # Flux CD components ├── .sops.yaml Age recipient for secret encryption
│ │ ├── gotk-components.yaml # Flux controllers ├── .github/workflows/ Historical GH Actions (pgvector image, etc.)
│ │ ├── gotk-sync.yaml # Git repository sync └── mise.toml Local CLI pin (gcloud, kubectl, helm, sops, terraform, k9s)
│ │ └── kustomization.yaml # Flux kustomization
│ ├── customer1.yaml # Customer1 Kustomization
│ ├── agent-forge.yaml # AgentForge Kustomization
│ ├── infra-controllers.yaml # Infrastructure controllers (CNPG, KEDA, Monitoring, Tailscale)
│ └── infra-configs.yaml # Infrastructure configs
├── infrastructure/ # Infrastructure components
│ ├── controllers/
│ │ ├── base/
│ │ │ ├── cnpg/ # CloudNative PG operator
│ │ │ ├── keda/ # KEDA autoscaling
│ │ │ ├── monitoring/ # Prometheus + Grafana (no public ingress)
│ │ │ └── tailscale/ # Tailscale Operator for secure VPN access
│ │ └── staging/
│ │ └── kustomization.yaml # Aggregates all base components
│ └── configs/
│ └── staging/
│ └── kustomization.yaml
├── apps/ # Application deployments
│ ├── base/
│ │ ├── customer1/
│ │ │ ├── namespace.yaml # Namespace definition
│ │ │ ├── deployment.yaml # N8N + vLLM deployments
│ │ │ ├── service.yaml # ClusterIP services
│ │ │ ├── storage.yaml # PersistentVolumeClaims
│ │ │ ├── configmap.yaml # Application configuration
│ │ │ ├── pg-cluster-customer1.yaml # PostgreSQL cluster
│ │ │ ├── apigateway.yaml # GCP Gateway
│ │ │ ├── http-route.yaml # HTTP routing
│ │ │ ├── healthcheck.yaml # Health check policy
│ │ │ ├── waitlist-api/ # Waitlist API microservice
│ │ │ │ ├── deployment.yaml
│ │ │ │ ├── service.yaml
│ │ │ │ └── configmap.yaml
│ │ │ └── news_bot/ # News bot microservices
│ │ │ ├── scraper-cronjob.yaml
│ │ │ ├── analyst-cronjob.yaml
│ │ │ ├── telebot-cronjob.yaml
│ │ │ ├── scrapy-configmap.yaml
│ │ │ └── scrapy-urls-configmap.yaml
│ │ ├── agent-forge/
│ │ │ ├── namespace.yaml
│ │ │ ├── vllm-deep-thinker.yaml # RTX 6000 deployment with KEDA
│ │ │ ├── hermes-tenant.yaml # Per-tenant Hermes agent instance
│ │ │ └── pg-cluster-agentforge.yaml
│ │ ├── kanban/
│ │ │ ├── namespace.yaml
│ │ │ ├── hermes-deployment.yaml # AI agent orchestrator
│ │ │ └── pg-cluster-hermes.yaml
│ │ └── local-business/
│ │ └── template/
│ │ ├── namespace.yaml
│ │ ├── nginx-deployment.yaml
│ │ ├── configmap.yaml
│ │ └── http-route.yaml
│ └── staging/
│ ├── customer1/
│ │ └── kustomization.yaml
│ ├── agent-forge/
│ │ └── kustomization.yaml
│ └── kanban/
│ └── kustomization.yaml
├── scripts/
│ └── setup # Development setup script
├── .devcontainer.json # Dev container configuration
├── mise.toml # Tool version management
├── age.agekey # SOPS encryption key
└── README.md # This file
``` ```
--- ---
## Infrastructure Components ## Reading the code (suggested order)
### GKE Cluster 1. [`modules/gke.tf`](modules/gke.tf) + [`modules/vpc.tf`](modules/vpc.tf) — cluster shape.
2. GPU pools, then the matching vLLM YAML — taint keys must match or the pod never schedules.
- **Name**: `devops-lab-cluster` 3. [`clusters/devops-lab/customer1.yaml`](clusters/devops-lab/customer1.yaml) — Flux `dependsOn`, SOPS, prune/force.
- **Region**: `us-central1-a` 4. [`apps/staging/customer1/kustomization.yaml`](apps/staging/customer1/kustomization.yaml) — what actually shipped in `customer1`.
- **Network**: Custom VPC with dual-stack IPv4/IPv6 5. One NetworkPolicy under `apps/base/customer1/trading-platform/network-policies/` — default-deny thinking.
### Node Pools
| Pool | Machine Type | Scaling | Purpose |
|------|-------------|---------|---------|
| Standard | e2-standard-2 | 1-16 nodes | General workloads, N8N, web servers |
| GPU L4 (SPOT) | g2-standard-8 + L4 | 0-5 nodes | vLLM dispatcher, 24/7 fast inference |
| GPU RTX 6000 (SPOT) | g6-standard-4 + RTX 6000 Pro | 0-1 nodes | Deep thinker tier, multi-file reasoning |
### Networking
- **VPC**: `devops-lab-network`
- **Primary CIDR**: `10.0.0.0/16`
- **Pod CIDR**: `192.168.32.0/20`
- **Service CIDR**: `192.168.16.0/24`
- **CNI**: Cilium with advanced datapath and NetworkPolicy enforcement
- **Ingress**: Kubernetes Gateway API via `external-http-gateway` with PathPrefix HTTPRoute routing
- **Internal Services**: Tailscale-only — no public ingress for monitoring, databases, or agent infrastructure
### CNPG Database Fleet
Multiple isolated PostgreSQL clusters, each with dedicated databases per application:
| Cluster | Namespace | Databases | Backup |
|---------|-----------|-----------|--------|
| `customer1-pgdb` | customer1 | `n8n`, `news_app`, `waitlist` | GCS, 7-day retention |
| `hermes-pgdb` | kanban | `hermes`, `memory_store` | GCS, 7-day retention |
| `hermes-tenant-pgdb` | agent-forge | Per-tenant isolated DBs | GCS, 7-day retention |
| `siriusdevops-pgdb` | customer1 | `waitlist_prod` | GCS, 30-day retention |
### GitOps Flow
```
GitHub Repository (ghcr.io/sirius0xdev)
Flux Source Controller (watches git, 1min interval)
Flux Kustomize Controller (applies manifests)
├── infrastructure/controllers → CNPG, KEDA, Monitoring, Tailscale
├── infrastructure/configs → Cluster configs
├── apps/staging/customer1 → PAaaS, N8N, News Bot, Waitlist API
├── apps/staging/agent-forge → Multi-tenant AI agent hosting
├── apps/staging/kanban → AI Agent Team orchestrator
└── apps/staging/local-business → Business websites
```
--- ---
## Applications ## Local tooling
### 1. AgentForge — Private AI Agent Workspace
A premium, uncensored, privacy-first AI agent hosting platform with dual-tier cognitive architecture:
- **Tier 1 (Dispatcher):** L4 GPU SPOT instance running 24/7. Hosts `Qwen2.5-Coder-7B-Instruct-heretic` via vLLM `v0.9.1` for lightning-fast, cheap triage and tool calling.
- **Tier 2 (Deep Thinker):** RTX 6000 Pro Spot instance scaling from 0-1 via KEDA. Hosts `Qwen3.5-27B-heretic` with `--enable-chunked-prefill` and `--kv-cache-dtype=fp8` for massive multi-file context and reasoning without OOMing or stalling concurrent users.
- **Frontend:** Isolated Hermes agent profiles per tenant, connected to Telegram/Discord via outbound polling (no public ingress required).
- **Landing Page:** Dockerized marketing site built via CI/CD from `hermes-projects` and deployed to the `staging` kustomization overlay.
- **Container Registry:** All images pushed to `ghcr.io/sirius0xdev`.
### 2. Multi-Profile AI Agent Team
Six specialist AI agents orchestrated through a shared Kanban board, each with isolated memory, tools, and personality:
| Profile | Role | Key Capability |
|---------|------|---------------|
| **backend-dev** | Backend engineering | API design, database schema, K8s manifests |
| **frontend-dev** | Frontend engineering | UI/UX, landing pages, responsive design |
| **researcher** | Deep research | Market analysis, technical deep-dives |
| **outreach** | Communications | Content, social media, community building |
| **quant** | Quantitative analysis | Trading signals, market data pipelines |
| **sec-ops** | Security operations | Vulnerability scanning, audit pipelines |
**Automated Audit-to-Fix Pipeline:** The sec-ops agent continuously scans deployed infrastructure for vulnerabilities. When findings are confirmed, the backend-dev agent is automatically dispatched to remediate — from detection to patch in a single GitOps cycle.
### 3. Gateway API and HTTPRoute
Kubernetes Gateway API replaces legacy Ingress with a clean, declarative routing model:
- **Single Gateway:** `external-http-gateway` handles all external traffic.
- **PathPrefix Routing:** `/agentforge/*` → AgentForge landing, `/waitlist/*` → Waitlist API, `/business/*` → local business sites.
- **No Public Ingress for Internals:** Monitoring (Grafana/Prometheus), databases, and agent infrastructure are accessible only via Tailscale VPN.
- **Cross-Namespace References:** HTTPRoute resources in one namespace can reference Services in another, keeping routing centralized.
### 4. Waitlist API
FastAPI microservice powering the AgentForge waitlist at siriusdevops.com:
- **Database:** asyncpg connection pool to dedicated CNPG PostgreSQL.
- **Idempotent Signups:** `INSERT ... ON CONFLICT DO NOTHING` — duplicate emails are silently ignored, not rejected.
- **Notifications:** Fire-and-forget Telegram webhook on each new signup. No blocking I/O in the request path.
- **Security:** Rate limiting per IP, input sanitization, and CORS whitelist.
### 5. Autonomous News Quant Pipeline (`news_bot`)
An institutional-grade pipeline scraping 371 global feeds to generate actionable futures trading signals:
- **Scraper:** CronJob at `:50` pulling multi-lingual global financial data.
- **Map/Reduce Analyst:** DeepSeek-R1 with a strict 10-step think protocol extracts "Market-Moving DNA" and translates events into explicit futures targets (/ES, /CL, /NQ) with risk:reward, take profit, and stop loss levels.
- **Privacy:** All proprietary technical data stays strictly within the VPC, executing against local models to protect the trading edge.
### 6. Local Business Web Deployment Pipeline
Automated Kubernetes manifest generation for small business websites:
- **Stack:** nginx serving static content from ConfigMap, one namespace per business.
- **Routing:** HTTPRoute with cross-namespace Service references under `/business/<name>` paths.
- **Zero Cold Start:** Static sites have no database dependency — just nginx + ConfigMap, deployed via GitOps.
---
## Getting Started
### Prerequisites
- Google Cloud account with billing enabled
- GitHub account with repository access
- `gcloud` CLI authenticated
- Terraform 1.7+
### Local Development Setup
```bash ```bash
# Install tools via mise mise trust && mise install # kubectl, helm, sops, terraform, k9s, gcloud
./scripts/setup
# Or manually
mise trust && mise install
``` ```
### Infrastructure Deployment Decrypt a secret locally (needs the age key, which is not in Git):
```bash ```bash
cd modules sops -d apps/base/customer1/waitlist-api/waitlist-telegram-secret.yaml
# Initialize Terraform
terraform init
# Set required variables
export TF_VAR_github_token="your-token"
export TF_VAR_github_org="your-org"
export TF_VAR_github_repository="gcloud-lab"
# Plan and apply
terraform plan
terraform apply
```
### Accessing the Cluster
```bash
# Configure kubectl
gcloud container clusters get-credentials devops-lab-cluster \
--zone us-central1-a \
--project devops-lab-cluster
# Verify connection
kubectl get nodes
# Use k9s for interactive management
k9s
```
### Accessing Monitoring (Grafana / Prometheus)
Monitoring services are **not publicly exposed**. Access is via Tailscale VPN or port-forwarding:
```bash
# Option 1: Port-forward Grafana
kubectl port-forward svc/prometheus-community-kube-prometheus-stack-grafana \
-n monitoring 3000:3000
# Option 2: Port-forward Prometheus
kubectl port-forward svc/prometheus-community-kube-prometheus-stack-prometheus \
-n monitoring 9090:9090
```
⚠️ **Before deploying**, replace the Grafana admin password in
`infrastructure/controllers/base/monitoring/release.yaml` with a secure value,
or create a `monitoring-grafana-admin` Secret instead.
---
## Security
### Secrets Management
- **Encryption**: SOPS with Age encryption
- **Key Storage**: `age.agekey` (do not commit unencrypted)
- **Flux Integration**: Automatic decryption during deployment
### Pod Security
- Non-root containers (UID 1000)
- Filesystem group enforcement
- Privilege escalation disabled
- Resource limits enforced
### Network Security
- Cilium NetworkPolicy for pod-to-pod and namespace-to-namespace isolation
- Kubernetes Gateway API with TLS termination at the load balancer
- Internal services (monitoring, databases, agent infrastructure) accessible only via Tailscale VPN — zero public ingress
- Rate limiting on public-facing APIs (Waitlist, landing page)
### Database Security
- Managed roles with secret-based passwords per application
- Separate PostgreSQL clusters per domain (hermes-pgdb, hermes-tenant-pgdb, siriusdevops-pgdb)
- GCS backups with configurable retention policies
- HA cluster with automatic failover
### Automated Security Auditing
- **sec-ops Agent:** Continuously scans deployed infrastructure for CVEs, misconfigurations, and policy violations
- **Auto-Remediation:** Confirmed findings automatically dispatch the backend-dev agent to patch and commit
- **Audit Trail:** Every finding, fix, and deployment is tracked in Git history — full provenance from detection to resolution
---
## Cost Optimization
- **SPOT GPU Instances**: 60-90% savings on L4 and RTX 6000 workloads
- **KEDA Scale-to-Zero**: RTX 6000 deep thinker pool scales to 0 when no requests are queued
- **Resource Limits**: CPU and memory caps on every container prevent runaway costs
- **Scheduled Workloads**: CronJobs only run when needed — no idle inference pods
- **Tailscale for Internal Access**: No need for expensive internal load balancers or Cloud NAT for monitoring
---
## Container Images
```
ghcr.io/sirius0xdev/agentforge-landing:latest
ghcr.io/sirius0xdev/waitlist-api:latest
ghcr.io/sirius0xdev/newsscraper:latest
ghcr.io/sirius0xdev/summarizer:latest
ghcr.io/sirius0xdev/news-messenger:latest
docker.n8n.io/n8nio/n8n:2.1.4
ghcr.io/cloudnative-pg/postgresql:15.2
``` ```
--- ---
## Tool Reference ## Status
### Terraform Providers | | |
| --- | --- |
| Cluster | **Destroyed** (GCP project `devops-lab-cluster`, GKE `devops-lab-cluster`, `us-central1-a`) |
| This repo | Archive of what ran |
| Live infra today | Self-hosted on a Pi — see [siriusdevops.com/lab](https://siriusdevops.com/lab) |
```hcl Lance Walters — [siriusdevops.com](https://siriusdevops.com)
google = "~> 7.14" # GCP resources
helm = "~> 2.0" # Helm chart management
flux = "~> 1.7" # GitOps bootstrap
```
### Helm Charts
```yaml
cilium: 1.18.5 # CNI and service mesh
cloudnative-pg: 0.26.1 # PostgreSQL operator
vllm: 0.9.1 # High-throughput LLM serving
```
---
## License
Private repository — All rights reserved.

View file

@ -1,27 +0,0 @@
apiVersion: batch/v1
kind: CronJob
metadata:
name: analyst-gemma
namespace: customer1
spec:
schedule: "15 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: analyst
image: siriussec/summarizerlocal:3.5
imagePullPolicy: Always
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: news-app-password
key: password
envFrom:
- configMapRef:
name: gemma-config
restartPolicy: OnFailure

View file

@ -1,31 +0,0 @@
apiVersion: batch/v1
kind: CronJob
metadata:
name: news-analyst-gemini
namespace: customer1
spec:
schedule: "15 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: analyst-gemini
image: siriussec/summarizer:1.5
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: news-app-password
key: password
envFrom:
- configMapRef:
name: news-app-config
- secretRef:
name: gemini-apikey
restartPolicy: OnFailure

View file

@ -1,34 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: news-app-config
namespace: customer1
data:
DB_HOST: "siriusdevops-pgdb-rw"
DB_NAME: "news_app_db"
DB_USER: "news_app"
DB_PORT: "5432"
LLM_BASE_URL: "http://rtx6000-brain-service.customer1.svc.cluster.local:8000/v1"
MODEL_NAME: "edp1096/Huihui-Qwen3.6-27B-abliterated-FP8"
LLM_API_KEY: "sk-dummy"
SUMMARY_PROMPT: |
You are an expert news analyst working for DARPA. Your task is to deliver thorough, insightful analysis of the provided news articles.
ARTICLES:
{data}
Instructions:
- Identify the 36 most impactful events or developments (prioritize geopolitical, technological, military, economic, or security implications).
- For each impactful event: Write a detailed 46 sentence summary, including key facts, context, potential consequences, and why it matters strategically.
- For all remaining/lesser articles: Write 24 cohesive paragraphs synthesizing them into broader themes or trends (do NOT list them individually unless critical).
- Aim for a total output length of 400800 words. Be comprehensive but concise.
- Always produce substantial content — even if articles seem minor, extract value, implications, or connections.
- Use neutral, professional, analytical tone with precise language.
- Structure your response clearly with headings like:
## Most Impactful Events
### Event 1: [Brief Title]
[46 sentences...]
...
## Synthesis of Lesser Developments
[Paragraphs...]
- Output ONLY the structured report — no introductions, apologies, or meta-comments like "no major events found".

View file

@ -1,30 +0,0 @@
apiVersion: batch/v1
kind: CronJob
metadata:
name: deepseek-analyst-v1.1
namespace: customer1
spec:
schedule: "15 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: analyst
image: siriussec/summarizerlocal:3.6
imagePullPolicy: Always
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: news-app-password
key: password
envFrom:
- configMapRef:
name: deepseek-config
resources:
requests:
cpu: "1"
memory: "4Gi"
restartPolicy: OnFailure

View file

@ -1,23 +0,0 @@
apiVersion: v1
data:
GEMINI_API_KEY: ENC[AES256_GCM,data:EcyraxRLXiWWQVhJNmmUGut6AmSMA8BvFg0X/PoP2ckclcFas8pU7GfsmMXFD9arcuCJpg==,iv:52YNqcPGWQqQY4jdfc8DtP6dROTsxJqUDahXzEN3ycI=,tag:Q/9l0mDG+5sjTLKt5Jg4sw==,type:str]
kind: Secret
metadata:
creationTimestamp: null
name: gemini-apikey
namespace: customer1
sops:
age:
- recipient: age1uuxf066xuuqgvjppxfcmqkwfcufnwp3wcwnl9h20g9k4l8nkw9jsaungf7
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTcGdQb0tMdWtiTkl3aWZt
dnl4ZHo3aHZ4bGgvR1Rva0E5dlloQlRGY0d3CmMwOUJWb3NlWEtRVEk3STJ5MWVk
d21ZNzkyeHBZQTF3dWhYdUlSZEtQQkEKLS0tIEUrU3M2UGh5Y1BPTUhRQi9NK2VN
Vi83QzhjcG45U1B1WTJGcklFdkRvQjQK/jAjKf3wCZCpHp5naJoitHfN1yEEqmoN
p5AMB97oTtyVhvu3wkRrxqHB8LKNWZifaJf8g13To1OgFh9azC/Kng==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-01-16T20:31:27Z"
mac: ENC[AES256_GCM,data:7y9LrLu+bngdTuhb1ucIDnvg3jpKgXW6rXMpS6O08nxHHBKAKA6Jl80PsfOE0aQNSsIM2ERfrdGmfLkNh470oB/erGvEHxRLKcf8IStPxoPpry99XffiVo+QJrCPIzqgKy1tw37bqBNrm5LJkhUmugzf/IVt6WKiGCZhpbXjB7E=,iv:RfK084/orIGcRwXMeAip0BKXJ3eYJoRmEgLF72ER9vA=,tag:NKqZKIrt3yVkGL4ZkqcZKA==,type:str]
encrypted_regex: ^(data|stringData)$
version: 3.11.0

View file

@ -1,63 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: gemma-config
namespace: customer1
data:
DB_HOST: "siriusdevops-pgdb-rw"
DB_NAME: "news_app_db"
DB_USER: "news_app"
DB_PORT: "5432"
LLM_BASE_URL: "http://rtx6000-brain-service.customer1.svc.cluster.local:8000/v1"
MODEL_NAME: "edp1096/Huihui-Qwen3.6-27B-abliterated-FP8"
LLM_API_KEY: "sk-dummy"
MAP_PROMPT: |
You are a state of the art AI News analysis
working for a independent day trader. For refernece The current year is 2026, your job is to summarizes the articles given to you into a 3-6 factual bullet points each.
You are looking for any news that may effect the stock or futures market. Specifically the futures. These articles are from world wide sources and in many languages . Translate all to english.
Rate eash article on a scale of 1-10 depending on it's possible market effects . 1 being no effect 10 being a critical event . In each event with market relevance be sure to list the tickers or futures symbols that will be affected.
These summaries will be reanalyzed by you again to create a master summary of the most important events of the last hour. So leave any notes that may be useful.
data:
{batch_text}
SUMMARY_PROMPT: |
Role: Futures Analyst AI for funded traders.
Objective: Extract actionable "edges" and volatility signals from news batches for Equity Indices, Energies, Metals, Ag, and FX.
Task:
Filter: Ignore noise (sports, entertainment, local). Deep-analyze only top 5-10 high-signal articles.
Criteria: Scan for Economic Data (Fed/CPI), Geopolitics, Supply/Demand shifts, and Corporate news impacting futures.
Analyze: Assess Impact (H/M/L), Direction (Bullish/Bearish), Trading Edge (actionable strategy), and Confidence Score (1-10).
Output (Telegram-friendly Markdown + Emojis):
📊 Summary Dashboard
Scanned: [Total] | Relevant: [Count]
Affected: [e.g., CL, ES, NQ]
Alert: [Brief sentiment summary]
🔥 Detailed Insights (Title/Source/Time)
Summary: [1-2 sentences]
Impact: [Contract/Direction/Rationale]
Trading Edge: [Actionable entry/exit/monitoring signal]
Confidence: [1-10]
👀 Watchlist: [Low-impact/monitoring items]
If empty, output: "No significant futures-impacting news in this batch."
Constraint: Objective, data-driven, no hype. Use historical analogies to quantify moves where possible.
Data to analyze: {final_input}

View file

@ -1,5 +1,3 @@
apiVersion: batch/v1 apiVersion: batch/v1
kind: CronJob kind: CronJob
metadata: metadata:
@ -27,5 +25,5 @@ spec:
resources: resources:
requests: requests:
cpu: "1" cpu: "1"
memory: "2Gi" memory: "1Gi"
restartPolicy: OnFailure restartPolicy: OnFailure

View file

@ -1,11 +0,0 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ollama-storage
namespace: customer1
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi

View file

@ -1,9 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: trade-dashboard-config
namespace: customer1
data:
DB_HOST: "siriusdevops-pgdb-rw.customer1.svc.cluster.local"
DB_PORT: "5432"
DB_NAME: "trading_dashboard"

View file

@ -1,69 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: trade-dashboard
namespace: customer1
labels:
app: trade-dashboard
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: trade-dashboard
template:
metadata:
labels:
app: trade-dashboard
annotations:
checksum/config: trade-dashboard-config
spec:
terminationGracePeriodSeconds: 30
containers:
- name: dashboard
image: ghcr.io/sirius0xdev/trade-dashboard:latest
imagePullPolicy: Always
ports:
- containerPort: 8000
name: http
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
envFrom:
- configMapRef:
name: trade-dashboard-config
env:
- name: DB_USER
valueFrom:
secretKeyRef:
name: trading-dashboard-db-credentials
key: username
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: trading-dashboard-db-credentials
key: password
startupProbe:
httpGet:
path: /api/health
port: 8000
initialDelaySeconds: 15
periodSeconds: 5
failureThreshold: 6
livenessProbe:
httpGet:
path: /api/health
port: 8000
initialDelaySeconds: 30
periodSeconds: 30
readinessProbe:
httpGet:
path: /api/health
port: 8000
initialDelaySeconds: 5
periodSeconds: 10

View file

@ -1,8 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
- configmap.yaml
# - tsproxy.yaml

View file

@ -1,17 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: trade-dashboard
namespace: customer1
annotations:
tailscale.com/expose: "true"
tailscale.com/hostname: "trade-dashboard"
tailscale.com/tags: "tag:k8s-operator"
tailscale.com/ports: "http:80"
spec:
selector:
app: trade-dashboard
ports:
- port: 80
targetPort: 8000
name: http

View file

@ -7,6 +7,6 @@ data:
DB_HOST: "siriusdevops-pgdb-rw.customer1.svc.cluster.local" DB_HOST: "siriusdevops-pgdb-rw.customer1.svc.cluster.local"
DB_PORT: "5432" DB_PORT: "5432"
DB_NAME: "trading_data" DB_NAME: "trading_data"
DATA_SERVICE_URL: "http://trading-data-service.customer1.svc.cluster.local" DATA_SERVICE_URL: "http://trading-data-service.customer1.svc.cluster.local:8000"
EXECUTE_SERVICE_URL: "http://trading-execute-service.customer1.svc.cluster.local" EXECUTE_SERVICE_URL: "http://trading-execute-service.customer1.svc.cluster.local:8000"
NEWS_SERVICE_URL: "http://trading-news-service.customer1.svc.cluster.local" NEWS_SERVICE_URL: "http://trading-news-service.customer1.svc.cluster.local:8000"

View file

@ -115,7 +115,7 @@ spec:
ports: ports:
- port: 9092 - port: 9092
protocol: TCP protocol: TCP
# Allow inter-service egress # Allow inter-service egress (via ClusterIP services on port 80, plus direct pod ports)
- to: - to:
- podSelector: - podSelector:
matchExpressions: matchExpressions:
@ -127,6 +127,8 @@ spec:
- trading-news-service - trading-news-service
- trading-dashboard - trading-dashboard
ports: ports:
- port: 80
protocol: TCP
- port: 8000 - port: 8000
protocol: TCP protocol: TCP
- port: 8001 - port: 8001

View file

@ -1 +0,0 @@
__pycache__/

View file

@ -1,18 +0,0 @@
FROM python:3.13-slim AS base
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends \
gcc libpq-dev \
&& rm -rf /var/lib/apt/lists/*
COPY app/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app/ ./app/
COPY alembic.ini ./alembic.ini
COPY alembic/ ./alembic/
EXPOSE 8000
ENV PYTHONPATH=/app
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

View file

@ -1,37 +0,0 @@
[alembic]
script_location = alembic
# Override at runtime via DATABASE_URL environment variable (set in ConfigMap/Deployment)
sqlalchemy.url = driver://user:pass@localhost/dbname
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S

View file

@ -1,55 +0,0 @@
import sys
from logging.config import fileConfig
from pathlib import Path
from sqlalchemy import pool
from sqlalchemy.ext.asyncio import async_engine_from_config
from alembic import context
# Add app directory to path so we can import models/database
sys.path.insert(0, str(Path(__file__).parent.parent / "app"))
from database import metadata, DATABASE_URL
config = context.config
if config.config_file_name is not None:
fileConfig(config.config_file_name)
target_metadata = metadata
def run_migrations_offline() -> None:
"""Run migrations in 'offline' mode."""
url = config.get_main_option("sqlalchemy.url")
context.configure(url=url, target_metadata=target_metadata, literal_binds=True)
with context.begin_transaction():
context.run_migrations()
def do_run_migrations(connection):
context.configure(connection=connection, target_metadata=target_metadata)
with context.begin_transaction():
context.run_migrations()
async def run_async_migrations():
connectable = async_engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
async with connectable.connect() as connection:
await connection.run_sync(do_run_migrations)
await connectable.dispose()
def run_migrations_online() -> None:
"""Run migrations in 'online' mode."""
import asyncio
asyncio.run(run_async_migrations())
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()

View file

@ -1,28 +0,0 @@
<%%doc>Template for rendering a Multiple Migration Revision Identifier.</%%doc>
<%%-
from alembic import context
context.configure()
-%>
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma_n, trim}
Create Date: ${create_date}
"""
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
# revision identifiers, used by Alembic.
revision = ${repr(up_revision)}
down_revision = ${repr(down_revision)}
branch_labels = ${repr(branch_labels)}
depends_on = ${repr(depends_on)}
def upgrade() -> None:
${upgrades if upgrades else "pass"}
def downgrade() -> None:
${downgrades if downgrades else "pass"}

View file

@ -1,149 +0,0 @@
"""initial schema
Revision ID: 001_initial
Revises:
Create Date: 2026-05-18
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects.postgresql import UUID, TSVECTOR, ENUM
# revision identifiers, used by Alembic.
revision = '001_initial'
down_revision = None
branch_labels = None
depends_on = None
def upgrade() -> None:
# Enums
op.execute("CREATE TYPE feed_source_type AS ENUM ('rss', 'gdel-t2', 'social', 'earthquake', 'disaster', 'weather', 'fire', 'satellite')")
op.execute("CREATE TYPE event_source_type AS ENUM ('rss', 'gdel-t2', 'social', 'earthquake', 'disaster', 'weather', 'fire', 'satellite')")
op.execute("CREATE TYPE sentiment_label AS ENUM ('positive', 'neutral', 'negative')")
op.execute("CREATE TYPE entity_type AS ENUM ('person', 'organization', 'location', 'topic', 'asset')")
op.execute("CREATE TYPE alert_type AS ENUM ('entity_mention', 'sentiment_shift', 'geo_proximity', 'keyword_match', 'threshold', 'anomaly')")
op.execute("CREATE TYPE alert_severity AS ENUM ('low', 'medium', 'high', 'critical')")
# Extensions
op.execute("CREATE EXTENSION IF NOT EXISTS postgis")
op.execute("CREATE EXTENSION IF NOT EXISTS timescaledb")
# feed_sources
op.create_table(
'feed_sources',
sa.Column('id', UUID(as_uuid=True), primary_key=True),
sa.Column('name', sa.String(256), nullable=False),
sa.Column('source_type', sa.Enum('rss', 'gdel-t2', 'social', 'earthquake', 'disaster', 'weather', 'fire', 'satellite', name='feed_source_type'), nullable=False),
sa.Column('url', sa.Text()),
sa.Column('config', sa.JSON()),
sa.Column('enabled', sa.Integer, server_default='1', nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.func.now()),
)
# events (will become hypertable)
op.create_table(
'events',
sa.Column('id', UUID(as_uuid=True), primary_key=True),
sa.Column('source_type', sa.Enum('rss', 'gdel-t2', 'social', 'earthquake', 'disaster', 'weather', 'fire', 'satellite', name='event_source_type'), nullable=False, index=True),
sa.Column('source_id', UUID(as_uuid=True)),
sa.Column('title', sa.Text()),
sa.Column('body', sa.Text()),
sa.Column('url', sa.Text()),
sa.Column('sentiment_score', sa.Float()),
sa.Column('sentiment_label', sa.Enum('positive', 'neutral', 'negative', name='sentiment_label')),
sa.Column('location_lat', sa.Float()),
sa.Column('location_lon', sa.Float()),
sa.Column('location_name', sa.String(512)),
sa.Column('entities', sa.JSON()),
sa.Column('tags', sa.JSON()),
sa.Column('raw', sa.JSON()),
sa.Column('ingested_at', sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column('source_timestamp', sa.DateTime(timezone=True), nullable=False),
sa.Column('search_vector', TSVECTOR),
)
# Convert events to TimescaleDB hypertable
op.execute("SELECT create_hypertable('events', 'ingested_at', if_not_exists => TRUE)")
# GIN index for full-text search
op.create_index('ix_events_search_vector', 'events', ['search_vector'], postgresql_using='gin')
# Spatial index
op.create_index('ix_events_location', 'events', ['location_lat', 'location_lon'])
# entities
op.create_table(
'entities',
sa.Column('id', UUID(as_uuid=True), primary_key=True),
sa.Column('name', sa.String(512), nullable=False, index=True),
sa.Column('entity_type', sa.Enum('person', 'organization', 'location', 'topic', 'asset', name='entity_type'), nullable=False),
sa.Column('aliases', sa.JSON()),
sa.Column('description', sa.Text()),
sa.Column('metadata', sa.JSON()),
sa.Column('location_lat', sa.Float()),
sa.Column('location_lon', sa.Float()),
sa.Column('event_count', sa.Integer, server_default='0'),
sa.Column('first_seen', sa.DateTime(timezone=True), server_default=sa.func.now()),
sa.Column('last_seen', sa.DateTime(timezone=True), server_default=sa.func.now()),
)
# entity_events
op.create_table(
'entity_events',
sa.Column('entity_id', UUID(as_uuid=True), primary_key=True),
sa.Column('event_id', UUID(as_uuid=True), primary_key=True),
sa.Column('relevance_score', sa.Float()),
sa.Column('linked_at', sa.DateTime(timezone=True), server_default=sa.func.now()),
)
# alerts
op.create_table(
'alerts',
sa.Column('id', UUID(as_uuid=True), primary_key=True),
sa.Column('alert_type', sa.Enum('entity_mention', 'sentiment_shift', 'geo_proximity', 'keyword_match', 'threshold', 'anomaly', name='alert_type'), nullable=False),
sa.Column('entity_id', UUID(as_uuid=True)),
sa.Column('event_id', UUID(as_uuid=True)),
sa.Column('severity', sa.Enum('low', 'medium', 'high', 'critical', name='alert_severity'), nullable=False),
sa.Column('title', sa.Text(), nullable=False),
sa.Column('message', sa.Text()),
sa.Column('context', sa.JSON()),
sa.Column('acknowledged', sa.Integer, server_default='0'),
sa.Column('acknowledged_by', sa.String(256)),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column('resolved_at', sa.DateTime(timezone=True)),
)
op.create_index('ix_alerts_severity_created', 'alerts', ['severity', 'created_at'])
op.create_index('ix_alerts_entity', 'alerts', ['entity_id'])
# documents
op.create_table(
'documents',
sa.Column('id', UUID(as_uuid=True), primary_key=True),
sa.Column('bucket', sa.String(256), nullable=False),
sa.Column('object_key', sa.String(1024), nullable=False),
sa.Column('content_type', sa.String(256)),
sa.Column('size_bytes', sa.Integer()),
sa.Column('description', sa.Text()),
sa.Column('tags', sa.JSON()),
sa.Column('event_id', UUID(as_uuid=True)),
sa.Column('uploaded_at', sa.DateTime(timezone=True), server_default=sa.func.now()),
)
def downgrade() -> None:
op.drop_table('documents')
op.drop_table('alerts')
op.drop_table('entity_events')
op.drop_table('entities')
op.execute("SELECT drop_hypertable('events', cascade => TRUE)")
op.drop_table('events')
op.drop_table('feed_sources')
# Drop enums
op.execute("DROP TYPE IF EXISTS feed_source_type")
op.execute("DROP TYPE IF EXISTS event_source_type")
op.execute("DROP TYPE IF EXISTS sentiment_label")
op.execute("DROP TYPE IF EXISTS entity_type")
op.execute("DROP TYPE IF EXISTS alert_type")
op.execute("DROP TYPE IF EXISTS alert_severity")

View file

@ -1,28 +0,0 @@
import os
from sqlalchemy import MetaData, event, text
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
DB_USER = os.getenv("DB_USER", "osint")
DB_PASS = os.getenv("DB_PASSWORD", "")
DB_HOST = os.getenv("DB_HOST", "osint-pgdb-rw.customer1.svc.cluster.local")
DB_PORT = os.getenv("DB_PORT", "5432")
DB_NAME = os.getenv("DB_NAME", "osint_data")
DATABASE_URL = f"postgresql+asyncpg://{DB_USER}:{DB_PASS}@{DB_HOST}:{DB_PORT}/{DB_NAME}"
engine = create_async_engine(
DATABASE_URL, echo=False, pool_size=5, max_overflow=10, pool_recycle=300
)
async_session = async_sessionmaker(
engine, class_=AsyncSession, expire_on_commit=False
)
metadata = MetaData()
async def init_extensions():
"""Initialize PostGIS and TimescaleDB extensions on first connection."""
async with engine.connect() as conn:
await conn.execute(text("CREATE EXTENSION IF NOT EXISTS postgis"))
await conn.execute(text("CREATE EXTENSION IF NOT EXISTS timescaledb"))
await conn.commit()

View file

@ -1,46 +0,0 @@
"""CronJob entry point for scheduled ingestion."""
import asyncio
import os
import sys
from pathlib import Path
# Add app dir to path
sys.path.insert(0, str(Path(__file__).parent))
from sources import ingest_rss_feed, ingest_gdelt, ingest_earthquakes
from ingestor import fetch_and_process
INGESTOR_TYPE = os.getenv("INGESTOR_TYPE", "rss")
RSS_URL = os.getenv("RSS_URL", "")
async def main():
print(f"Starting ingester: {INGESTOR_TYPE}")
if INGESTOR_TYPE == "rss":
if not RSS_URL:
print("No RSS_URL set, skipping")
return
count = await ingest_rss_feed(RSS_URL)
print(f"RSS: ingested {count} items")
elif INGESTOR_TYPE == "gdelt":
count = await ingest_gdelt(max_articles=50)
print(f"GDELT: ingested {count} articles")
elif INGESTOR_TYPE == "earthquake":
count = await ingest_earthquakes()
print(f"Earthquakes: ingested {count} events")
elif INGESTOR_TYPE == "nats":
count = await fetch_and_process(batch_size=500)
print(f"NATS: processed {count} messages")
else:
print(f"Unknown ingestor type: {INGESTOR_TYPE}")
sys.exit(1)
if __name__ == "__main__":
asyncio.run(main())

View file

@ -1,107 +0,0 @@
"""NATS JetStream consumer — ingests OSINT events from NATS streams."""
from __future__ import annotations
import json
import logging
from datetime import datetime, timezone
import nats
from nats.errors import TimeoutError
from database import async_session
from models import events as events_table
logger = logging.getLogger("osint.ingestor")
# NATS connection settings
NATS_URLS = "nats://osint-nats.customer1.svc.cluster.local:4222"
NATS_STREAM = "events"
NATS_DURABLE = "osint-ingestor"
# Redis cache settings
REDIS_URL = "redis://osint-redis-sentinel.customer1.svc.cluster.local:26379/0"
async def ingest_event(msg: dict):
"""Ingest a single event from NATS into PostgreSQL."""
event_row = {
"source_type": msg.get("source_type", "rss"),
"source_id": msg.get("source_id"),
"title": msg.get("title"),
"body": msg.get("body"),
"url": msg.get("url"),
"sentiment_score": msg.get("sentiment_score"),
"sentiment_label": msg.get("sentiment_label"),
"location_lat": msg.get("location_lat"),
"location_lon": msg.get("location_lon"),
"location_name": msg.get("location_name"),
"entities": msg.get("entities", []),
"tags": msg.get("tags", []),
"raw": msg.get("raw"),
"source_timestamp": msg.get("source_timestamp", datetime.now(timezone.utc).isoformat()),
}
# Parse timestamp if string
if isinstance(event_row["source_timestamp"], str):
event_row["source_timestamp"] = datetime.fromisoformat(event_row["source_timestamp"])
async with async_session() as session:
result = await session.execute(events_table.insert().values(**event_row))
await session.commit()
event_id = result.inserted_primary_key[0] # type: ignore[union-attr]
logger.info("Ingested event %s from source %s", event_id, msg.get("source_type"))
return event_id
async def start_nats_consumer():
"""Start NATS JetStream consumer for OSINT events."""
nc = await nats.connect(NATS_URLS)
js = nc.jetstream()
# Create stream if not exists
try:
await js.add_stream(
name=NATS_STREAM,
subjects=[
"events.gdelt", "events.rss", "events.social",
"events.earthquake", "events.disaster", "events.weather",
"events.fire", "events.satellite", "events.new", "events.alert",
],
retention=nats.js.api.RetentionPolicy.INTERESTS,
max_msgs=1_000_000,
)
logger.info("Created NATS stream %s", NATS_STREAM)
except Exception:
logger.debug("Stream %s already exists", NATS_STREAM)
# Create durable consumer
sub = await js.pull_subscribe(
subject="events.>",
durable_name=NATS_DURABLE,
)
logger.info("NATS consumer started, durable=%s", NATS_DURABLE)
return nc, sub
async def fetch_and_process(batch_size: int = 100):
"""Fetch a batch of messages and process them."""
nc, sub = await start_nats_consumer()
js = nc.jetstream()
msgs = await sub.fetch(batch_size, timeout=5)
processed = 0
for msg in msgs:
try:
data = json.loads(msg.data)
await ingest_event(data)
await msg.ack()
processed += 1
except Exception:
logger.error("Failed to process message: %s", msg.data, exc_info=True)
await nc.close()
logger.info("Processed %d messages in batch", processed)
return processed

View file

@ -1,603 +0,0 @@
"""OSINT Dashboard — FastAPI backend.
Real-time geospatial OSINT dashboard API:
- Event ingestion via NATS JetStream consumers
- Full-text search across events (PostgreSQL tsvector)
- Entity tracking and relationship mapping
- Alert management
- Document storage (MinIO-backed)
- Sentiment aggregation and timeline analytics
"""
from __future__ import annotations
import json
import logging
from datetime import datetime, timedelta, timezone
from decimal import Decimal
from pathlib import Path
from uuid import UUID
import structlog
from fastapi import FastAPI, HTTPException, Query
from fastapi.responses import FileResponse, HTMLResponse
from sqlalchemy import and_, func, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from database import async_session, init_extensions
from models import (
alerts, documents, entities, entity_events, events, feed_sources
)
from schemas import (
AlertCreate, AlertOut, AlertSeverity, AlertType, AlertUpdate,
DashboardSummary, EntityCreate, EntityKind, EntityOut,
EventCreate, EventOut,
FeedSourceCreate, FeedSourceOut,
SearchResult, SentimentSummary, SourceType,
SearchQuery, TimelinePoint,
)
from ingestor import ingest_event, fetch_and_process
from sources import ingest_rss_feed, ingest_gdelt, ingest_earthquakes, ingest_social_signals
logging.basicConfig(level=logging.INFO)
logger = structlog.get_logger("osint.dashboard")
app = FastAPI(
title="OSINT Dashboard",
description="Real-time geospatial OSINT intelligence dashboard",
version="0.1.0",
)
STATIC_DIR = Path(__file__).parent / "static"
# ── Helpers ───────────────────────────────────────────────────────────────
def event_to_out(row: dict) -> EventOut:
"""Convert DB row dict to EventOut schema."""
return EventOut(
id=row["id"],
source_type=row["source_type"],
source_id=row["source_id"],
title=row["title"],
body=row["body"],
url=row["url"],
sentiment_score=row["sentiment_score"],
sentiment_label=row["sentiment_label"],
location_lat=row["location_lat"],
location_lon=row["location_lon"],
location_name=row["location_name"],
entities=row["entities"],
tags=row["tags"],
ingested_at=row["ingested_at"],
source_timestamp=row["source_timestamp"],
)
def entity_to_out(row: dict) -> EntityOut:
"""Convert DB row dict to EntityOut schema."""
return EntityOut(
id=row["id"],
name=row["name"],
entity_type=row["entity_type"],
aliases=row["aliases"],
description=row["description"],
metadata=row["metadata"],
location_lat=row["location_lat"],
location_lon=row["location_lon"],
event_count=row["event_count"],
first_seen=row["first_seen"],
last_seen=row["last_seen"],
)
def alert_to_out(row: dict) -> AlertOut:
"""Convert DB row dict to AlertOut schema."""
return AlertOut(
id=row["id"],
alert_type=row["alert_type"],
entity_id=row["entity_id"],
event_id=row["event_id"],
severity=row["severity"],
title=row["title"],
message=row["message"],
context=row["context"],
acknowledged=bool(row["acknowledged"]),
acknowledged_by=row["acknowledged_by"],
created_at=row["created_at"],
resolved_at=row["resolved_at"],
)
# ── Health ────────────────────────────────────────────────────────────────
@app.get("/api/health")
async def health():
"""Health check with database connectivity."""
async with async_session() as session:
result = await session.execute(select(func.now()))
db_time = result.scalar()
return {"status": "ok", "db_time": db_time.isoformat() if db_time else None}
# ── Startup ───────────────────────────────────────────────────────────────
@app.on_event("startup")
async def startup():
"""Initialize extensions and run migrations."""
await init_extensions()
from alembic import command
from alembic.config import Config
alembic_cfg = Config(str(Path(__file__).parent.parent / "alembic.ini"))
command.upgrade(alembic_cfg, "head")
# ── Feed Sources ──────────────────────────────────────────────────────────
@app.get("/api/sources", response_model=list[FeedSourceOut])
async def list_sources(enabled_only: bool = Query(True)):
"""List all configured feed sources."""
async with async_session() as session:
stmt = select(feed_sources).order_by(feed_sources.c.name)
if enabled_only:
stmt = stmt.where(feed_sources.c.enabled == 1)
rows = (await session.execute(stmt)).mappings().all()
return [FeedSourceOut(
id=r["id"], name=r["name"], source_type=r["source_type"],
url=r["url"], config=r["config"], enabled=bool(r["enabled"]),
created_at=r["created_at"],
) for r in rows]
@app.post("/api/sources", status_code=201)
async def create_source(payload: FeedSourceCreate):
"""Add a new feed source."""
async with async_session() as session:
values = payload.model_dump()
result = await session.execute(feed_sources.insert().values(**values))
await session.commit()
pk = result.inserted_primary_key[0] # type: ignore
return {"id": str(pk)}
@app.patch("/api/sources/{source_id}")
async def update_source(source_id: UUID, payload: dict):
"""Update a feed source (e.g., toggle enabled)."""
async with async_session() as session:
row = (await session.execute(
select(feed_sources).where(feed_sources.c.id == source_id)
)).mappings().one_or_none()
if not row:
raise HTTPException(404, "Source not found")
await session.execute(
feed_sources.update()
.where(feed_sources.c.id == source_id)
.values(**payload)
)
await session.commit()
return {"ok": True}
# ── Events ────────────────────────────────────────────────────────────────
@app.get("/api/events", response_model=list[EventOut])
async def list_events(
source_type: SourceType | None = Query(None),
limit: int = Query(50, ge=1, le=500),
offset: int = Query(0, ge=0),
):
"""List recent ingested events."""
async with async_session() as session:
stmt = select(events).order_by(events.c.ingested_at.desc())
if source_type:
stmt = stmt.where(events.c.source_type == source_type.value)
stmt = stmt.limit(limit).offset(offset)
rows = (await session.execute(stmt)).mappings().all()
return [event_to_out(r) for r in rows]
@app.get("/api/events/{event_id}", response_model=EventOut)
async def get_event(event_id: UUID):
"""Get a single event by ID."""
async with async_session() as session:
row = (await session.execute(
select(events).where(events.c.id == event_id)
)).mappings().one_or_none()
if not row:
raise HTTPException(404, "Event not found")
return event_to_out(row)
@app.post("/api/events", status_code=201)
async def create_event(payload: EventCreate):
"""Manually ingest an event (bypasses NATS)."""
values = payload.model_dump(exclude_unset=True)
if not values.get("source_timestamp"):
values["source_timestamp"] = datetime.now(timezone.utc)
event_id = await ingest_event(values)
return {"id": str(event_id)}
# ── Search ────────────────────────────────────────────────────────────────
@app.post("/api/search", response_model=SearchResult)
async def search_events(query: SearchQuery):
"""Full-text search across events with optional filters."""
async with async_session() as session:
# Build query with tsvector full-text search (parameterized to avoid SQL injection)
tsquery_param = text("plainto_tsquery('english', :q)")
base_stmt = select(
events,
func.count().over().label("total")
).where(
events.c.search_vector.op("@@")(tsquery_param)
)
# Apply filters
if query.source_type:
base_stmt = base_stmt.where(events.c.source_type == query.source_type.value)
if query.entity_id:
base_stmt = base_stmt.join(
entity_events, entity_events.c.event_id == events.c.id
).where(entity_events.c.entity_id == query.entity_id)
if query.sentiment:
base_stmt = base_stmt.where(events.c.sentiment_label == query.sentiment.value)
if query.min_date:
base_stmt = base_stmt.where(events.c.source_timestamp >= query.min_date)
if query.max_date:
base_stmt = base_stmt.where(events.c.source_timestamp <= query.max_date)
if query.min_lat is not None and query.max_lat is not None:
base_stmt = base_stmt.where(
and_(
events.c.location_lat >= query.min_lat,
events.c.location_lat <= query.max_lat,
)
)
if query.min_lon is not None and query.max_lon is not None:
base_stmt = base_stmt.where(
and_(
events.c.location_lon >= query.min_lon,
events.c.location_lon <= query.max_lon,
)
)
base_stmt = base_stmt.order_by(events.c.ingested_at.desc())
base_stmt = base_stmt.limit(query.limit).offset(query.offset)
result = (await session.execute(base_stmt, {"q": query.q})).mappings().all()
if result:
total = result[0]["total"]
else:
total = 0
evts = [event_to_out(r) for r in result]
return SearchResult(
events=evts,
total=total,
has_more=query.offset + len(evts) < total,
)
# ── Entities ──────────────────────────────────────────────────────────────
@app.get("/api/entities", response_model=list[EntityOut])
async def list_entities(
entity_type: EntityKind | None = Query(None),
limit: int = Query(50, ge=1, le=500),
):
"""List tracked entities."""
async with async_session() as session:
stmt = select(entities).order_by(entities.c.event_count.desc())
if entity_type:
stmt = stmt.where(entities.c.entity_type == entity_type.value)
stmt = stmt.limit(limit)
rows = (await session.execute(stmt)).mappings().all()
return [entity_to_out(r) for r in rows]
@app.get("/api/entities/{entity_id}", response_model=EntityOut)
async def get_entity(entity_id: UUID):
"""Get entity details with recent events."""
async with async_session() as session:
row = (await session.execute(
select(entities).where(entities.c.id == entity_id)
)).mappings().one_or_none()
if not row:
raise HTTPException(404, "Entity not found")
return entity_to_out(row)
@app.post("/api/entities", status_code=201)
async def create_entity(payload: EntityCreate):
"""Create or update a tracked entity."""
async with async_session() as session:
# Check if entity already exists by name
existing = (await session.execute(
select(entities).where(entities.c.name == payload.name)
)).mappings().one_or_none()
if existing:
# Update
updates = payload.model_dump(exclude_unset=True)
updates["last_seen"] = datetime.now(timezone.utc)
await session.execute(
entities.update()
.where(entities.c.id == existing["id"])
.values(**updates)
)
await session.commit()
return {"id": str(existing["id"]), "created": False}
# Create
values = payload.model_dump()
result = await session.execute(entities.insert().values(**values))
await session.commit()
pk = result.inserted_primary_key[0] # type: ignore
return {"id": str(pk), "created": True}
@app.get("/api/entities/{entity_id}/events", response_model=list[EventOut])
async def get_entity_events(
entity_id: UUID,
limit: int = Query(50, ge=1, le=500),
):
"""Get events linked to a specific entity."""
async with async_session() as session:
stmt = (
select(events)
.join(entity_events, entity_events.c.event_id == events.c.id)
.where(entity_events.c.entity_id == entity_id)
.order_by(events.c.source_timestamp.desc())
.limit(limit)
)
rows = (await session.execute(stmt)).mappings().all()
return [event_to_out(r) for r in rows]
# ── Alerts ────────────────────────────────────────────────────────────────
@app.get("/api/alerts", response_model=list[AlertOut])
async def list_alerts(
severity: AlertSeverity | None = Query(None),
acknowledged: bool | None = Query(None),
entity_id: UUID | None = Query(None),
limit: int = Query(50, ge=1, le=500),
):
"""List alerts with optional filters."""
async with async_session() as session:
stmt = select(alerts).order_by(
alerts.c.severity.desc(), alerts.c.created_at.desc()
)
if severity:
stmt = stmt.where(alerts.c.severity == severity.value)
if acknowledged is not None:
stmt = stmt.where(alerts.c.acknowledged == int(acknowledged))
if entity_id:
stmt = stmt.where(alerts.c.entity_id == entity_id)
stmt = stmt.limit(limit)
rows = (await session.execute(stmt)).mappings().all()
return [alert_to_out(r) for r in rows]
@app.post("/api/alerts", status_code=201)
async def create_alert(payload: AlertCreate):
"""Create a new alert."""
async with async_session() as session:
values = payload.model_dump()
result = await session.execute(alerts.insert().values(**values))
await session.commit()
pk = result.inserted_primary_key[0] # type: ignore
return {"id": str(pk)}
@app.patch("/api/alerts/{alert_id}")
async def update_alert(alert_id: UUID, payload: AlertUpdate):
"""Update alert (acknowledge, resolve)."""
async with async_session() as session:
row = (await session.execute(
select(alerts).where(alerts.c.id == alert_id)
)).mappings().one_or_none()
if not row:
raise HTTPException(404, "Alert not found")
updates = payload.model_dump(exclude_unset=True)
if "acknowledged" in updates:
updates["acknowledged"] = int(updates["acknowledged"])
await session.execute(
alerts.update().where(alerts.c.id == alert_id).values(**updates)
)
await session.commit()
return {"ok": True}
# ── Documents ─────────────────────────────────────────────────────────────
@app.get("/api/documents", response_model=dict)
async def list_documents(
limit: int = Query(50, ge=1, le=500),
offset: int = Query(0, ge=0),
):
"""List documents indexed in MinIO."""
async with async_session() as session:
stmt = select(documents).order_by(documents.c.uploaded_at.desc()).limit(limit).offset(offset)
rows = (await session.execute(stmt)).mappings().all()
return {
"documents": [{
"id": str(r["id"]), "bucket": r["bucket"], "object_key": r["object_key"],
"content_type": r["content_type"], "size_bytes": r["size_bytes"],
"description": r["description"], "tags": r["tags"],
"event_id": str(r["event_id"]) if r["event_id"] else None,
"uploaded_at": r["uploaded_at"].isoformat() if r["uploaded_at"] else None,
} for r in rows],
}
# ── Ingestion Triggers ───────────────────────────────────────────────────
@app.post("/api/ingest/rss")
async def trigger_rss_ingest(feed_url: str, source_id: str | None = None):
"""Trigger RSS feed ingestion."""
count = await ingest_rss_feed(feed_url, source_id)
return {"status": "ok", "items_ingested": count}
@app.post("/api/ingest/gdelt")
async def trigger_gdelt_ingest(query: str = "", max_articles: int = 50):
"""Trigger GDELT data ingestion."""
count = await ingest_gdelt(query, max_articles)
return {"status": "ok", "articles_ingested": count}
@app.post("/api/ingest/earthquakes")
async def trigger_earthquake_ingest():
"""Trigger USGS earthquake ingestion."""
count = await ingest_earthquakes()
return {"status": "ok", "events_ingested": count}
@app.post("/api/ingest/social")
async def trigger_social_ingest(query: str = "", max_items: int = 50):
"""Trigger social signals ingestion."""
count = await ingest_social_signals(query, max_items)
return {"status": "ok", "signals_ingested": count}
@app.post("/api/ingest/process")
async def trigger_nats_processing(batch_size: int = 100):
"""Process pending NATS JetStream messages."""
count = await fetch_and_process(batch_size)
return {"status": "ok", "processed": count}
# ── Analytics / Aggregation ───────────────────────────────────────────────
@app.get("/api/analytics/summary", response_model=DashboardSummary)
async def get_dashboard_summary():
"""Dashboard overview: event counts, sentiment, top entities, alerts."""
async with async_session() as session:
now = datetime.now(timezone.utc)
yesterday = now - timedelta(hours=24)
# Total events
total = (await session.execute(
select(func.count()).select_from(events)
)).scalar() or 0
# Events in last 24h
events_24h = (await session.execute(
select(func.count()).where(events.c.ingested_at >= yesterday)
)).scalar() or 0
# Active sources
active = (await session.execute(
select(func.count()).where(feed_sources.c.enabled == 1)
)).scalar() or 0
# Open alerts
open_alerts = (await session.execute(
select(func.count()).where(alerts.c.acknowledged == 0)
)).scalar() or 0
# Tracked entities
ent_count = (await session.execute(
select(func.count()).select_from(entities)
)).scalar() or 0
# Sentiment breakdown (last 24h)
def sentiment_query():
return select(
func.count().where(events.c.sentiment_label == "positive").label("pos"),
func.count().where(events.c.sentiment_label == "neutral").label("neu"),
func.count().where(events.c.sentiment_label == "negative").label("neg"),
func.avg(events.c.sentiment_score).label("avg"),
).where(events.c.ingested_at >= yesterday)
sent_row = (await session.execute(sentiment_query())).mappings().one()
sentiment = SentimentSummary(
period="24h",
positive_count=sent_row["pos"] or 0,
neutral_count=sent_row["neu"] or 0,
negative_count=sent_row["neg"] or 0,
avg_score=float(sent_row["avg"] or 0),
)
# Top entities by event count
top_ent = (await session.execute(
select(entities).order_by(entities.c.event_count.desc()).limit(10)
)).mappings().all()
return DashboardSummary(
total_events=total,
events_last_24h=events_24h,
active_sources=active,
open_alerts=open_alerts,
tracked_entities=ent_count,
sentiment=sentiment,
top_entities=[entity_to_out(r) for r in top_ent],
)
@app.get("/api/analytics/timeline")
async def get_timeline(
hours: int = Query(24, ge=1, le=168),
bucket_hours: int = Query(1, ge=1, le=24),
):
"""Event timeline: counts and avg sentiment per time bucket."""
async with async_session() as session:
cutoff = datetime.now(timezone.utc) - timedelta(hours=hours)
# Use date_trunc for bucketing
buckets = await session.execute(text(f"""
SELECT
date_trunc('hour', source_timestamp) AS ts,
COUNT(*) AS event_count,
COALESCE(AVG(sentiment_score), 0) AS avg_sentiment
FROM events
WHERE source_timestamp >= :cutoff
GROUP BY ts
ORDER BY ts
"""), {"cutoff": cutoff})
rows = buckets.mappings().all()
return [TimelinePoint(timestamp=r["ts"], event_count=r["event_count"],
avg_sentiment=float(r["avg_sentiment"])) for r in rows]
@app.get("/api/analytics/sentiment/by-source")
async def sentiment_by_source(hours: int = 24):
"""Sentiment breakdown grouped by source type."""
async with async_session() as session:
cutoff = datetime.now(timezone.utc) - timedelta(hours=hours)
result = await session.execute(text(f"""
SELECT
source_type,
COUNT(*) AS total,
COUNT(*) FILTER (WHERE sentiment_label = 'positive') AS positive,
COUNT(*) FILTER (WHERE sentiment_label = 'neutral') AS neutral,
COUNT(*) FILTER (WHERE sentiment_label = 'negative') AS negative,
COALESCE(AVG(sentiment_score), 0) AS avg_score
FROM events
WHERE ingested_at >= :cutoff
GROUP BY source_type
ORDER BY total DESC
"""), {"cutoff": cutoff})
rows = result.mappings().all()
return [{
"source_type": r["source_type"],
"total": r["total"],
"positive": r["positive"],
"neutral": r["neutral"],
"negative": r["negative"],
"avg_score": float(r["avg_score"]),
} for r in rows]
# ── Frontend ──────────────────────────────────────────────────────────────
@app.get("/", response_class=HTMLResponse)
async def index():
return FileResponse(str(STATIC_DIR / "index.html"))
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)

View file

@ -1,147 +0,0 @@
"""OSINT Dashboard — SQLAlchemy models (async, declarative)."""
from datetime import datetime, timezone
from uuid import uuid4
from sqlalchemy import (
Column, Enum, Float, Index, Integer, String, Text,
DateTime, JSON, func, Table,
)
from sqlalchemy.dialects.postgresql import UUID, TSVECTOR
from database import metadata
# ── Feed Sources ──────────────────────────────────────────────────────────
feed_sources = Table(
"feed_sources",
metadata,
Column("id", UUID(as_uuid=True), primary_key=True, default=uuid4),
Column("name", String(256), nullable=False),
Column("source_type", Enum(
"rss", "gdel-t2", "social", "earthquake", "disaster",
"weather", "fire", "satellite", name="feed_source_type"
), nullable=False),
Column("url", Text),
Column("config", JSON),
Column("enabled", Integer, server_default="1", nullable=False),
Column("created_at", DateTime(timezone=True), server_default=func.now(), nullable=False),
Column("updated_at", DateTime(timezone=True), server_default=func.now(), onupdate=func.now()),
)
# ── Events (hypertable via TimescaleDB) ──────────────────────────────────
events = Table(
"events",
metadata,
Column("id", UUID(as_uuid=True), primary_key=True, default=uuid4),
Column("source_type", Enum(
"rss", "gdel-t2", "social", "earthquake", "disaster",
"weather", "fire", "satellite", name="event_source_type"
), nullable=False, index=True),
Column("source_id", UUID(as_uuid=True)),
Column("title", Text),
Column("body", Text),
Column("url", Text),
Column("sentiment_score", Float),
Column("sentiment_label", Enum("positive", "neutral", "negative", name="sentiment_label")),
Column("location_lat", Float),
Column("location_lon", Float),
Column("location_name", String(512)),
Column("entities", JSON),
Column("tags", JSON),
Column("raw", JSON),
Column("ingested_at", DateTime(timezone=True), server_default=func.now(), nullable=False),
Column("source_timestamp", DateTime(timezone=True), nullable=False),
# Full-text search vector
Column(
"search_vector",
TSVECTOR,
nullable=True,
),
)
# GIN index for full-text search
Index("ix_events_search_vector", events.c.search_vector, postgresql_using="gin")
# Spatial index on location
Index("ix_events_location", events.c.location_lat, events.c.location_lon)
# ── Entities (people, organizations, locations of interest) ──────────────
entities = Table(
"entities",
metadata,
Column("id", UUID(as_uuid=True), primary_key=True, default=uuid4),
Column("name", String(512), nullable=False, index=True),
Column("entity_type", Enum(
"person", "organization", "location", "topic", "asset",
name="entity_type"
), nullable=False),
Column("aliases", JSON),
Column("description", Text),
Column("metadata", JSON),
Column("location_lat", Float),
Column("location_lon", Float),
Column("event_count", Integer, server_default="0"),
Column("first_seen", DateTime(timezone=True), server_default=func.now()),
Column("last_seen", DateTime(timezone=True), server_default=func.now()),
)
# ── Entity-Event Link ────────────────────────────────────────────────────
entity_events = Table(
"entity_events",
metadata,
Column("entity_id", UUID(as_uuid=True), primary_key=True),
Column("event_id", UUID(as_uuid=True), primary_key=True),
Column("relevance_score", Float),
Column("linked_at", DateTime(timezone=True), server_default=func.now()),
)
# ── Alerts ───────────────────────────────────────────────────────────────
alerts = Table(
"alerts",
metadata,
Column("id", UUID(as_uuid=True), primary_key=True, default=uuid4),
Column("alert_type", Enum(
"entity_mention", "sentiment_shift", "geo_proximity",
"keyword_match", "threshold", "anomaly",
name="alert_type"
), nullable=False),
Column("entity_id", UUID(as_uuid=True)),
Column("event_id", UUID(as_uuid=True)),
Column("severity", Enum("low", "medium", "high", "critical", name="alert_severity"), nullable=False),
Column("title", Text, nullable=False),
Column("message", Text),
Column("context", JSON),
Column("acknowledged", Integer, server_default="0"),
Column("acknowledged_by", String(256)),
Column("created_at", DateTime(timezone=True), server_default=func.now(), nullable=False),
Column("resolved_at", DateTime(timezone=True)),
)
Index("ix_alerts_severity_created", alerts.c.severity, alerts.c.created_at.desc())
Index("ix_alerts_entity", alerts.c.entity_id)
# ── Documents (stored in MinIO, indexed here) ────────────────────────────
documents = Table(
"documents",
metadata,
Column("id", UUID(as_uuid=True), primary_key=True, default=uuid4),
Column("bucket", String(256), nullable=False),
Column("object_key", String(1024), nullable=False),
Column("content_type", String(256)),
Column("size_bytes", Integer),
Column("description", Text),
Column("tags", JSON),
Column("event_id", UUID(as_uuid=True)),
Column("uploaded_at", DateTime(timezone=True), server_default=func.now()),
)

View file

@ -1,13 +0,0 @@
fastapi>=0.115
uvicorn[standard]>=0.34
sqlalchemy[asyncio]>=2.0
asyncpg>=0.30
nats-py>=2.9
redis[hiredis]>=5.2
minio>=7.2
pydantic>=2.10
alembic>=1.14
httpx>=0.28
feedparser>=6.0
python-dateutil>=2.9
structlog>=24.4

View file

@ -1,242 +0,0 @@
"""OSINT Dashboard — Pydantic schemas."""
from __future__ import annotations
from datetime import datetime
from enum import Enum
from typing import Optional
from uuid import UUID
from pydantic import BaseModel, Field
# ─── Enums ───────────────────────────────────────────────────────────────
class SourceType(str, Enum):
rss = "rss"
gdel_t2 = "gdel-t2"
social = "social"
earthquake = "earthquake"
disaster = "disaster"
weather = "weather"
fire = "fire"
satellite = "satellite"
class EntityKind(str, Enum):
person = "person"
organization = "organization"
location = "location"
topic = "topic"
asset = "asset"
class Sentiment(str, Enum):
positive = "positive"
neutral = "neutral"
negative = "negative"
class AlertType(str, Enum):
entity_mention = "entity_mention"
sentiment_shift = "sentiment_shift"
geo_proximity = "geo_proximity"
keyword_match = "keyword_match"
threshold = "threshold"
anomaly = "anomaly"
class AlertSeverity(str, Enum):
low = "low"
medium = "medium"
high = "high"
critical = "critical"
# ─── Feed Sources ───────────────────────────────────────────────────────
class FeedSourceCreate(BaseModel):
name: str
source_type: SourceType
url: Optional[str] = None
config: Optional[dict] = None
class FeedSourceOut(BaseModel):
id: UUID
name: str
source_type: SourceType
url: Optional[str]
config: Optional[dict]
enabled: bool
created_at: datetime
# ─── Events ─────────────────────────────────────────────────────────────
class EventCreate(BaseModel):
source_type: SourceType
source_id: Optional[UUID] = None
title: Optional[str] = None
body: Optional[str] = None
url: Optional[str] = None
sentiment_score: Optional[float] = None
sentiment_label: Optional[Sentiment] = None
location_lat: Optional[float] = None
location_lon: Optional[float] = None
location_name: Optional[str] = None
entities: Optional[list[dict]] = None
tags: Optional[list[str]] = None
raw: Optional[dict] = None
source_timestamp: Optional[datetime] = None
class EventOut(BaseModel):
id: UUID
source_type: SourceType
source_id: Optional[UUID]
title: Optional[str]
body: Optional[str]
url: Optional[str]
sentiment_score: Optional[float]
sentiment_label: Optional[Sentiment]
location_lat: Optional[float]
location_lon: Optional[float]
location_name: Optional[str]
entities: Optional[list[dict]]
tags: Optional[list[str]]
ingested_at: datetime
source_timestamp: datetime
# ─── Search ──────────────────────────────────────────────────────────────
class SearchQuery(BaseModel):
q: str = Field(..., min_length=1, max_length=500)
source_type: Optional[SourceType] = None
entity_id: Optional[UUID] = None
sentiment: Optional[Sentiment] = None
min_date: Optional[datetime] = None
max_date: Optional[datetime] = None
min_lat: Optional[float] = None
max_lat: Optional[float] = None
min_lon: Optional[float] = None
max_lon: Optional[float] = None
limit: int = Field(50, ge=1, le=500)
offset: int = Field(0, ge=0)
class SearchResult(BaseModel):
events: list[EventOut]
total: int
has_more: bool
# ─── Entities ────────────────────────────────────────────────────────────
class EntityCreate(BaseModel):
name: str
entity_type: EntityKind
aliases: Optional[list[str]] = None
description: Optional[str] = None
metadata: Optional[dict] = None
location_lat: Optional[float] = None
location_lon: Optional[float] = None
class EntityOut(BaseModel):
id: UUID
name: str
entity_type: EntityKind
aliases: Optional[list[str]]
description: Optional[str]
metadata: Optional[dict]
location_lat: Optional[float]
location_lon: Optional[float]
event_count: int
first_seen: datetime
last_seen: datetime
# ─── Alerts ──────────────────────────────────────────────────────────────
class AlertCreate(BaseModel):
alert_type: AlertType
entity_id: Optional[UUID] = None
event_id: Optional[UUID] = None
severity: AlertSeverity
title: str
message: Optional[str] = None
context: Optional[dict] = None
class AlertUpdate(BaseModel):
acknowledged: Optional[bool] = None
acknowledged_by: Optional[str] = None
resolved_at: Optional[datetime] = None
class AlertOut(BaseModel):
id: UUID
alert_type: AlertType
entity_id: Optional[UUID]
event_id: Optional[UUID]
severity: AlertSeverity
title: str
message: Optional[str]
context: Optional[dict]
acknowledged: bool
acknowledged_by: Optional[str]
created_at: datetime
resolved_at: Optional[datetime]
# ─── Documents ───────────────────────────────────────────────────────────
class DocumentCreate(BaseModel):
bucket: str
object_key: str
content_type: Optional[str] = None
size_bytes: Optional[int] = None
description: Optional[str] = None
tags: Optional[list[str]] = None
event_id: Optional[UUID] = None
class DocumentOut(BaseModel):
id: UUID
bucket: str
object_key: str
content_type: Optional[str]
size_bytes: Optional[int]
description: Optional[str]
tags: Optional[list[str]]
event_id: Optional[UUID]
uploaded_at: datetime
# ─── Aggregations ────────────────────────────────────────────────────────
class SentimentSummary(BaseModel):
period: str
positive_count: int
neutral_count: int
negative_count: int
avg_score: float
class TimelinePoint(BaseModel):
timestamp: datetime
event_count: int
avg_sentiment: float
class DashboardSummary(BaseModel):
total_events: int
events_last_24h: int
active_sources: int
open_alerts: int
tracked_entities: int
sentiment: SentimentSummary
top_entities: list[EntityOut]

View file

@ -1,186 +0,0 @@
"""Data source ingestors — fetch from external APIs and push to NATS."""
from __future__ import annotations
import json
import logging
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
import httpx
import feedparser
import nats
logger = logging.getLogger("osint.sources")
def _parse_rfc822(date_str: object) -> str | None:
"""Parse RFC-822 date string from feedparser entries."""
if not isinstance(date_str, str):
return None
try:
return parsedate_to_datetime(date_str).astimezone(timezone.utc).isoformat()
except (ValueError, TypeError):
return None
# NATS connection
NATS_URLS = "nats://osint-nats.customer1.svc.cluster.local:4222"
async def publish_event(subject: str, event: dict):
"""Publish an event to NATS JetStream."""
nc = await nats.connect(NATS_URLS)
js = nc.jetstream()
await js.publish(subject, json.dumps(event).encode())
await nc.close()
logger.debug("Published event to %s", subject)
# ─── RSS Feed Ingestor ──────────────────────────────────────────────────
async def ingest_rss_feed(feed_url: str, source_id: str = None):
"""Fetch and parse an RSS feed, publish items to NATS."""
async with httpx.AsyncClient(timeout=30) as client:
resp = await client.get(feed_url)
resp.raise_for_status()
feed = feedparser.parse(resp.text)
count = 0
for entry in feed.entries[:100]: # max 100 per run
event = {
"source_type": "rss",
"source_id": source_id,
"title": entry.get("title"),
"body": entry.get("summary") or entry.get("description"),
"url": entry.get("link"),
"source_timestamp": _parse_rfc822(entry.get("published"))
or datetime.now(timezone.utc).isoformat(),
"tags": [t.get("term") for t in entry.get("tags", []) if t.get("term")],
"raw": {
"feed_title": feed.feed.get("title"),
"author": entry.get("author"),
"categories": [c.get("term") for c in entry.get("categories", [])],
},
}
await publish_event("events.rss", event)
count += 1
logger.info("Ingested %d items from RSS feed %s", count, feed_url)
return count
# ─── GDELT 2.0 Ingestor ─────────────────────────────────────────────────
GDELT_API = "https://api.gdeltproject.org/gdeltv2"
async def ingest_gdelt(query: str = "", max_articles: int = 50):
"""Fetch articles from GDELT 2.0 API."""
params = {
"mode": "artlist",
"format": "json",
"maxrecords": max_articles,
"mode": "artlist",
}
if query:
params["search"] = query
async with httpx.AsyncClient(timeout=60) as client:
resp = await client.get(GDELT_API, params=params)
resp.raise_for_status()
data = resp.json()
count = 0
for article in data.get("articles", []):
event = {
"source_type": "gdel-t2",
"title": article.get("title"),
"body": article.get("articleBody"),
"url": article.get("url"),
"sentiment_score": _parse_gdelt_tone(article.get("Tone", "0")),
"location_lat": article.get("Latitude"),
"location_lon": article.get("Longitude"),
"location_name": article.get("Location"),
"source_timestamp": article.get("FirstCreated"),
"entities": [
{"name": e.get("Topic"), "type": "topic"}
for e in article.get("Mentions", [])
if e.get("Topic")
],
"raw": article,
}
await publish_event("events.gdelt", event)
count += 1
logger.info("Ingested %d articles from GDELT", count)
return count
def _parse_gdelt_tone(tone: str) -> float | None:
"""Parse GDELT tone string to a -1..1 sentiment score."""
try:
tone_float = float(tone)
return max(-1.0, min(1.0, tone_float / 4249.0)) # GDELT tone range
except (ValueError, TypeError):
return None
# ─── Earthquake Ingestor (USGS) ─────────────────────────────────────────
USGS_API = "https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.geojson"
async def ingest_earthquakes():
"""Fetch recent earthquakes from USGS."""
async with httpx.AsyncClient(timeout=30) as client:
resp = await client.get(USGS_API)
resp.raise_for_status()
data = resp.json()
count = 0
for feature in data.get("features", []):
props = feature.get("properties", {})
geometry = feature.get("geometry", {}).get("coordinates", [])
event = {
"source_type": "earthquake",
"title": props.get("title"),
"body": props.get("description"),
"url": props.get("url"),
"location_lat": geometry[1] if len(geometry) > 1 else None,
"location_lon": geometry[0] if len(geometry) > 0 else None,
"location_name": props.get("place"),
"sentiment_label": "neutral",
"tags": [f"magnitude:{props.get('mag')}"] if props.get("mag") else [],
"source_timestamp": (
datetime.utcfromtimestamp(props.get("time", 0) / 1000)
.replace(tzinfo=timezone.utc)
.isoformat()
),
"raw": props,
}
await publish_event("events.earthquake", event)
count += 1
logger.info("Ingested %d earthquake events", count)
return count
# ─── Social Signals (Twitter/X-like placeholder) ────────────────────────
async def ingest_social_signals(query: str = "", max_items: int = 50):
"""Placeholder for social media signal ingestion.
In production, this would connect to Twitter API, Reddit, NewsAPI, etc.
For now, it publishes a heartbeat to signal the pipeline is active.
"""
event = {
"source_type": "social",
"title": f"Social signal scan: {query}",
"body": f"Scanned for '{query}' — placeholder connector",
"source_timestamp": datetime.now(timezone.utc).isoformat(),
"tags": [query] if query else [],
"raw": {"query": query, "max_items": max_items, "connector": "placeholder"},
}
await publish_event("events.social", event)
logger.info("Social signal scan complete for '%s'", query)
return 1

View file

@ -1,307 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>OSINT Dashboard</title>
<style>
:root { --bg: #0f172a; --surface: #1e293b; --border: #334155; --text: #e2e8f0; --muted: #94a3b8; --accent: #38bdf8; --green: #4ade80; --red: #f87171; --yellow: #fbbf24; }
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, system-ui, sans-serif; background: var(--bg); color: var(--text); min-height: 100vh; }
header { background: var(--surface); border-bottom: 1px solid var(--border); padding: 1rem 2rem; display: flex; justify-content: space-between; align-items: center; }
header h1 { font-size: 1.25rem; color: var(--accent); }
.status { font-size: 0.85rem; color: var(--muted); }
.status .ok { color: var(--green); }
.container { max-width: 1400px; margin: 0 auto; padding: 1.5rem; }
.grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(280px, 1fr)); gap: 1rem; margin-bottom: 1.5rem; }
.card { background: var(--surface); border: 1px solid var(--border); border-radius: 8px; padding: 1.25rem; }
.card h3 { font-size: 0.8rem; text-transform: uppercase; color: var(--muted); margin-bottom: 0.5rem; }
.card .value { font-size: 2rem; font-weight: 700; }
.card .sub { font-size: 0.85rem; color: var(--muted); margin-top: 0.25rem; }
.section { margin-bottom: 1.5rem; }
.section h2 { font-size: 1rem; margin-bottom: 0.75rem; color: var(--accent); }
table { width: 100%; border-collapse: collapse; font-size: 0.9rem; }
th, td { text-align: left; padding: 0.6rem 0.75rem; border-bottom: 1px solid var(--border); }
th { color: var(--muted); font-weight: 500; font-size: 0.8rem; }
.badge { display: inline-block; padding: 0.15rem 0.5rem; border-radius: 9999px; font-size: 0.75rem; font-weight: 500; }
.badge-positive { background: #052e16; color: var(--green); }
.badge-negative { background: #450a0a; color: var(--red); }
.badge-neutral { background: #1e293b; color: var(--muted); }
.badge-high { background: #450a0a; color: var(--red); }
.badge-medium { background: #451a03; color: var(--yellow); }
.badge-low { background: #052e16; color: var(--green); }
.search-box { display: flex; gap: 0.5rem; margin-bottom: 1rem; }
.search-box input { flex: 1; background: var(--surface); border: 1px solid var(--border); border-radius: 6px; padding: 0.6rem 1rem; color: var(--text); font-size: 0.9rem; }
.search-box input:focus { outline: none; border-color: var(--accent); }
.search-box button { background: var(--accent); color: #0f172a; border: none; border-radius: 6px; padding: 0.6rem 1.25rem; font-weight: 500; cursor: pointer; }
.btn { background: var(--surface); border: 1px solid var(--border); color: var(--text); border-radius: 6px; padding: 0.5rem 1rem; cursor: pointer; font-size: 0.85rem; }
.btn:hover { border-color: var(--accent); }
.sentiment-bar { display: flex; height: 24px; border-radius: 4px; overflow: hidden; margin-top: 0.5rem; }
.sentiment-bar div { transition: width 0.3s; }
.tab-bar { display: flex; gap: 0.5rem; margin-bottom: 1rem; }
.tab-bar .btn.active { background: var(--accent); color: #0f172a; }
</style>
</head>
<body>
<header>
<h1>OSINT Dashboard</h1>
<div class="status">
Status: <span id="health" class="ok">checking...</span>
&nbsp;|&nbsp; Last update: <span id="last-update">-</span>
</div>
</header>
<div class="container">
<!-- Summary Cards -->
<div class="grid" id="summary-cards">
<div class="card"><h3>Total Events</h3><div class="value" id="total-events">-</div></div>
<div class="card"><h3>Events (24h)</h3><div class="value" id="events-24h">-</div><div class="sub">last 24 hours</div></div>
<div class="card"><h3>Active Sources</h3><div class="value" id="active-sources">-</div></div>
<div class="card"><h3>Open Alerts</h3><div class="value" id="open-alerts" style="color:var(--red)">-</div></div>
<div class="card"><h3>Tracked Entities</h3><div class="value" id="tracked-entities">-</div></div>
<div class="card">
<h3>Sentiment (24h)</h3>
<div class="sentiment-bar">
<div id="sent-pos" style="background:var(--green)"></div>
<div id="sent-neu" style="background:var(--muted)"></div>
<div id="sent-neg" style="background:var(--red)"></div>
</div>
<div class="sub" id="sent-detail"></div>
</div>
</div>
<!-- Search -->
<div class="section">
<h2>Search Events</h2>
<div class="search-box">
<input type="text" id="search-q" placeholder="Search events by keyword..." />
<button onclick="searchEvents()">Search</button>
</div>
</div>
<!-- Tabs -->
<div class="tab-bar">
<button class="btn active" onclick="showTab('recent')">Recent Events</button>
<button class="btn" onclick="showTab('alerts')">Alerts</button>
<button class="btn" onclick="showTab('entities')">Entities</button>
<button class="btn" onclick="showTab('ingest')">Ingest</button>
</div>
<!-- Recent Events -->
<div class="section" id="tab-recent">
<h2>Recent Events</h2>
<table>
<thead><tr><th>Time</th><th>Source</th><th>Title</th><th>Sentiment</th><th>Location</th></tr></thead>
<tbody id="events-body"></tbody>
</table>
</div>
<!-- Alerts -->
<div class="section" id="tab-alerts" style="display:none">
<h2>Open Alerts</h2>
<table>
<thead><tr><th>Time</th><th>Severity</th><th>Type</th><th>Title</th></tr></thead>
<tbody id="alerts-body"></tbody>
</table>
</div>
<!-- Entities -->
<div class="section" id="tab-entities" style="display:none">
<h2>Tracked Entities</h2>
<table>
<thead><tr><th>Name</th><th>Type</th><th>Events</th><th>Last Seen</th></tr></thead>
<tbody id="entities-body"></tbody>
</table>
</div>
<!-- Ingest -->
<div class="section" id="tab-ingest" style="display:none">
<h2>Data Ingestion</h2>
<div class="grid">
<div class="card">
<h3>RSS Feed</h3>
<div style="margin-top:0.5rem;display:flex;gap:0.5rem">
<input type="text" id="rss-url" placeholder="https://example.com/feed" style="flex:1;background:var(--bg);border:1px solid var(--border);border-radius:4px;padding:0.4rem;color:var(--text);font-size:0.85rem">
<button class="btn" onclick="ingestRSS()">Fetch</button>
</div>
</div>
<div class="card">
<h3>GDELT Articles</h3>
<p class="sub" style="margin:0.5rem 0">Global news monitoring</p>
<button class="btn" onclick="ingestGDELT()">Fetch Latest</button>
</div>
<div class="card">
<h3>Earthquakes (USGS)</h3>
<p class="sub" style="margin:0.5rem 0">Last hour of seismic data</p>
<button class="btn" onclick="ingestEarthquakes()">Fetch Latest</button>
</div>
<div class="card">
<h3>NATS Consumer</h3>
<p class="sub" style="margin:0.5rem 0">Process pending messages</p>
<button class="btn" onclick="processNATS()">Process</button>
</div>
</div>
<div id="ingest-result" style="margin-top:1rem;color:var(--muted);font-size:0.9rem"></div>
</div>
<!-- Search Results -->
<div class="section" id="search-results" style="display:none">
<h2>Search Results (<span id="search-total">0</span>)</h2>
<table>
<thead><tr><th>Time</th><th>Source</th><th>Title</th><th>Sentiment</th><th>Location</th></tr></thead>
<tbody id="search-body"></tbody>
</table>
</div>
</div>
<script>
const API = '';
async function loadSummary() {
try {
const r = await fetch(`${API}/api/analytics/summary`);
const d = await r.json();
document.getElementById('total-events').textContent = d.total_events;
document.getElementById('events-24h').textContent = d.events_last_24h;
document.getElementById('active-sources').textContent = d.active_sources;
document.getElementById('open-alerts').textContent = d.open_alerts;
document.getElementById('tracked-entities').textContent = d.tracked_entities;
const s = d.sentiment;
const total = s.positive_count + s.neutral_count + s.negative_count || 1;
document.getElementById('sent-pos').style.width = ((s.positive_count/total)*100)+'%';
document.getElementById('sent-neu').style.width = ((s.neutral_count/total)*100)+'%';
document.getElementById('sent-neg').style.width = ((s.negative_count/total)*100)+'%';
document.getElementById('sent-detail').textContent = `+${s.positive_count} | ~${s.neutral_count} | -${s.negative_count} (avg: ${s.avg_score.toFixed(3)})`;
} catch(e) { console.error('Summary load failed', e); }
}
async function loadEvents() {
try {
const r = await fetch(`${API}/api/events?limit=30`);
const d = await r.json();
const tb = document.getElementById('events-body');
tb.innerHTML = d.map(e => `<tr>
<td>${new Date(e.ingested_at).toLocaleString()}</td>
<td>${e.source_type}</td>
<td>${(e.title||'').substring(0,80)}</td>
<td>${sentimentBadge(e.sentiment_label)}</td>
<td>${e.location_name || '-'}</td>
</tr>`).join('');
} catch(e) { console.error('Events load failed', e); }
}
async function loadAlerts() {
try {
const r = await fetch(`${API}/api/alerts?limit=20`);
const d = await r.json();
const tb = document.getElementById('alerts-body');
tb.innerHTML = d.map(a => `<tr>
<td>${new Date(a.created_at).toLocaleString()}</td>
<td><span class="badge badge-${a.severity}">${a.severity}</span></td>
<td>${a.alert_type}</td>
<td>${a.title}</td>
</tr>`).join('');
} catch(e) { console.error('Alerts load failed', e); }
}
async function loadEntities() {
try {
const r = await fetch(`${API}/api/entities?limit=20`);
const d = await r.json();
const tb = document.getElementById('entities-body');
tb.innerHTML = d.map(e => `<tr>
<td>${e.name}</td>
<td>${e.entity_type}</td>
<td>${e.event_count}</td>
<td>${new Date(e.last_seen).toLocaleString()}</td>
</tr>`).join('');
} catch(e) { console.error('Entities load failed', e); }
}
async function searchEvents() {
const q = document.getElementById('search-q').value.trim();
if (!q) return;
try {
const r = await fetch(`${API}/api/search`, {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({q, limit: 50})
});
const d = await r.json();
document.getElementById('search-total').textContent = d.total;
document.getElementById('search-results').style.display = 'block';
const tb = document.getElementById('search-body');
tb.innerHTML = d.events.map(e => `<tr>
<td>${new Date(e.ingested_at).toLocaleString()}</td>
<td>${e.source_type}</td>
<td>${(e.title||'').substring(0,80)}</td>
<td>${sentimentBadge(e.sentiment_label)}</td>
<td>${e.location_name || '-'}</td>
</tr>`).join('');
} catch(e) { console.error('Search failed', e); }
}
function sentimentBadge(label) {
if (!label) return '-';
const cls = {positive:'badge-positive',negative:'badge-negative',neutral:'badge-neutral'}[label]||'badge-neutral';
return `<span class="badge ${cls}">${label}</span>`;
}
function showTab(name) {
['recent','alerts','entities','ingest'].forEach(t => {
document.getElementById('tab-'+t).style.display = t===name?'block':'none';
});
document.querySelectorAll('.tab-bar .btn').forEach((b,i) => {
b.classList.toggle('active', ['recent','alerts','entities','ingest'][i]===name);
});
if (name==='alerts') loadAlerts();
if (name==='entities') loadEntities();
}
async function ingestRSS() {
const url = document.getElementById('rss-url').value;
const r = await fetch(`${API}/api/ingest/rss?feed_url=${encodeURIComponent(url)}`, {method:'POST'});
const d = await r.json();
document.getElementById('ingest-result').textContent = `RSS: ${d.items_ingested} items ingested`;
loadSummary(); loadEvents();
}
async function ingestGDELT() {
const r = await fetch(`${API}/api/ingest/gdelt?max_articles=50`, {method:'POST'});
const d = await r.json();
document.getElementById('ingest-result').textContent = `GDELT: ${d.articles_ingested} articles ingested`;
loadSummary(); loadEvents();
}
async function ingestEarthquakes() {
const r = await fetch(`${API}/api/ingest/earthquakes`, {method:'POST'});
const d = await r.json();
document.getElementById('ingest-result').textContent = `USGS: ${d.events_ingested} events ingested`;
loadSummary(); loadEvents();
}
async function processNATS() {
const r = await fetch(`${API}/api/ingest/process?batch_size=100`, {method:'POST'});
const d = await r.json();
document.getElementById('ingest-result').textContent = `NATS: ${d.processed} messages processed`;
loadSummary(); loadEvents();
}
async function checkHealth() {
try {
const r = await fetch(`${API}/api/health`);
const d = await r.json();
document.getElementById('health').textContent = 'healthy';
document.getElementById('last-update').textContent = new Date().toLocaleTimeString();
} catch(e) {
document.getElementById('health').textContent = 'unreachable';
document.getElementById('health').className = '';
}
}
// Initial load
loadSummary(); loadEvents(); checkHealth();
setInterval(() => { loadSummary(); loadEvents(); checkHealth(); }, 30000);
</script>
</body>
</html>

View file

@ -1,21 +0,0 @@
FROM python:3.13-slim
LABEL maintainer="sirius0xdev" \
description="VWAP Wave Breach Scanner — monitors Gold, NASDAQ, S&P, Crude Oil"
WORKDIR /app
COPY app/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app/scanner.py .
# Default scan interval: 120s (2 min), threshold: 2σ
ENV SCAN_INTERVAL_SEC=120 \
BREACH_THRESHOLD=2.0
# Health check: ensure process is alive
HEALTHCHECK --interval=60s --timeout=10s --retries=3 \
CMD ["python3", "-c", "import os; assert os.path.exists('/proc/1/fd/0')"]
CMD ["python3", "scanner.py"]

View file

@ -1,3 +0,0 @@
yfinance>=0.2.54
requests>=2.32
apscheduler>=3.10

View file

@ -1,205 +0,0 @@
#!/usr/bin/env python3
"""
VWAP Wave Breach Scanner
========================
Continuous monitoring daemon. Scans instruments on a schedule,
detects VWAP wave breaches, and pushes alerts via Telegram.
Environment variables:
TELEGRAM_BOT_TOKEN Telegram Bot API token (required)
TELEGRAM_CHAT_ID Chat ID to send alerts to (required)
SCAN_INTERVAL_SEC Seconds between scans (default: 120)
BREACH_THRESHOLD Sigma threshold for alerts (default: 2.0)
No LLM overhead pure Python, ~20MB RAM.
"""
import os
import sys
import time
import logging
from datetime import datetime, timezone
import requests
import yfinance as yf
import pandas as pd
from apscheduler.schedulers.background import BlockingScheduler
# ── Config ──────────────────────────────────────────────────────────────────
TELEGRAM_BOT_TOKEN = os.environ["TELEGRAM_BOT_TOKEN"]
TELEGRAM_CHAT_ID = os.environ["TELEGRAM_CHAT_ID"]
SCAN_INTERVAL_SEC = int(os.environ.get("SCAN_INTERVAL_SEC", "120"))
BREACH_THRESHOLD = float(os.environ.get("BREACH_THRESHOLD", "2.0"))
INSTRUMENTS = {
"Gold Futures": {"ticker": "GC=F", "decimal": 2},
"NASDAQ": {"ticker": "^IXIC", "decimal": 2},
"S&P 500": {"ticker": "^GSPC", "decimal": 2},
"Crude Oil": {"ticker": "CL=F", "decimal": 2},
}
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)-5s %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
log = logging.getLogger(__name__)
# Track last alert state to prevent spam (no repeat within same threshold direction)
_last_alert = {}
# ── Telegram ────────────────────────────────────────────────────────────────
def send_telegram(message: str) -> bool:
"""Send a message via Telegram Bot API."""
url = f"https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/sendMessage"
payload = {
"chat_id": TELEGRAM_CHAT_ID,
"text": message,
"parse_mode": "Markdown",
"disable_web_page_preview": True,
}
try:
resp = requests.post(url, json=payload, timeout=10)
resp.raise_for_status()
log.info("Telegram alert sent: %s", message[:80])
return True
except Exception as e:
log.error("Telegram send failed: %s", e)
return False
# ── VWAP Calculation ────────────────────────────────────────────────────────
def compute_vwap(data: pd.DataFrame) -> dict | None:
"""Compute cumulative VWAP, σ, and deviation."""
df = data.copy()
df["typical"] = (df["High"] + df["Low"] + df["Close"]) / 3.0
df["tp_vol"] = df["typical"] * df["Volume"]
cum_tp_vol = df["tp_vol"].cumsum()
cum_vol = df["Volume"].cumsum().replace(0, 1)
df["cum_vwap"] = cum_tp_vol / cum_vol
df["deviation"] = df["typical"] - df["cum_vwap"]
df["cum_var"] = (df["deviation"] ** 2).cumsum() / cum_vol
df["sigma"] = df["cum_var"] ** 0.5
last = df.iloc[-1]
if last["sigma"] <= 0:
return None
return {
"price": last["Close"],
"vwap": last["cum_vwap"],
"sigma": last["sigma"],
"dev_sigmas": (last["Close"] - last["cum_vwap"]) / last["sigma"],
}
def fetch_data(ticker: str) -> pd.DataFrame:
"""Fetch recent intraday data via yfinance."""
try:
data = yf.Ticker(ticker).history(period="1d", interval="1m", auto_adjust=True)
except Exception:
data = pd.DataFrame()
if len(data) < 30:
data = yf.Ticker(ticker).history(period="5d", interval="1m", auto_adjust=True)
cutoff = pd.Timestamp.now(tz=data.index.tz) - pd.Timedelta(hours=24)
data = data[data.index >= cutoff]
return data
# ── Scanner ─────────────────────────────────────────────────────────────────
def run_scan() -> None:
"""Execute a full scan cycle and push any breaches."""
ts = datetime.now(timezone.utc).strftime("%H:%M:%S UTC")
log.info("── Scan %s ──", ts)
breaches = []
for name, cfg in INSTRUMENTS.items():
try:
data = fetch_data(cfg["ticker"])
if data.empty or len(data) < 20:
log.warning("SKIP %s — insufficient data (%d bars)", name, len(data))
continue
info = compute_vwap(data)
if info is None:
log.warning("SKIP %s — zero sigma", name)
continue
d = cfg["decimal"]
dev = info["dev_sigmas"]
log.info(" %-14s $%10.2f | VWAP $%10.2f | %+.2fσ", name, info["price"], info["vwap"], dev)
if abs(dev) >= BREACH_THRESHOLD:
# Prevent repeat spam: only alert if state changed
alert_key = f"{name}:{dev > 0}"
if _last_alert.get(alert_key) == "breach":
log.info("%s already in breach, skipping repeat", name)
continue
breaches.append((name, cfg["decimal"], dev, info["price"], info["vwap"], info["sigma"]))
_last_alert[alert_key] = "breach"
else:
_last_alert[f"{name}:True"] = "clean"
_last_alert[f"{name}:False"] = "clean"
except Exception as e:
log.error("ERROR %s: %s", name, e)
# Push alerts
for name, d, dev, price, vwap, sigma in breaches:
direction = "⬆️ UP" if dev > 0 else "⬇️ DOWN"
band_label = f"±{int(abs(dev))}σ"
severity = "🚨 **EXTREME**" if abs(dev) >= 3.0 else "⚡ **BREACH**"
msg = (
f"{severity} — VWAP Wave Alert\n\n"
f"**{name}** broke through **{band_label}** band\n"
f"Deviation: **{dev:+.2f}σ**\n"
f"Price: **${price:.{d}f}** | VWAP: **${vwap:.{d}f}**\n"
f"σ: ${sigma:.{d}f} | {direction}\n\n"
f"_at {ts}_"
)
send_telegram(msg)
# ── Main ────────────────────────────────────────────────────────────────────
def main() -> None:
log.info("=" * 60)
log.info(" VWAP Wave Breach Scanner")
log.info(" Interval: %d sec | Threshold: ±%.1fσ", SCAN_INTERVAL_SEC, BREACH_THRESHOLD)
log.info(" Telegram: @chat_id=%s", TELEGRAM_CHAT_ID)
log.info("=" * 60)
# Validate Telegram connectivity
send_telegram(
"🟢 *VWAP Breach Scanner* is online.\n"
f"Scanning every **{SCAN_INTERVAL_SEC}s** — threshold ±**{BREACH_THRESHOLD:.1f}σ**\n"
f"Monitoring: Gold, NASDAQ, S&P 500, Crude Oil"
)
scheduler = BlockingScheduler()
scheduler.add_job(run_scan, "interval", seconds=SCAN_INTERVAL_SEC, id="scan")
# Run immediately on start
run_scan()
log.info("Scanner running. Press Ctrl+C to stop.")
try:
scheduler.start()
except KeyboardInterrupt:
log.info("Shutting down...")
scheduler.shutdown()
if __name__ == "__main__":
main()

View file

@ -1,5 +0,0 @@
# ─── Config ─────────────────────────────────────────────────────────────────
# Edit these values. They will be injected into the deployment automatically.
SCAN_INTERVAL_SEC=120
BREACH_THRESHOLD=2.0

View file

@ -1,70 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: vwap-monitor
namespace: customer1
labels:
app: vwap-monitor
component: scanner
spec:
replicas: 1
strategy:
type: Recreate # only one instance should run
selector:
matchLabels:
app: vwap-monitor
template:
metadata:
labels:
app: vwap-monitor
component: scanner
annotations:
# Restart if config changes
checksum/config: "vwap-monitor-config"
spec:
terminationGracePeriodSeconds: 30
containers:
- name: scanner
image: us-central1-docker.pkg.dev/devops-lab-cluster/customer1/vwap-monitor:latest
imagePullPolicy: Always
resources:
requests:
cpu: 100m
memory: 64Mi
limits:
cpu: 250m
memory: 256Mi
envFrom:
- configMapRef:
name: vwap-monitor-config
env:
- name: TELEGRAM_BOT_TOKEN
valueFrom:
secretKeyRef:
name: vwap-monitor-secrets
key: telegram-bot-token
- name: TELEGRAM_CHAT_ID
valueFrom:
secretKeyRef:
name: vwap-monitor-secrets
key: telegram-chat-id
startupProbe:
exec:
command: ["/bin/sh", "-c", "python3 -c 'import scanner'"]
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 3
livenessProbe:
exec:
command: ["/bin/sh", "-c", "kill -0 1"]
initialDelaySeconds: 30
periodSeconds: 60
---
apiVersion: v1
kind: ConfigMap
metadata:
name: vwap-monitor-config
namespace: customer1
data:
SCAN_INTERVAL_SEC: "120"
BREACH_THRESHOLD: "2.0"

View file

@ -1,41 +0,0 @@
# ─── Kustomization ──────────────────────────────────────────────────────────
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: customer1
resources:
- secret.yaml
- deployment.yaml
configMapGenerator:
- name: vwap-monitor-config
envs:
- config.env
patches:
- patch: |-
apiVersion: apps/v1
kind: Deployment
metadata:
name: vwap-monitor
spec:
template:
spec:
containers:
- name: scanner
envFrom:
- configMapRef:
name: vwap-monitor-config
env:
- name: TELEGRAM_BOT_TOKEN
valueFrom:
secretKeyRef:
name: vwap-monitor-secrets
key: telegram-bot-token
- name: TELEGRAM_CHAT_ID
valueFrom:
secretKeyRef:
name: vwap-monitor-secrets
key: telegram-chat-id

View file

@ -1,15 +0,0 @@
# ─── Secret Template ────────────────────────────────────────────────────────
# Replace the values below before applying.
# Alternatively, store in a real Secret Manager (GCP Secret Manager, external-secrets).
apiVersion: v1
kind: Secret
metadata:
name: vwap-monitor-secrets
namespace: customer1
type: Opaque
stringData:
# Get from: https://t.me/botfather → /newbot → copy token
telegram-bot-token: "YOUR_BOT_TOKEN_HERE"
# Get from: @userinfobot or inspect network tab in Telegram Web
telegram-chat-id: "YOUR_CHAT_ID_HERE"

View file

@ -1,8 +0,0 @@
apiVersion: v1
data:
client_id: cGxhY2Vob2xkZXI=
client_secret: cGxhY2Vob2xkZXI=
kind: Secret
metadata:
name: operator-oauth
namespace: tailscale

View file

@ -1,5 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1 apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization kind: Kustomization
resources: resources:
#- ../base/vllm-servers/ # - ../base/vllm-servers/
# - ../base/keda-gpu-scaling/ # - ../base/keda-gpu-scaling/

View file

@ -1,28 +0,0 @@
# OpenClaw Brain v1.1 Implementation Plan
> **Status:** Ready for subagent-driven-development. <48hr goal.
**Goal:** Full product launch per spec. US GKE DeepSeek-V4-Pro API, flat subs, unlimited tokens.
**Updated Pricing Confirmed:** Spot $3.40-4.55/hr node → $2.5K-3.3K/mo full util. Breakeven: 6 Personal ($49) or 2 Team ($199) subs/mo.
**Approach:** Extend gcloud-lab OpenClaw PAaaS (customer1). New namespace `openclaw-brain`. Stripe webhooks for subs/keys.
## Tasks (Bite-Sized TDD)
### Task 1: Scaffold dirs
**Files:** mkdir apps/base/openclaw-brain apps/staging/openclaw-brain
**Step 1:** `mkdir -p apps/{base,staging}/openclaw-brain`
**Step 2:** namespace.yaml (copy customer1 pattern)
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: openclaw-brain
```
**Verify:** `kubectl apply --dry-run=client -f apps/base/openclaw-brain/namespace.yaml`
**Commit:** git add apps/ ; git commit -m \"feat(openclaw-brain): scaffold\"
*(Abbrev; full 30+ tasks: Terraform nodepools w/ machine_type='a3-ultragpu-8g' spot=true gpu=8, vLLM args --model=DeepSeek/DeepSeek-V4-Pro --tp=8 --max-model-len=1e6 --enable-prefix-caching, FastAPI w/ Stripe Subscriptions API + redis-py quotas, KEDA ScaledObject on http_requests &gt;5/min throttle, landing HTML w/ Stripe Checkout.js, flux kustomize add, terraform apply, smoke tests)*
**Next:** Task 1 scaffold + git commit.

View file

@ -1,25 +0,0 @@
# Dual-Tier AI Architecture: L4 Dispatcher & A100 Deep Thinker
This document outlines the cost-optimized, dual-tier LLM architecture deployed in the cluster using OpenClaw, vLLM, and KEDA.
## Concept
Instead of running an expensive A100 GPU 24/7 for all requests, we split the cognitive load into two tiers: a lightweight "Dispatcher" and a heavyweight "Deep Thinker." This mimics a senior/junior developer dynamic, optimizing both response latency and cloud GCP billing.
## Tier 1: The Dispatcher (L4 GPU)
- **Hardware:** 1x NVIDIA L4 (24GB VRAM)
- **Model:** `Qwen2.5-Coder-7B-Instruct`
- **Status:** Runs 24/7 (1 replica)
- **Role:** Acts as the baseline consciousness for OpenClaw. Handles daily chatter, log parsing, straightforward tool routing, and triage. Lightning-fast token generation at a fraction of the cost.
## Tier 2: The Deep Thinker (A100 GPU)
- **Hardware:** 1x NVIDIA A100 (80GB VRAM)
- **Model:** `Qwen3.6-27B-heretic`
- **Status:** Scaled to zero by default.
- **Role:** Activated only for massive context tasks, deep research, and complex multi-file architectural reasoning.
## Scaling & Routing Mechanics (KEDA + OpenClaw)
1. **Scale-to-Zero:** The A100 deployment is managed by a KEDA `HTTPScaledObject`. It scales down to `0` replicas after 15 minutes of inactivity.
2. **Default Routing:** OpenClaw's global default model is set to the L4 endpoint. All standard messages hit the L4 immediately.
3. **Sub-Agent Handoff:** When a complex task is requested, the L4 agent uses the `sessions_spawn` tool to create an isolated sub-agent, overriding the model target to the A100 endpoint.
4. **Cold Start:** KEDA intercepts the sub-agent's request, scales the A100 node from 0 to 1, waits for vLLM to load (~1-2 minutes), and then passes the request through.
5. **Manual Override:** A user can bypass the L4 entirely for a specific session by typing `/model local-vllm/coder3101/Qwen3.5-27B-heretic` in the OpenClaw chat.

View file

@ -1 +0,0 @@
HauhauCS/Qwen3.5-27B-Uncensored-HauhauCS-Aggressive

View file

@ -1,109 +0,0 @@
# Build and push container images to Artifact Registry
name: Build & Push Images
on:
push:
branches: [main, develop]
paths:
- "trading-platform/**"
- "!trading-platform/infra/**"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
env:
GCP_PROJECT_ID: customer1-gke
GCP_REGION: us-central1
ARTIFACT_REGISTRY: us-central1-docker.pkg.dev/${{ env.GCP_PROJECT_ID }}/trading
jobs:
build-and-push:
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@v4
- name: Authenticate to Google Cloud
uses: google-github-actions/auth@v2
with:
workload_identity_provider: projects/${{ env.GCP_PROJECT_ID }}/locations/global/workloadIdentityPools/github-pool/providers/github-provider
service_account: ci-builder@${{ env.GCP_PROJECT_ID }}.iam.gserviceaccount.com
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
- name: Configure Docker for Artifact Registry
run: gcloud auth configure-docker ${{ env.GCP_REGION }}-docker.pkg.dev --quiet
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Generate image tags
id: tags
run: |
SHORT_SHA="${GITHUB_SHA::8}"
BRANCH="${GITHUB_REF#refs/heads/}"
echo "tag_sha=${SHORT_SHA}" >> $GITHUB_OUTPUT
echo "tag_branch=${BRANCH}" >> $GITHUB_OUTPUT
echo "tag_latest=${BRANCH}" >> $GITHUB_OUTPUT
# ---- Execute Service ----
- name: Build and push execute-service
uses: docker/build-push-action@v5
with:
context: trading-platform/execute-service
file: trading-platform/infra/dockerfiles/execute-service/Dockerfile
push: true
tags: |
${{ env.ARTIFACT_REGISTRY }}/execute-service:${{ steps.tags.outputs.tag_sha }}
${{ env.ARTIFACT_REGISTRY }}/execute-service:${{ steps.tags.outputs.tag_branch }}
cache-from: type=gha
cache-to: type=gha,mode=max
# ---- News Service ----
- name: Build and push news-service
uses: docker/build-push-action@v5
with:
context: trading-platform/news-service
file: trading-platform/infra/dockerfiles/news-service/Dockerfile
push: true
tags: |
${{ env.ARTIFACT_REGISTRY }}/news-service:${{ steps.tags.outputs.tag_sha }}
${{ env.ARTIFACT_REGISTRY }}/news-service:${{ steps.tags.outputs.tag_branch }}
cache-from: type=gha
cache-to: type=gha,mode=max
# ---- Data Service ----
- name: Build and push data-service
uses: docker/build-push-action@v5
with:
context: trading-platform/data-service
file: trading-platform/infra/dockerfiles/data-service/Dockerfile
push: true
tags: |
${{ env.ARTIFACT_REGISTRY }}/data-service:${{ steps.tags.outputs.tag_sha }}
${{ env.ARTIFACT_REGISTRY }}/data-service:${{ steps.tags.outputs.tag_branch }}
cache-from: type=gha
cache-to: type=gha,mode=max
# ---- Dashboard ----
- name: Build and push dashboard
uses: docker/build-push-action@v5
with:
context: trading-platform/dashboard
file: trading-platform/infra/dockerfiles/dashboard/Dockerfile
push: true
tags: |
${{ env.ARTIFACT_REGISTRY }}/dashboard:${{ steps.tags.outputs.tag_sha }}
${{ env.ARTIFACT_REGISTRY }}/dashboard:${{ steps.tags.outputs.tag_branch }}
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Notify deployment pipeline
run: |
echo "Images pushed successfully with tag ${{ steps.tags.outputs.tag_sha }}"
# This can trigger the deploy workflow via repository dispatch
# or be used by the deploy workflow as a workflow_run trigger

View file

@ -1,135 +0,0 @@
# Build and test on pull requests
name: Build & Test
on:
pull_request:
branches: [main, develop]
paths:
- "trading-platform/execute-service/**"
- "trading-platform/news-service/**"
- "trading-platform/data-service/**"
- "trading-platform/dashboard/**"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
# ---- Python services ----
test-execute-service:
runs-on: ubuntu-latest
defaults:
run:
working-directory: trading-platform/execute-service
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install uv
uses: astral-sh/setup-uv@v3
with:
version: "latest"
- name: Install dependencies
run: uv sync --all-extras --dev
- name: Run tests
run: uv run pytest --asyncio-mode=auto -v --tb=short
- name: Lint
run: uv run ruff check app/ tests/
test-news-service:
runs-on: ubuntu-latest
defaults:
run:
working-directory: trading-platform/news-service
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install dependencies
run: |
pip install -r requirements.txt
pip install pytest pytest-asyncio
- name: Run tests
run: python -m pytest -v --tb=short || true
test-data-service:
runs-on: ubuntu-latest
defaults:
run:
working-directory: trading-platform/data-service
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install uv
uses: astral-sh/setup-uv@v3
with:
version: "latest"
- name: Install dependencies
run: uv sync --all-extras --dev
- name: Run tests
run: uv run pytest --asyncio-mode=auto -v --tb=short
# ---- Dashboard ----
test-dashboard:
runs-on: ubuntu-latest
defaults:
run:
working-directory: trading-platform/dashboard
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: "20"
cache: "npm"
cache-dependency-path: trading-platform/dashboard/package-lock.json
- name: Install dependencies
run: npm ci
- name: Build
run: npm run build
- name: Lint
run: npm run lint
# ---- Docker build validation ----
docker-build-check:
needs: [test-execute-service, test-news-service, test-data-service, test-dashboard]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build execute-service
uses: docker/build-push-action@v5
with:
context: trading-platform/execute-service
file: trading-platform/infra/dockerfiles/execute-service/Dockerfile
push: false
load: false
- name: Build news-service
uses: docker/build-push-action@v5
with:
context: trading-platform/news-service
file: trading-platform/infra/dockerfiles/news-service/Dockerfile
push: false
load: false
- name: Build data-service
uses: docker/build-push-action@v5
with:
context: trading-platform/data-service
file: trading-platform/infra/dockerfiles/data-service/Dockerfile
push: false
load: false
- name: Build dashboard
uses: docker/build-push-action@v5
with:
context: trading-platform/dashboard
file: trading-platform/infra/dockerfiles/dashboard/Dockerfile
push: false
load: false

View file

@ -1,132 +0,0 @@
# Deploy to staging/prod via Helm on GKE
name: Deploy
on:
workflow_dispatch:
inputs:
environment:
description: "Target environment"
required: true
default: "staging"
type: choice
options:
- staging
- production
image_tag:
description: "Container image tag (SHA or branch name)"
required: true
type: string
workflow_run:
workflows: ["Build & Push Images"]
types: [completed]
branches: [main, develop]
permissions:
contents: read
id-token: write
jobs:
deploy:
runs-on: ubuntu-latest
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success')
environment: ${{ github.event.inputs.environment || (github.ref == 'refs/heads/main' && 'production' || 'staging') }}
steps:
- uses: actions/checkout@v4
- name: Authenticate to Google Cloud
uses: google-github-actions/auth@v2
with:
workload_identity_provider: projects/customer1-gke/locations/global/workloadIdentityPools/github-pool/providers/github-provider
service_account: ci-deployer@customer1-gke.iam.gserviceaccount.com
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
- name: Configure kubectl for GKE
run: |
gcloud container clusters get-credentials \
${{ github.event.inputs.environment || (github.ref == 'refs/heads/main' && 'prod' || 'staging') }}-cluster \
--region us-central1 \
--project customer1-gke
- name: Install Helm
uses: azure/setup-helm@v3
with:
version: v3.14.0
- name: Install SOPS + Age
run: |
curl -Lo /tmp/sops.zip https://github.com/getsops/sops/releases/download/v3.8.1/sops-v3.8.1_linux.amd64.zip
unzip /tmp/sops.zip -d /tmp/
sudo mv /tmp/sops /usr/local/bin/sops
go install github.com/getsops/gopgs@latest || true
go install filippo.io/age/cmd/age@latest || true
- name: Create namespace
run: |
kubectl create namespace trading --dry-run=client -o yaml | kubectl apply -f -
- name: Decrypt secrets
run: |
# Copy age key for SOPS decryption
mkdir -p /etc/sops
echo "${{ secrets.SOPS_AGE_KEY }}" > /etc/sops/age.key
chmod 600 /etc/sops/age.key
export SOPS_AGE_KEY_FILE=/etc/sops/age.key
# Decrypt secrets
sops -d trading-platform/infra/helm/trading-platform/trading-secrets.yaml > trading-platform/infra/helm/trading-platform/trading-secrets-decrypted.yaml
- name: Deploy with Helm
run: |
IMAGE_TAG="${{ github.event.inputs.image_tag }}"
ENVIRONMENT="${{ github.event.inputs.environment || (github.ref == 'refs/heads/main' && 'production' || 'staging') }}"
helm upgrade --install trading-platform \
trading-platform/infra/helm/trading-platform \
--namespace trading \
--create-namespace \
--set global.environment=${ENVIRONMENT} \
--set executeService.image.tag=${IMAGE_TAG} \
--set newsService.image.tag=${IMAGE_TAG} \
--set dataService.image.tag=${IMAGE_TAG} \
--set dashboard.image.tag=${IMAGE_TAG} \
--wait \
--timeout 10m \
--atomic
- name: Apply decrypted secrets
run: |
export SOPS_AGE_KEY_FILE=/etc/sops/age.key
sops -d trading-platform/infra/helm/trading-platform/trading-secrets.yaml | kubectl apply -f -
- name: Verify deployment
run: |
echo "=== Pod Status ==="
kubectl get pods -n trading
echo ""
echo "=== Service Status ==="
kubectl get svc -n trading
echo ""
echo "=== Ingress ==="
kubectl get ingress -n trading
- name: Post-deployment smoke test
run: |
# Wait for readiness
kubectl wait --for=condition=available --timeout=5m \
deployment/execute-service -n trading
kubectl wait --for=condition=available --timeout=5m \
deployment/news-service -n trading
kubectl wait --for=condition=available --timeout=5m \
deployment/data-service -n trading
kubectl wait --for=condition=available --timeout=5m \
deployment/dashboard -n trading
echo "All services deployed and healthy"
- name: Rollback on failure
if: failure()
run: |
helm rollback trading-platform -n trading --timeout 10m || true
echo "Rolled back to previous release"

View file

@ -1,112 +0,0 @@
# Trading Platform — Kubernetes Deployment
Kubernetes deployment infrastructure for the trading platform microservices running on GKE (customer1 namespace).
## Directory Structure
```
trading-platform/
├── dockerfiles/ # Multi-stage Dockerfiles for each service
│ ├── dashboard/ # Next.js frontend (port 3000)
│ ├── data-service/ # Data pipeline service (port 8000)
│ ├── execute-service/ # Trading engine: Hyperliquid + Solana (port 8000)
│ └── news-service/ # CNPG connector + Kafka producer (port 8000)
├── helm/ # Helm chart for full platform deployment
│ ├── Chart.yaml # Chart metadata
│ ├── values.yaml # Default values (images, replicas, resources, infra)
│ ├── .sops.yaml # SOPS configuration for secret encryption
│ ├── trading-secrets.yaml # SOPS-encrypted secrets template
│ └── templates/ # 19 Kubernetes manifest templates
│ ├── _helpers.tpl # Template helpers
│ ├── namespace.yaml # Namespace resource
│ ├── configmap.yaml # Shared ConfigMap
│ ├── secrets.yaml # Secrets (SOPS-encrypted via trading-secrets.yaml)
│ ├── ingress.yaml # GCE Ingress for all services
│ ├── NOTES.txt # Post-install notes
│ ├── dashboard/ # Dashboard Deployment + Service
│ ├── data-service/ # Data Service Deployment + Service
│ ├── execute-service/ # Execute Service Deployment + Service
│ ├── news-service/ # News Service Deployment + Service
│ ├── infrastructure/ # PostgreSQL, Redis, Kafka
│ ├── network-policies/ # Default deny + explicit allow policies
│ └── cert-manager/ # Certificates & issuers
├── deploy/ # Additional deployment resources
│ ├── k8s/base/ # Raw K8s manifests (non-Helm fallback)
│ ├── helm/ # Individual per-service Helm charts
│ ├── dockerfiles/ # Alternative Dockerfiles (api-gateway, services)
│ ├── docker-compose/ # Local dev compose files
│ ├── scripts/ # deploy.sh, generate-mtls-certs.sh
│ └── mtls/ # mTLS documentation
└── .github/workflows/ # CI/CD pipelines
├── build-test.yml # Build + unit tests on PR
├── build-push.yml # Build + push to GAR on merge
└── deploy.yml # Helm deploy to GKE on push to master
```
## Services
| Service | Port | Description |
|---------|------|-------------|
| Dashboard | 3000 | Next.js trading dashboard |
| Data Service | 8000 | Data pipeline, Postgres + Redis + Kafka consumers |
| Execute Service | 8000 | Trading engine with Hyperliquid + Solana integration |
| News Service | 8000 | CryptoPanic/GNews connector, Kafka producer |
## Infrastructure Components
- **PostgreSQL 17** — Primary database for trades, orders, user data
- **Redis 7** — Caching layer with 3-node cluster
- **Kafka 3.9** (KRaft mode) — Event streaming (trades, orders, news topics)
- **GCE Ingress** — External traffic routing with TLS termination
- **Cert-Manager** — Automatic TLS certificates (Let's Encrypt + internal CA)
- **Network Policies** — Default deny ingress/egress with explicit allow rules
## Deploying
### Prerequisites
- GKE cluster: `customer1-gke` (us-central1)
- Helm 3 installed locally or in CI
- SOPS configured with Age key (`trading-secrets.yaml` must be encrypted)
- Access to `us-central1-docker.pkg.dev/customer1-gke/trading` registry
### Quick Deploy
```bash
# 1. Encrypt secrets (must use the SOPS Age key)
cd helm
sops -e -i trading-secrets.yaml
# 2. Install/upgrade the Helm release
helm upgrade --install trading-platform ./helm \
--namespace customer1 \
--create-namespace \
--values helm/values.yaml \
--set global.environment=production
```
### CI/CD
- **PR opened**`build-test.yml` runs unit tests
- **Merged to master**`build-push.yml` builds images and pushes to GAR
- **Push to master**`deploy.yml` runs `helm upgrade` on GKE
## Secrets
Secrets are managed via [SOPS](https://github.com/getsops/sops) with Age encryption.
The `.sops.yaml` file configures which keys to use for each path.
```bash
# Encrypt the secrets file
sops -e -i helm/trading-secrets.yaml
# Decrypt (for debugging)
sops -d helm/trading-secrets.yaml
```
**Never commit unencrypted secrets to git.**
## Namespace
The platform deploys into the `customer1` namespace on the GKE cluster.
Update `global.namespace` in `helm/values.yaml` or override via `--set` during install.

View file

@ -1,237 +0,0 @@
name: Build, Test, and Deploy Trading Platform
on:
push:
branches: [main]
paths:
- 'trading-platform/**'
pull_request:
branches: [main]
paths:
- 'trading-platform/**'
workflow_dispatch:
inputs:
environment:
description: 'Deploy environment'
type: choice
options:
- staging
- production
default: staging
env:
REGISTRY: ghcr.io
IMAGE_PREFIX: ${{ github.repository_owner }}/trading-platform
permissions:
contents: read
packages: write
jobs:
# ── Test All Services ──────────────────────────────────────────────────
test-python-services:
name: Test Python Services
runs-on: ubuntu-latest
strategy:
matrix:
service: [execute-service, data-service, news-service, solana-quant-bot]
defaults:
run:
working-directory: trading-platform/${{ matrix.service }}
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
cache: 'pip'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e ".[dev]"
working-directory: trading-platform/${{ matrix.service }}
- name: Run tests with coverage
run: |
pytest tests/ --cov=app --cov-report=xml --cov-report=term-missing -v
working-directory: trading-platform/${{ matrix.service }}
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
with:
file: trading-platform/${{ matrix.service }}/coverage.xml
flags: ${{ matrix.service }}
test-dashboard:
name: Test Dashboard (Next.js)
runs-on: ubuntu-latest
defaults:
run:
working-directory: trading-platform/dashboard
steps:
- uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: trading-platform/dashboard/package-lock.json
- name: Install dependencies
run: npm ci
working-directory: trading-platform/dashboard
- name: Run linting
run: npm run lint
working-directory: trading-platform/dashboard
- name: Build application
run: npm run build
working-directory: trading-platform/dashboard
test-api-gateway:
name: Lint API Gateway Configs
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Validate nginx config syntax
run: |
docker run --rm -v $(pwd)/deploy/k8s/base/gateway:/etc/nginx/conf.d:ro nginx:1.25-alpine nginx -t
# ── Build and Push Container Images ─────────────────────────────────────
build-and-push:
needs: [test-python-services, test-dashboard, test-api-gateway]
name: Build & Push Images
runs-on: ubuntu-latest
if: github.event_name == 'push' || github.event_name == 'workflow_dispatch'
strategy:
matrix:
service: [execute-service, data-service, news-service, api-gateway, dashboard, solana-quant-bot]
steps:
- uses: actions/checkout@v4
- name: Log in to Container Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata (tags, labels)
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}/${{ matrix.service }}
tags: |
type=sha,prefix=
type=ref,event=branch
type=semver,pattern={{version}}
type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }}
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
context: .
file: trading-platform/deploy/dockerfiles/${{ matrix.service }}.Dockerfile
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
# ── Deploy to Kubernetes (Helm) ────────────────────────────────────────
deploy-staging:
needs: [build-and-push]
name: Deploy to Staging
runs-on: ubuntu-latest
if: github.event_name == 'push' || (github.event_name == 'workflow_dispatch' && github.event.inputs.environment == 'staging')
environment: staging
steps:
- uses: actions/checkout@v4
- name: Set up kubectl
uses: azure/setup-kubectl@v3
with:
version: 'v1.29.0'
- name: Configure kubeconfig
run: |
echo "${{ secrets.STAGING_KUBECONFIG }}" | base64 -d > $HOME/.kube/config
env:
STAGING_KUBECONFIG: ${{ secrets.STAGING_KUBECONFIG }}
- name: Install Helm
uses: azure/setup-helm@v3
with:
version: 'v3.14.0'
- name: Deploy with Helm (staging)
run: |
helm upgrade --install trading-platform-staging \\
deploy/helm/trading-platform \\
--namespace customer1-staging \\
--create-namespace \\
--set image.tag=${{ github.sha }} \\
--wait --timeout 10m
- name: Verify deployment
run: |
kubectl rollout status deployment/execute-service -n customer1-staging --timeout=5m
kubectl rollout status deployment/data-service -n customer1-staging --timeout=5m
kubectl rollout status deployment/news-service -n customer1-staging --timeout=5m
kubectl rollout status deployment/api-gateway -n customer1-staging --timeout=5m
kubectl rollout status deployment/dashboard -n customer1-staging --timeout=5m
kubectl rollout status deployment/solana-quant-bot -n customer1-staging --timeout=5m
deploy-production:
needs: [deploy-staging]
name: Deploy to Production
runs-on: ubuntu-latest
if: github.event_name == 'workflow_dispatch' && github.event.inputs.environment == 'production'
environment: production
steps:
- uses: actions/checkout@v4
- name: Set up kubectl
uses: azure/setup-kubectl@v3
with:
version: 'v1.29.0'
- name: Configure kubeconfig
run: |
echo "${{ secrets.PRODUCTION_KUBECONFIG }}" | base64 -d > $HOME/.kube/config
- name: Install Helm
uses: azure/setup-helm@v3
with:
version: 'v3.14.0'
- name: Deploy with Helm (production)
run: |
helm upgrade --install trading-platform-production \\
deploy/helm/trading-platform \\
--namespace customer1 \\
--create-namespace \\
--set image.tag=${{ github.sha }} \\
--values deploy/helm/trading-platform/values-production.yaml \\
--wait --timeout 15m
- name: Verify deployment
run: |
kubectl rollout status deployment/execute-service -n customer1 --timeout=5m
kubectl rollout status deployment/data-service -n customer1 --timeout=5m
kubectl rollout status deployment/news-service -n customer1 --timeout=5m
kubectl rollout status deployment/api-gateway -n customer1 --timeout=5m
kubectl rollout status deployment/dashboard -n customer1 --timeout=5m
- name: Run post-deployment health checks
run: |
# Check all services respond to health endpoints
for service in execute-service data-service news-service api-gateway dashboard; do
echo "Health check: $service"
kubectl run healthcheck-$service --rm --restart=Never --image=curlimages/curl \\
--command -- curl -sf http://$service:$(kubectl get svc $service -o jsonpath='{.spec.ports[0].port}')/health || exit 1
done

View file

@ -1,204 +0,0 @@
# =============================================================================
# Docker Compose — Local Development Environment
# =============================================================================
# Brings up all microservices + infrastructure for local development
#
# Usage:
# docker compose -f trading-platform/deploy/docker-compose/docker-compose.dev.yml up -d
# docker compose -f trading-platform/deploy/docker-compose/docker-compose.dev.yml down -v
# docker compose -f trading-platform/deploy/docker-compose/docker-compose.dev.yml logs -f execute-service
# =============================================================================
x-common-env: &common-env
TRADING_ENV: development
LOG_LEVEL: debug
services:
# ── Infrastructure ────────────────────────────────────────────────────
postgres:
image: postgres:16-alpine
container_name: trading-postgres-dev
environment:
POSTGRES_USER: trading
POSTGRES_PASSWORD: trading_dev_password
POSTGRES_DB: trading_db
ports:
- "5432:5432"
volumes:
- postgres-data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U trading -d trading_db"]
interval: 5s
timeout: 3s
retries: 5
restart: unless-stopped
networks:
- trading-network
redis:
image: redis:7-alpine
container_name: trading-redis-dev
ports:
- "6379:6379"
volumes:
- redis-data:/data
command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 5
restart: unless-stopped
networks:
- trading-network
kafka:
image: apache/kafka:3.7.0
container_name: trading-kafka-dev
ports:
- "9092:9092"
- "9093:9093"
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
KAFKA_CLUSTER_ID: MkU3OEVBNTcwT0FBQT0=
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_LOG_RETENTION_HOURS: 24
KAFKA_LOG_SEGMENT_BYTES: 1073741824
volumes:
- kafka-data:/var/lib/kafka/data
healthcheck:
test: ["CMD-SHELL", "kafka-topics.sh --bootstrap-server localhost:9092 --list || exit 1"]
interval: 10s
timeout: 5s
retries: 10
start_period: 30s
restart: unless-stopped
networks:
- trading-network
# ── Microservices ─────────────────────────────────────────────────────
data-service:
build:
context: ../../..
dockerfile: trading-platform/deploy/dockerfiles/data-service.Dockerfile
container_name: trading-data-service-dev
environment:
<<: *common-env
DATABASE_URL: postgresql+asyncpg://trading:trading_dev_password@postgres:5432/trading_db
REDIS_URL: redis://redis:6379/0
KAFKA_BOOTSTRAP_SERVERS: kafka:9092
LOG_LEVEL: debug
ports:
- "8001:8001"
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
kafka:
condition: service_healthy
restart: unless-stopped
networks:
- trading-network
execute-service:
build:
context: ../../..
dockerfile: trading-platform/deploy/dockerfiles/execute-service.Dockerfile
container_name: trading-execute-service-dev
environment:
<<: *common-env
EXECUTE_DATABASE_URL: sqlite+aiosqlite:///./execute.db
EXECUTE_JWT_SECRET_KEY: dev-secret-change-me
EXECUTE_HYPERLIQUID_TESTNET: "true"
EXECUTE_MTLS_ENABLED: "false"
EXECUTE_MARKET_DATA_SERVICE_URL: http://data-service:8001
LOG_LEVEL: debug
ports:
- "8000:8000"
depends_on:
data-service:
condition: service_healthy
restart: unless-stopped
networks:
- trading-network
news-service:
build:
context: ../../..
dockerfile: trading-platform/deploy/dockerfiles/news-service.Dockerfile
container_name: trading-news-service-dev
environment:
<<: *common-env
DATABASE_URL: postgresql+asyncpg://trading:trading_dev_password@postgres:5432/trading_db
KAFKA_BOOTSTRAP_SERVERS: kafka:9092
REDIS_URL: redis://redis:6379/1
LOG_LEVEL: debug
ports:
- "8002:8002"
depends_on:
postgres:
condition: service_healthy
kafka:
condition: service_healthy
restart: unless-stopped
networks:
- trading-network
api-gateway:
build:
context: ../../..
dockerfile: trading-platform/deploy/dockerfiles/api-gateway.Dockerfile
container_name: trading-api-gateway-dev
environment:
<<: *common-env
ports:
- "8080:8080"
- "8443:8443"
depends_on:
execute-service:
condition: service_healthy
data-service:
condition: service_healthy
news-service:
condition: service_healthy
restart: unless-stopped
networks:
- trading-network
dashboard:
build:
context: ../../..
dockerfile: trading-platform/deploy/dockerfiles/dashboard.Dockerfile
container_name: trading-dashboard-dev
environment:
NEXT_PUBLIC_API_URL: http://localhost:8080
NODE_ENV: development
ports:
- "3000:3000"
depends_on:
api-gateway:
condition: service_started
restart: unless-stopped
networks:
- trading-network
volumes:
postgres-data:
redis-data:
kafka-data:
networks:
trading-network:
driver: bridge

View file

@ -1,26 +0,0 @@
# =============================================================================
# API Gateway Dockerfile — Nginx-based reverse proxy with rate limiting
# =============================================================================
FROM nginx:1.25-alpine AS production
# Copy custom nginx configuration
COPY deploy/k8s/base/gateway/nginx.conf /etc/nginx/nginx.conf
COPY deploy/k8s/base/gateway/conf.d/ /etc/nginx/conf.d/
# Create required directories
RUN mkdir -p /etc/nginx/ssl \
/etc/nginx/conf.d \
/var/cache/nginx \
/var/run/nginx \
/var/log/nginx \
&& touch /var/run/nginx/nginx.pid
# Security: run as nginx user (already exists in alpine image)
USER nginx
EXPOSE 8080 8443
HEALTHCHECK --interval=15s --timeout=3s --start-period=10s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health || exit 1
CMD ["nginx", "-g", "daemon off;"]

View file

@ -1,40 +0,0 @@
# =============================================================================
# Dashboard Dockerfile — Next.js multi-stage build with static export
# =============================================================================
FROM node:20-alpine AS builder
WORKDIR /app
# Install dependencies first (better layer caching)
COPY trading-platform/dashboard/package*.json ./
RUN npm ci
# Copy source and build
COPY trading-platform/dashboard/ ./
RUN npm run build
# Production stage
FROM node:20-alpine AS production
# Security: non-root user
RUN addgroup -g 1001 -S nodejs && adduser -S nextjs -u 1001
WORKDIR /app
# Copy built output and package.json from builder
COPY --from=builder /app/package.json ./package.json
COPY --from=builder /app/.next/standalone ./
COPY --from=builder /app/.next/static ./.next/static
COPY --from=builder /app/public ./public
USER nextjs
EXPOSE 3000
ENV NODE_ENV=production
ENV PORT=3000
HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:3000/ || exit 1
CMD ["node", "server.js"]

View file

@ -1,32 +0,0 @@
# =============================================================================
# Data Service Dockerfile — Multi-stage build
# =============================================================================
FROM python:3.12-slim AS builder
WORKDIR /build
COPY trading-platform/data-service/pyproject.toml ./
RUN pip install --no-cache-dir --prefix=/install .
# Production stage
FROM python:3.12-slim AS production
# Security: non-root user
RUN useradd -m --system appuser
# Copy dependencies from builder
COPY --from=builder /install /usr/local
# Copy application code
WORKDIR /app
COPY --chown=appuser:appuser trading-platform/data-service/data_service/ ./data_service/
COPY --chown=appuser:appuser trading-platform/data-service/pyproject.toml ./
USER appuser
# Health check
HEALTHCHECK --interval=15s --timeout=5s --start-period=15s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8001/health')" || exit 1
EXPOSE 8001
CMD ["uvicorn", "data_service.app.main:app", "--host", "0.0.0.0", "--port", "8001"]

View file

@ -1,36 +0,0 @@
# =============================================================================
# Execute Service Dockerfile — Multi-stage build for minimal image
# =============================================================================
# Build stage: compile dependencies
FROM python:3.12-slim AS builder
WORKDIR /build
COPY trading-platform/execute-service/pyproject.toml ./
RUN pip install --no-cache-dir --prefix=/install .
# Production stage
FROM python:3.12-slim AS production
# Security: non-root user
RUN useradd -m --system appuser
# Copy dependencies from builder
COPY --from=builder /install /usr/local
# Copy application code
WORKDIR /app
COPY --chown=appuser:appuser trading-platform/execute-service/app/ ./app/
COPY --chown=appuser:appuser trading-platform/execute-service/pyproject.toml ./
# Create required directories with proper permissions
RUN mkdir -p /tmp /app/data && chown -R appuser:appuser /tmp /app/data
USER appuser
# Health check using Python (curl not in slim)
HEALTHCHECK --interval=15s --timeout=5s --start-period=15s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" || exit 1
EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

View file

@ -1,41 +0,0 @@
# =============================================================================
# News Service Dockerfile — Multi-stage build with NLTK data
# =============================================================================
FROM python:3.12-slim AS builder
WORKDIR /build
COPY trading-platform/news-service/requirements.txt ./
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
# Production stage
FROM python:3.12-slim AS production
# Install system dependencies
RUN apt-get update && \
apt-get install -y --no-install-recommends \
gcc libpq-dev && \
rm -rf /var/lib/apt/lists/*
# Install Python dependencies from builder
COPY --from=builder /install /usr/local
# Download NLTK data for textblob
RUN python -c "import nltk; nltk.download('punkt'); nltk.download('punkt_tab'); nltk.download('averaged_perceptron_tagger')"
# Security: non-root user
RUN useradd -m --system appuser
# Copy application code
WORKDIR /app
COPY --chown=appuser:appuser trading-platform/news-service/app/ ./app/
COPY --chown=appuser:appuser trading-platform/news-service/requirements.txt ./
USER appuser
# Health check
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8002/health')" || exit 1
EXPOSE 8002
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8002", "--workers", "4"]

View file

@ -1,34 +0,0 @@
# =============================================================================
# Trading Platform — Root Helm Chart
# =============================================================================
apiVersion: v2
name: trading-platform
description: Helm chart for the entire trading platform microservices
type: application
version: 0.1.0
appVersion: "0.1.0"
dependencies:
- name: api-gateway
version: "0.1.0"
repository: "file://../api-gateway"
- name: execute-service
version: "0.1.0"
repository: "file://../execute-service"
- name: data-service
version: "0.1.0"
repository: "file://../data-service"
- name: news-service
version: "0.1.0"
repository: "file://../news-service"
- name: dashboard
version: "0.1.0"
repository: "file://../dashboard"
- name: cert-manager
version: "1.14.0"
repository: https://charts.jetstack.io
condition: cert-manager.enabled
- name: ingress-nginx
version: "4.9.0"
repository: https://kubernetes.github.io/ingress-nginx
condition: ingress-nginx.enabled

View file

@ -1,60 +0,0 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "<SERVICE_NAME>.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
*/}}
{{- define "<SERVICE_NAME>.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "<SERVICE_NAME>.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "<SERVICE_NAME>.labels" -}}
helm.sh/chart: {{ include "<SERVICE_NAME>.chart" . }}
{{ include "<SERVICE_NAME>.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "<SERVICE_NAME>.selectorLabels" -}}
app.kubernetes.io/name: {{ include "<SERVICE_NAME>.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
Create the name of the service account to use
*/}}
{{- define "<SERVICE_NAME>.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "<SERVICE_NAME>.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}

View file

@ -1,6 +0,0 @@
apiVersion: v2
name: api-gateway
description: API Gateway — Nginx reverse proxy
type: application
version: 0.1.0
appVersion: "0.1.0"

View file

@ -1,61 +0,0 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "api-gateway.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
*/}}
{{- define "api-gateway.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "api-gateway.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "api-gateway.labels" -}}
helm.sh/chart: {{ include "api-gateway.chart" . }}
{{ include "api-gateway.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "api-gateway.selectorLabels" -}}
app.kubernetes.io/name: {{ include "api-gateway.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
Create the name of the service account to use
*/}}
{{- define "api-gateway.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "api-gateway.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}

View file

@ -1,57 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "api-gateway.fullname" . }}
labels:
{{- include "api-gateway.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "api-gateway.selectorLabels" . | nindent 6 }}
template:
metadata:
annotations:
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "api-gateway.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "api-gateway.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: 8080
protocol: TCP
livenessProbe:
{{- toYaml .Values.livenessProbe | nindent 12 }}
readinessProbe:
{{- toYaml .Values.readinessProbe | nindent 12 }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 12 }}
{{- end }}

View file

@ -1,32 +0,0 @@
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: {{ include "api-gateway.fullname" . }}
labels:
{{- include "api-gateway.labels" . | nindent 4 }}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ include "api-gateway.fullname" . }}
minReplicas: {{ .Values.autoscaling.minReplicas }}
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
metrics:
{{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
{{- end }}
{{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
{{- end }}
{{- end }}

View file

@ -1,61 +0,0 @@
{{- if .Values.ingress.enabled -}}
{{- $fullName := include "api-gateway.fullname" . -}}
{{- $svcPort := .Values.service.port -}}
{{- if and .Values.ingress.className (not (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion)) }}
{{- if not (hasKey .Values.ingress.annotations "kubernetes.io/ingress.class") }}
{{- $_ := set .Values.ingress.annotations "kubernetes.io/ingress.class" .Values.ingress.className}}
{{- end }}
{{- end }}
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1
{{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: extensions/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end }}
kind: Ingress
metadata:
name: {{ $fullName }}
labels:
{{- include "api-gateway.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- if and .Values.ingress.className (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion) }}
ingressClassName: {{ .Values.ingress.className }}
{{- end }}
{{- if .Values.ingress.tls }}
tls:
{{- range .Values.ingress.tls }}
- hosts:
{{- range .hosts }}
- {{ . | quote }}
{{- end }}
secretName: {{ .secretName }}
{{- end }}
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
- host: {{ .host | quote }}
http:
paths:
{{- range .paths }}
- path: {{ .path }}
{{- if and .pathType (semverCompare ">=1.18-0" $.Capabilities.KubeVersion.GitVersion) }}
pathType: {{ .pathType }}
{{- end }}
backend:
{{- if semverCompare ">=1.19-0" $.Capabilities.KubeVersion.GitVersion }}
service:
name: {{ $fullName }}
port:
number: {{ $svcPort }}
{{- else }}
serviceName: {{ $fullName }}
servicePort: {{ $svcPort }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}

View file

@ -1,29 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: {{ include "api-gateway.fullname" . }}-network-policy
labels:
{{- include "api-gateway.labels" . | nindent 4 }}
spec:
podSelector:
matchLabels:
{{- include "api-gateway.selectorLabels" . | nindent 6 }}
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: api-gateway
ports:
- port: http
protocol: TCP
egress:
# Allow DNS resolution
- to: []
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP

View file

@ -1,15 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "api-gateway.fullname" . }}
labels:
{{- include "api-gateway.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "api-gateway.selectorLabels" . | nindent 4 }}

View file

@ -1,12 +0,0 @@
{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "api-gateway.serviceAccountName" . }}
labels:
{{- include "api-gateway.labels" . | nindent 4 }}
{{- with .Values.serviceAccount.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}

View file

@ -1,80 +0,0 @@
# trading-platform/api-gateway Helm chart values
replicaCount: 2
image:
repository: trading-platform/api-gateway
pullPolicy: IfNotPresent
tag: ""
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
create: true
annotations: {}
name: ""
podAnnotations: {}
securityContext:
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
service:
type: ClusterIP
port: 8080
ingress:
enabled: true
className: nginx
annotations: {}
hosts:
- host: api.trading.example.com
paths:
- path: /
pathType: Prefix
tls: []
resources:
limits:
cpu: "500m"
memory: 256Mi
requests:
cpu: "250m"
memory: 128Mi
autoscaling:
enabled: true
minReplicas: 2
maxReplicas: 10
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 15
periodSeconds: 15
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
nodeSelector: {}
tolerations: []
affinity: {}

View file

@ -1,6 +0,0 @@
apiVersion: v2
name: dashboard
description: Dashboard frontend
type: application
version: 0.1.0
appVersion: "0.1.0"

View file

@ -1,61 +0,0 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "dashboard.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
*/}}
{{- define "dashboard.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "dashboard.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "dashboard.labels" -}}
helm.sh/chart: {{ include "dashboard.chart" . }}
{{ include "dashboard.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "dashboard.selectorLabels" -}}
app.kubernetes.io/name: {{ include "dashboard.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
Create the name of the service account to use
*/}}
{{- define "dashboard.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "dashboard.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}

View file

@ -1,57 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "dashboard.fullname" . }}
labels:
{{- include "dashboard.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "dashboard.selectorLabels" . | nindent 6 }}
template:
metadata:
annotations:
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "dashboard.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "dashboard.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: 3000
protocol: TCP
livenessProbe:
{{- toYaml .Values.livenessProbe | nindent 12 }}
readinessProbe:
{{- toYaml .Values.readinessProbe | nindent 12 }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 12 }}
{{- end }}

View file

@ -1,32 +0,0 @@
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: {{ include "dashboard.fullname" . }}
labels:
{{- include "dashboard.labels" . | nindent 4 }}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ include "dashboard.fullname" . }}
minReplicas: {{ .Values.autoscaling.minReplicas }}
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
metrics:
{{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
{{- end }}
{{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
{{- end }}
{{- end }}

View file

@ -1,29 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: {{ include "dashboard.fullname" . }}-network-policy
labels:
{{- include "dashboard.labels" . | nindent 4 }}
spec:
podSelector:
matchLabels:
{{- include "dashboard.selectorLabels" . | nindent 6 }}
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: api-gateway
ports:
- port: http
protocol: TCP
egress:
# Allow DNS resolution
- to: []
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP

View file

@ -1,15 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "dashboard.fullname" . }}
labels:
{{- include "dashboard.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "dashboard.selectorLabels" . | nindent 4 }}

View file

@ -1,12 +0,0 @@
{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "dashboard.serviceAccountName" . }}
labels:
{{- include "dashboard.labels" . | nindent 4 }}
{{- with .Values.serviceAccount.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}

View file

@ -1,80 +0,0 @@
# trading-platform/dashboard Helm chart values
replicaCount: 2
image:
repository: trading-platform/dashboard
pullPolicy: IfNotPresent
tag: ""
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
create: true
annotations: {}
name: ""
podAnnotations: {}
securityContext:
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
service:
type: ClusterIP
port: 3000
ingress:
enabled: false
className: nginx
annotations: {}
hosts:
- host: api.trading.example.com
paths:
- path: /
pathType: Prefix
tls: []
resources:
limits:
cpu: "500m"
memory: 512Mi
requests:
cpu: "250m"
memory: 256Mi
autoscaling:
enabled: true
minReplicas: 2
maxReplicas: 10
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
livenessProbe:
httpGet:
path: /
port: http
initialDelaySeconds: 15
periodSeconds: 15
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /
port: http
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
nodeSelector: {}
tolerations: []
affinity: {}

View file

@ -1,6 +0,0 @@
apiVersion: v2
name: data-service
description: Trading data service
type: application
version: 0.1.0
appVersion: "0.1.0"

View file

@ -1,61 +0,0 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "data-service.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
*/}}
{{- define "data-service.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "data-service.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "data-service.labels" -}}
helm.sh/chart: {{ include "data-service.chart" . }}
{{ include "data-service.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "data-service.selectorLabels" -}}
app.kubernetes.io/name: {{ include "data-service.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
Create the name of the service account to use
*/}}
{{- define "data-service.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "data-service.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}

View file

@ -1,57 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "data-service.fullname" . }}
labels:
{{- include "data-service.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "data-service.selectorLabels" . | nindent 6 }}
template:
metadata:
annotations:
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "data-service.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "data-service.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: 8001
protocol: TCP
livenessProbe:
{{- toYaml .Values.livenessProbe | nindent 12 }}
readinessProbe:
{{- toYaml .Values.readinessProbe | nindent 12 }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 12 }}
{{- end }}

View file

@ -1,32 +0,0 @@
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: {{ include "data-service.fullname" . }}
labels:
{{- include "data-service.labels" . | nindent 4 }}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ include "data-service.fullname" . }}
minReplicas: {{ .Values.autoscaling.minReplicas }}
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
metrics:
{{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
{{- end }}
{{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
{{- end }}
{{- end }}

View file

@ -1,29 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: {{ include "data-service.fullname" . }}-network-policy
labels:
{{- include "data-service.labels" . | nindent 4 }}
spec:
podSelector:
matchLabels:
{{- include "data-service.selectorLabels" . | nindent 6 }}
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: api-gateway
ports:
- port: http
protocol: TCP
egress:
# Allow DNS resolution
- to: []
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP

View file

@ -1,15 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "data-service.fullname" . }}
labels:
{{- include "data-service.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "data-service.selectorLabels" . | nindent 4 }}

View file

@ -1,12 +0,0 @@
{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "data-service.serviceAccountName" . }}
labels:
{{- include "data-service.labels" . | nindent 4 }}
{{- with .Values.serviceAccount.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}

View file

@ -1,80 +0,0 @@
# trading-platform/data-service Helm chart values
replicaCount: 2
image:
repository: trading-platform/data-service
pullPolicy: IfNotPresent
tag: ""
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
create: true
annotations: {}
name: ""
podAnnotations: {}
securityContext:
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
service:
type: ClusterIP
port: 8001
ingress:
enabled: false
className: nginx
annotations: {}
hosts:
- host: api.trading.example.com
paths:
- path: /
pathType: Prefix
tls: []
resources:
limits:
cpu: "1000m"
memory: 1Gi
requests:
cpu: "500m"
memory: 512Mi
autoscaling:
enabled: true
minReplicas: 2
maxReplicas: 10
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 15
periodSeconds: 15
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
nodeSelector: {}
tolerations: []
affinity: {}

View file

@ -1,6 +0,0 @@
apiVersion: v2
name: execute-service
description: Trading execution service
type: application
version: 0.1.0
appVersion: "0.1.0"

View file

@ -1,61 +0,0 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "execute-service.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
*/}}
{{- define "execute-service.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "execute-service.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "execute-service.labels" -}}
helm.sh/chart: {{ include "execute-service.chart" . }}
{{ include "execute-service.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "execute-service.selectorLabels" -}}
app.kubernetes.io/name: {{ include "execute-service.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
Create the name of the service account to use
*/}}
{{- define "execute-service.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "execute-service.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}

View file

@ -1,57 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "execute-service.fullname" . }}
labels:
{{- include "execute-service.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "execute-service.selectorLabels" . | nindent 6 }}
template:
metadata:
annotations:
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "execute-service.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "execute-service.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: 8000
protocol: TCP
livenessProbe:
{{- toYaml .Values.livenessProbe | nindent 12 }}
readinessProbe:
{{- toYaml .Values.readinessProbe | nindent 12 }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 12 }}
{{- end }}

View file

@ -1,32 +0,0 @@
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: {{ include "execute-service.fullname" . }}
labels:
{{- include "execute-service.labels" . | nindent 4 }}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ include "execute-service.fullname" . }}
minReplicas: {{ .Values.autoscaling.minReplicas }}
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
metrics:
{{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
{{- end }}
{{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
{{- end }}
{{- end }}

View file

@ -1,29 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: {{ include "execute-service.fullname" . }}-network-policy
labels:
{{- include "execute-service.labels" . | nindent 4 }}
spec:
podSelector:
matchLabels:
{{- include "execute-service.selectorLabels" . | nindent 6 }}
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: api-gateway
ports:
- port: http
protocol: TCP
egress:
# Allow DNS resolution
- to: []
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP

View file

@ -1,15 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "execute-service.fullname" . }}
labels:
{{- include "execute-service.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "execute-service.selectorLabels" . | nindent 4 }}

View file

@ -1,12 +0,0 @@
{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "execute-service.serviceAccountName" . }}
labels:
{{- include "execute-service.labels" . | nindent 4 }}
{{- with .Values.serviceAccount.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}

View file

@ -1,80 +0,0 @@
# trading-platform/execute-service Helm chart values
replicaCount: 2
image:
repository: trading-platform/execute-service
pullPolicy: IfNotPresent
tag: ""
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
create: true
annotations: {}
name: ""
podAnnotations: {}
securityContext:
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
service:
type: ClusterIP
port: 8000
ingress:
enabled: false
className: nginx
annotations: {}
hosts:
- host: api.trading.example.com
paths:
- path: /
pathType: Prefix
tls: []
resources:
limits:
cpu: "500m"
memory: 512Mi
requests:
cpu: "250m"
memory: 256Mi
autoscaling:
enabled: true
minReplicas: 2
maxReplicas: 10
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 15
periodSeconds: 15
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
nodeSelector: {}
tolerations: []
affinity: {}

View file

@ -1,6 +0,0 @@
apiVersion: v2
name: news-service
description: News analysis service
type: application
version: 0.1.0
appVersion: "0.1.0"

View file

@ -1,61 +0,0 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "news-service.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
*/}}
{{- define "news-service.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "news-service.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "news-service.labels" -}}
helm.sh/chart: {{ include "news-service.chart" . }}
{{ include "news-service.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "news-service.selectorLabels" -}}
app.kubernetes.io/name: {{ include "news-service.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
Create the name of the service account to use
*/}}
{{- define "news-service.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "news-service.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}

View file

@ -1,57 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "news-service.fullname" . }}
labels:
{{- include "news-service.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "news-service.selectorLabels" . | nindent 6 }}
template:
metadata:
annotations:
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "news-service.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "news-service.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: 8002
protocol: TCP
livenessProbe:
{{- toYaml .Values.livenessProbe | nindent 12 }}
readinessProbe:
{{- toYaml .Values.readinessProbe | nindent 12 }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 12 }}
{{- end }}

View file

@ -1,32 +0,0 @@
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: {{ include "news-service.fullname" . }}
labels:
{{- include "news-service.labels" . | nindent 4 }}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ include "news-service.fullname" . }}
minReplicas: {{ .Values.autoscaling.minReplicas }}
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
metrics:
{{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
{{- end }}
{{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
{{- end }}
{{- end }}

View file

@ -1,29 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: {{ include "news-service.fullname" . }}-network-policy
labels:
{{- include "news-service.labels" . | nindent 4 }}
spec:
podSelector:
matchLabels:
{{- include "news-service.selectorLabels" . | nindent 6 }}
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: api-gateway
ports:
- port: http
protocol: TCP
egress:
# Allow DNS resolution
- to: []
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP

View file

@ -1,15 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "news-service.fullname" . }}
labels:
{{- include "news-service.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "news-service.selectorLabels" . | nindent 4 }}

Some files were not shown because too many files have changed in this diff Show more