Compare commits
14 commits
2c00a5ab1c
...
4f235b52f4
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4f235b52f4 | ||
|
|
fe4fa1f34d | ||
|
|
eb27a51523 | ||
|
|
88db49b8b5 | ||
|
|
87632b2247 | ||
|
|
d7beb9f6ab | ||
|
|
2b54a18724 | ||
|
|
12766f293c | ||
|
|
18764cf6b5 | ||
|
|
79e0d98fbe | ||
|
|
f49e3c9037 | ||
|
|
37cf8d1276 | ||
|
|
789903c835 | ||
|
|
9b949ed0bd |
180 changed files with 213 additions and 11150 deletions
|
|
@ -1,203 +0,0 @@
|
|||
# Repository Reorganization Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Enforce the rule: **gcloud-lab = Kubernetes manifests only. No application code.**
|
||||
|
||||
Application code (source, Dockerfiles, CI/CD workflows, Helm charts, deployment scripts) belongs in `hermes-projects/`. gcloud-lab should contain only K8s manifests, Terraform infrastructure modules, cluster configs, and infra controller configs.
|
||||
|
||||
---
|
||||
|
||||
## Current State
|
||||
|
||||
### gcloud-lab/ — Full Directory Audit
|
||||
|
||||
```
|
||||
gcloud-lab/
|
||||
├── apps/
|
||||
│ ├── base/
|
||||
│ │ ├── customer1/ [KEEP] K8s manifests (deployments, services, configmaps, secrets)
|
||||
│ │ ├── monitoring/ [KEEP] K8s manifests (dashboards)
|
||||
│ │ └── osint-dashboard/ [KEEP] Helm chart (templates, values.yaml, Chart.yaml)
|
||||
│ ├── staging/
|
||||
│ │ ├── customer1/ [KEEP] K8s overlay (kustomization.yaml)
|
||||
│ │ └── osint-dashboard/ [KEEP] K8s overlay (kustomization.yaml)
|
||||
│ └── vwap-monitor/ [MOVE] App code (Dockerfile, app/, deploy/)
|
||||
├── clusters/ [KEEP] Cluster configs (devops-lab/*.yaml, flux-system/)
|
||||
├── infrastructure/ [KEEP] Infra controllers, gatewayapi, gpus, tailnet
|
||||
├── misc/ [KEEP] Terraform snippets + K8s YAML
|
||||
├── modules/ [KEEP] Terraform modules (gke.tf, nodepool.tf, etc.)
|
||||
├── scripts/ [KEEP] Setup scripts
|
||||
├── .github/workflows/
|
||||
│ ├── osint-dashboard-infra.yml [KEEP] Infra deployment workflow
|
||||
│ └── trade-dashboard.yml [MOVE] CI for trade-dashboard app → hermes-projects/
|
||||
├── .devcontainer.json [KEEP] Dev environment config
|
||||
├── .sops.yaml [KEEP] SOPS encryption config
|
||||
├── .terraform.lock.hcl [KEEP] Terraform lock
|
||||
├── .gitignore [KEEP] Git ignore rules
|
||||
├── mise.toml [KEEP] Tool version management
|
||||
├── README.md [KEEP] (will be updated)
|
||||
├── infra-tailnet.yaml [KEEP] Tailscale infra config
|
||||
├── tailscale-0auth.yaml [KEEP] Tailscale config
|
||||
├── rays-new-deployment.yaml [REVIEW] Orphan K8s deployment YAML — move to apps/base/
|
||||
├── trade-dashboard/ [MOVE] Full FastAPI app → hermes-projects/trade-dashboard/
|
||||
├── trading-platform/ [MOVE] Duplicate/deploy configs → hermes-projects/trading-platform/
|
||||
├── trading-scripts/ [MOVE] Application code → hermes-projects/trading-scripts/
|
||||
├── analyses/ [REMOVE] Research artifacts (not code, not infra)
|
||||
└── plans/ [REMOVE] Planning docs (not code, not infra)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Items to Move / Remove
|
||||
|
||||
### 1. `trade-dashboard/` → hermes-projects/trade-dashboard/
|
||||
|
||||
**What it is:** Full FastAPI application (not K8s manifests)
|
||||
- `Dockerfile` — build config for the app
|
||||
- `app/` — Python source code (main.py, models.py, schemas.py, database.py, requirements.txt, static/)
|
||||
- `alembic/` — database migration scripts (env.py, versions/)
|
||||
- `alembic.ini` — alembic config
|
||||
|
||||
**Also move:** `.github/workflows/trade-dashboard.yml` (CI workflow for this app)
|
||||
|
||||
**Already in gcloud-lab:** `apps/base/customer1/trade-dashboard/` — these are the K8s manifests for trade-dashboard (deployment.yaml, service.yaml, configmap.yaml, kustomization.yaml). **KEEP these** — they belong here.
|
||||
|
||||
### 2. `trading-platform/` → hermes-projects/trading-platform/
|
||||
|
||||
**What it is:** Duplicate/alternative deployment configs that overlap with hermes-projects/trading-platform/
|
||||
|
||||
Contents:
|
||||
- `.github/workflows/` — 3 CI/CD workflows (build-push.yml, build-test.yml, deploy.yml)
|
||||
- `README.md` — project readme
|
||||
- `deploy/` — deployment configs:
|
||||
- `ci-cd/` — additional CI workflows
|
||||
- `docker-compose/` — docker-compose.dev.yml
|
||||
- `dockerfiles/` — Dockerfiles (api-gateway, dashboard, data-service, execute-service, news-service)
|
||||
- `helm/` — Helm charts (api-gateway, dashboard, data-service, execute-service, news-service)
|
||||
- `k8s/` — raw K8s manifests (deployments, services, hpa, cert-manager, ingress)
|
||||
- `mtls/` — mTLS README
|
||||
- `scripts/` — deploy.sh, generate-mtls-certs.sh
|
||||
- `dockerfiles/` — Dockerfiles (dashboard, data-service, execute-service, news-service)
|
||||
- `helm/` — Helm chart with templates (trading-platform chart, values.yaml, secrets)
|
||||
|
||||
**Already in gcloud-lab:** `apps/base/customer1/trading-platform/` — these are the K8s manifests. **KEEP these** — they belong here.
|
||||
|
||||
**Already in hermes-projects:** `hermes-projects/trading-platform/` — source code exists here (dashboard, data-service, execute-service, news-service, data_infrastructure). The gcloud-lab trading-platform/ deploy/dockerfiles/helm content should be MERGED into hermes-projects/trading-platform/.
|
||||
|
||||
**Decision needed:** The `trading-platform/` in gcloud-lab has BOTH deploy configs (dockerfiles, helm, k8s manifests) AND CI workflows. The K8s manifests in `deploy/k8s/base/` are similar but NOT identical to what's in `apps/base/customer1/trading-platform/`. Need to decide which is authoritative.
|
||||
|
||||
### 3. `trading-scripts/` → hermes-projects/trading-scripts/
|
||||
|
||||
**What it is:** Application code (Python trading scripts)
|
||||
- `market_data.py` — market data script
|
||||
- `orb-monitor/` — monitoring tool (monitor.py, config.yaml)
|
||||
- `README.md`, `ROADMAP.md` — documentation
|
||||
|
||||
### 4. `apps/vwap-monitor/` → hermes-projects/vwap-monitor/
|
||||
|
||||
**What it is:** Application code with a Dockerfile
|
||||
- `Dockerfile` — build config
|
||||
- `app/` — source code (scanner.py, requirements.txt)
|
||||
- `deploy/` — deployment config (deployment.yaml, kustomization.yaml, secret.yaml, config.env)
|
||||
|
||||
**Note:** The `deploy/` subdirectory contains K8s manifests. These should be moved BACK into gcloud-lab as `apps/base/customer1/vwap-monitor/`. The app code (Dockerfile + app/) goes to hermes-projects.
|
||||
|
||||
### 5. `analyses/` → REMOVE from gcloud-lab
|
||||
|
||||
**What it is:** Research/analysis markdown documents
|
||||
- `telegram-webhook-container-analysis.md`
|
||||
- `telegram-webhook-failure-analysis.md`
|
||||
|
||||
These are one-time research artifacts, not infrastructure config. Remove from gcloud-lab entirely.
|
||||
|
||||
### 6. `plans/` → REMOVE from gcloud-lab
|
||||
|
||||
**What it is:** Planning/strategy markdown documents
|
||||
- `2026-04-25-openclaw-brain-v1.1.md`
|
||||
- `AI_ARCHITECTURE.md`
|
||||
- `models-to-try.md`
|
||||
|
||||
These are planning docs, not infrastructure config. Remove from gcloud-lab entirely.
|
||||
|
||||
### 7. `rays-new-deployment.yaml` → REVIEW
|
||||
|
||||
**What it is:** A standalone K8s deployment YAML at repo root.
|
||||
|
||||
**Action:** Move to `apps/base/customer1/hermes-agent/` (appears related to hermes-agent/rays deployment based on filename). Already similar files exist in that directory.
|
||||
|
||||
---
|
||||
|
||||
## Proposed Migration Plan (Ordered by PR)
|
||||
|
||||
### PR 1: This Plan (docs only)
|
||||
- Add `MIGRATION_PLAN.md` (this file)
|
||||
- Update `README.md` to document the new structure
|
||||
|
||||
### PR 2: Remove planning/research docs
|
||||
- Delete `analyses/` directory
|
||||
- Delete `plans/` directory
|
||||
- Low risk, no dependencies
|
||||
|
||||
### PR 3: Move trade-dashboard app to hermes-projects
|
||||
- Move `trade-dashboard/` → hermes-projects/trade-dashboard/
|
||||
- Move `.github/workflows/trade-dashboard.yml` → hermes-projects/.github/workflows/
|
||||
- K8s manifests in `apps/base/customer1/trade-dashboard/` stay in place
|
||||
- Verify image references in K8s manifests still point to correct registry
|
||||
|
||||
### PR 4: Move trading-platform deploy configs to hermes-projects
|
||||
- Move `trading-platform/` → merge with hermes-projects/trading-platform/
|
||||
- CI workflows → hermes-projects/trading-platform/.github/workflows/
|
||||
- Dockerfiles → hermes-projects/trading-platform/dockerfiles/
|
||||
- Helm charts → hermes-projects/trading-platform/helm/
|
||||
- K8s manifests from `trading-platform/deploy/k8s/` → reconcile with `apps/base/customer1/trading-platform/`
|
||||
- **Decision needed:** Which K8s manifests are authoritative? The ones in gcloud-lab/apps/ or trading-platform/deploy/k8s/?
|
||||
|
||||
### PR 5: Move trading-scripts to hermes-projects
|
||||
- Move `trading-scripts/` → hermes-projects/trading-scripts/
|
||||
- Simple move, no K8s manifest reconciliation needed
|
||||
|
||||
### PR 6: Split vwap-monitor (app → hermes-projects, K8s → gcloud-lab)
|
||||
- Move `apps/vwap-monitor/app/` + `apps/vwap-monitor/Dockerfile` → hermes-projects/vwap-monitor/
|
||||
- Move `apps/vwap-monitor/deploy/` K8s manifests → `apps/base/customer1/vwap-monitor/`
|
||||
- Update image references in K8s manifests
|
||||
|
||||
---
|
||||
|
||||
## Items That Stay in gcloud-lab (No Changes)
|
||||
|
||||
| Path | Reason |
|
||||
|------|--------|
|
||||
| `apps/base/customer1/` | K8s manifests (kustomize structure) |
|
||||
| `apps/base/monitoring/` | K8s manifests (dashboards) |
|
||||
| `apps/base/osint-dashboard/` | Helm chart for infra |
|
||||
| `apps/staging/` | K8s overlays |
|
||||
| `clusters/` | Cluster configs, flux-system |
|
||||
| `infrastructure/` | Controllers, gatewayapi, gpus, tailnet |
|
||||
| `misc/` | Terraform snippets + K8s YAML |
|
||||
| `modules/` | Terraform modules |
|
||||
| `scripts/` | Setup scripts |
|
||||
| Root config files | .sops.yaml, .devcontainer.json, mise.toml, .gitignore, .terraform.lock.hcl |
|
||||
| `infra-tailnet.yaml` | Tailscale infra config |
|
||||
| `tailscale-0auth.yaml` | Tailscale config |
|
||||
|
||||
---
|
||||
|
||||
## K8s Manifest Reference Check
|
||||
|
||||
After moves, verify these image references still resolve:
|
||||
|
||||
| K8s Manifest | Image Reference |
|
||||
|--------------|----------------|
|
||||
| `apps/base/customer1/trade-dashboard/deployment.yaml` | Check image tag matches hermes-projects build |
|
||||
| `apps/base/customer1/trading-platform/*/deployment.yaml` | Check image tags match hermes-projects build |
|
||||
| `apps/base/customer1/hermes-agent/deployment.yaml` | N/A (already correct) |
|
||||
| `apps/base/customer1/siriusdevops-site/deployment.yaml` | N/A (already correct) |
|
||||
|
||||
---
|
||||
|
||||
## Decisions Needed Before Proceeding
|
||||
|
||||
1. **trading-platform K8s manifest authority:** `trading-platform/deploy/k8s/base/` vs `apps/base/customer1/trading-platform/` — which is the source of truth?
|
||||
2. **analyses/ and plans/:** Delete entirely, or archive somewhere else?
|
||||
3. **rays-new-deployment.yaml:** Move to `apps/base/customer1/hermes-agent/` or delete?
|
||||
688
README.md
688
README.md
|
|
@ -1,524 +1,246 @@
|
|||
# GCloud-Lab DevOps Infrastructure
|
||||
# gcloud-lab
|
||||
|
||||
A production-grade cloud-native infrastructure laboratory demonstrating GitOps, multi-tenant AI agent hosting, and automated security pipelines — all run by a single DevOps engineer on Google Cloud Platform. Trusted by builders who ship.
|
||||
GitOps + Terraform source of truth for a **GKE lab** I designed, ran, and then **shut down** once the GPU bill stopped being worth it.
|
||||
|
||||
## Table of Contents
|
||||
This is not a live cluster. It is the manifests, node-pool definitions, and GitOps wiring from a real environment that served vLLM inference, CNPG databases, and a handful of in-cluster apps. The same cost model that made the GPU pools scale to zero is why the whole footprint went to zero.
|
||||
|
||||
- [Project Overview](#project-overview)
|
||||
- [Architecture](#architecture)
|
||||
- [DevOps Tools & Technologies](#devops-tools--technologies)
|
||||
- [Monitoring](#monitoring)
|
||||
- [Project Structure](#project-structure)
|
||||
- [Infrastructure Components](#infrastructure-components)
|
||||
- [Applications](#applications)
|
||||
- [Getting Started](#getting-started)
|
||||
- [Security](#security)
|
||||
- [Cost Optimization](#cost-optimization)
|
||||
- [License](#license)
|
||||
**Canonical copy:** [forgejo.siriusdevops.com/sirius/gcloud-lab](https://forgejo.siriusdevops.com/sirius/gcloud-lab)
|
||||
|
||||
---
|
||||
|
||||
## Project Overview
|
||||
## What this is evidence of
|
||||
|
||||
This repository is the single source of truth for a multi-application cloud platform running on GKE. Every deployment, database, and network policy flows through Git via Flux CD. What lives here:
|
||||
If you are reading this as a hiring screen, start here. Every claim below maps to a file in this repo.
|
||||
|
||||
1. **AgentForge** — Private multi-tenant AI agent workspace with dual-tier vLLM inference (L4 dispatcher + RTX 6000 deep thinker) and isolated CNPG databases per tenant.
|
||||
2. **Multi-Profile AI Agent Team** — Six specialist AI profiles (backend-dev, frontend-dev, researcher, outreach, quant, sec-ops) orchestrated through a shared Kanban board with automated audit-to-fix pipelines.
|
||||
3. **Waitlist API** — FastAPI landing page backend with idempotent signups, async PostgreSQL, and Telegram fire-and-forget notifications.
|
||||
4. **Autonomous News Quant Pipeline** — 371 global feed scraper with DeepSeek-R1 analysis generating actionable futures trading signals.
|
||||
5. **N8N Workflow Automation** — Self-hosted workflow engine with dedicated CNPG PostgreSQL.
|
||||
6. **Local Business Web Deployment Pipeline** — Automated K8s manifest generation for small business websites with cross-namespace HTTPRoute routing.
|
||||
| Claim | Where to look |
|
||||
| --- | --- |
|
||||
| GKE cluster, custom VPC, dual-stack, Cilium datapath | [`modules/gke.tf`](modules/gke.tf), [`modules/vpc.tf`](modules/vpc.tf) |
|
||||
| CPU + three GPU node pools (L4, RTX PRO 6000, A100 80GB), all SPOT except CPU | [`modules/nodepool.tf`](modules/nodepool.tf), [`modules/nodepool-gpu.tf`](modules/nodepool-gpu.tf), [`modules/pro6000-nodepool.tf`](modules/pro6000-nodepool.tf), [`modules/a100-nodepool.tf`](modules/a100-nodepool.tf) |
|
||||
| Flux CD applies the tree; SOPS decrypts secrets in-cluster | [`clusters/devops-lab/`](clusters/devops-lab/), [`.sops.yaml`](.sops.yaml) |
|
||||
| Production-style **vLLM** OpenAI-compatible servers (not Ollama) | [`infrastructure/gpus/base/vllm-servers/`](infrastructure/gpus/base/vllm-servers/) |
|
||||
| KEDA HTTP scale-to-zero on the expensive GPUs | [`infrastructure/gpus/base/keda-gpu-scaling/`](infrastructure/gpus/base/keda-gpu-scaling/) |
|
||||
| CloudNative-PG operator + per-app Postgres | [`infrastructure/controllers/base/cnpg/`](infrastructure/controllers/base/cnpg/), `apps/base/customer1/*-db/` |
|
||||
| Gateway API (not legacy Ingress) + Tailscale for internals | [`infrastructure/gatewayapi/`](infrastructure/gatewayapi/), [`infrastructure/tailnet/`](infrastructure/tailnet/) |
|
||||
| Workload NetworkPolicies (trading stack) | [`apps/base/customer1/trading-platform/network-policies/`](apps/base/customer1/trading-platform/network-policies/) |
|
||||
|
||||
I run 24/7 infrastructure now on a Raspberry Pi (Forgejo, Cloudflare tunnel, containerized sites). This repo is the cloud chapter that came before that.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
┌──────────────────────────────────────────────────────────────────────────────┐
|
||||
│ Google Cloud Platform │
|
||||
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ GKE Cluster (devops-lab-cluster) │ │
|
||||
│ │ │ │
|
||||
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │
|
||||
│ │ │ Standard │ │ L4 GPU Pool │ │ RTX 6000 GPU │ │ │
|
||||
│ │ │ Node Pool │ │ (SPOT L4) │ │ (SPOT RTX6K) │ │ │
|
||||
│ │ │ e2-std-2 │ │ 1 node (24/7)│ │ 0-1 nodes │ │ │
|
||||
│ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │
|
||||
│ │ │ │
|
||||
│ │ ┌──────────────────────────────────────────────────────────────────┐ │ │
|
||||
│ │ │ Cilium CNI + Hubble + NetworkPolicy │ │ │
|
||||
│ │ └──────────────────────────────────────────────────────────────────┘ │ │
|
||||
│ │ │ │
|
||||
│ │ ┌──────────────────────────────────────────────────────────────────┐ │ │
|
||||
│ │ │ Kubernetes Gateway API — external-http-gateway │ │ │
|
||||
│ │ │ HTTPRoute PathPrefix → AgentForge / Waitlist / Apps │ │ │
|
||||
│ │ └──────────────────────────────────────────────────────────────────┘ │ │
|
||||
│ │ │ │
|
||||
│ │ ┌───────────────────────────────┐ ┌───────────────────────────────┐ │ │
|
||||
│ │ │ customer1 namespace │ │ agent-forge namespace │ │ │
|
||||
│ │ │ - AgentForge (PAaaS) │ │ - Tenant-specific Hermes Agent │ │ │
|
||||
│ │ │ - Dual-tier vLLM │ │ - Isolated CNPG databases │ │ │
|
||||
│ │ │ L4 Dispatcher (24/7) │ │ - Qwen 3.6 27B Abliterated │ │ │
|
||||
│ │ │ RTX 6000 Deep Thinker (KEDA) │ │ - KEDA scale-to-zero │ │ │
|
||||
│ │ │ - Waitlist API (FastAPI) │ │ │ │ │
|
||||
│ │ │ - News Bot Pipeline │ │ ┌───────────────────────────┐ │ │ │
|
||||
│ │ │ - Landing Page │ │ │ sec-ops audit agent │ │ │
|
||||
│ │ │ - CNPG PostgreSQL Cluster │ │ │ Automated vuln scanning │ │ │
|
||||
│ │ └───────────────────────────────┘ │ │ → backend-dev auto-fix │ │ │
|
||||
│ │ │ └───────────────────────────┘ │ │ │
|
||||
│ │ ┌───────────────────────────────┐ └───────────────────────────────┘ │ │
|
||||
│ │ │ local-business namespaces │ │ │
|
||||
│ │ │ - nginx + ConfigMap per biz │ ┌───────────────────────────────┐ │ │
|
||||
│ │ │ - Cross-ns HTTPRoute refs │ │ monitoring namespace │ │ │
|
||||
│ │ └───────────────────────────────┘ │ - Prometheus + Grafana │ │ │
|
||||
│ │ │ - Tailscale-only access │ │ │
|
||||
│ │ ┌───────────────────────────────┐ │ - No public ingress │ │ │
|
||||
│ │ │ kanban namespace │ └───────────────────────────────┘ │ │
|
||||
│ │ │ - Hermes Agent Orchestrator │ │ │
|
||||
│ │ │ - 6 Specialist Profiles │ ┌───────────────────────────────┐ │ │
|
||||
│ │ │ - Isolated hermes-pgdb │ │ n8n namespace │ │ │
|
||||
│ │ └───────────────────────────────┘ │ - Workflow automation │ │ │
|
||||
│ │ │ - Dedicated PostgreSQL │ │ │
|
||||
│ │ └───────────────────────────────┘ │ │
|
||||
│ └────────────────────────────────────────────────────────────────────────┘ │
|
||||
└──────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
Git (this repo, branch master)
|
||||
│
|
||||
Flux source-controller (1m)
|
||||
│
|
||||
┌─────────────────┼──────────────────┐
|
||||
▼ ▼ ▼
|
||||
infra-controllers infra-gpus apps/staging
|
||||
CNPG / KEDA vLLM + KEDA customer1 overlay
|
||||
cert-manager HTTPScaledObject (kustomize)
|
||||
Tailscale L4 / RTX6000 / A100
|
||||
kube-prometheus
|
||||
│ │ │
|
||||
└──────────── GKE us-central1-a ─────┘
|
||||
│
|
||||
┌───────────────┬────────────┼────────────┬──────────────┐
|
||||
▼ ▼ ▼ ▼ ▼
|
||||
e2-standard-2 g2-standard-8 g4-standard-48 a2-ultragpu-1g
|
||||
CPU pool L4 SPOT RTX PRO 6000 A100 80GB SPOT
|
||||
1–5 nodes 1 node SPOT 0–1 SPOT 0–1
|
||||
(always on) KEDA 0↔1 KEDA 0↔1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## DevOps Tools & Technologies
|
||||
|
||||
### Infrastructure as Code (IaC)
|
||||
|
||||
| Tool | Version | Purpose |
|
||||
|------|---------|---------|
|
||||
| **Terraform** | 1.7+ | Infrastructure provisioning for GCP resources |
|
||||
| **Google Provider** | 7.14.1 | Terraform provider for GCP |
|
||||
| **Helm Provider** | Latest | Terraform provider for Helm charts |
|
||||
| **Flux Provider** | 1.7.6 | Terraform provider for Flux bootstrap |
|
||||
|
||||
### Container Orchestration & Networking
|
||||
|
||||
| Tool | Version | Purpose |
|
||||
|------|---------|---------|
|
||||
| **Google Kubernetes Engine (GKE)** | Latest | Managed Kubernetes cluster |
|
||||
| **Cilium** | 1.18.5 | CNI plugin with eBPF-based networking |
|
||||
| **Hubble** | 1.18.5 | Network observability and monitoring |
|
||||
| **Kubernetes Gateway API** | v1 | Ingress routing and traffic management |
|
||||
|
||||
### GitOps & Configuration Management
|
||||
|
||||
| Tool | Version | Purpose |
|
||||
|------|---------|---------|
|
||||
| **Flux CD** | 1.7.6 | GitOps continuous delivery |
|
||||
| **Kustomize** | v1beta1 | Kubernetes manifest customization |
|
||||
| **Helm** | 3+ | Kubernetes package manager |
|
||||
| **SOPS** | Latest | Secrets encryption in Git |
|
||||
| **Age** | Latest | Modern encryption for SOPS |
|
||||
|
||||
### Database
|
||||
|
||||
| Tool | Version | Purpose |
|
||||
|------|---------|---------|
|
||||
| **CloudNative PG** | 0.26.1 | PostgreSQL Kubernetes operator |
|
||||
| **PostgreSQL** | 15.2 | Relational database (multi-cluster fleet) |
|
||||
|
||||
### AI/ML Infrastructure
|
||||
|
||||
| Tool | Version | Purpose |
|
||||
|------|---------|---------|
|
||||
| **vLLM** | v0.9.1 | High-throughput LLM inference server |
|
||||
| **Qwen 3.6 27B Abliterated** | Latest | Uncensored reasoning model (RTX 6000 deep thinker tier) |
|
||||
| **Qwen 2.5 Coder 7B Abliterated** | Latest | Fast tool-calling dispatcher (L4 24/7 tier) |
|
||||
| **NVIDIA L4 GPU** | - | 24/7 GPU for fast triage and dispatch |
|
||||
| **NVIDIA RTX 6000 Pro** | - | SPOT GPU for deep reasoning and multi-file context |
|
||||
|
||||
### Development Environment
|
||||
|
||||
| Tool | Version | Purpose |
|
||||
|------|---------|---------|
|
||||
| **Mise** | Latest | Development tool version manager |
|
||||
| **Dev Containers** | Latest | Consistent development environment |
|
||||
| **k9s** | Latest | Kubernetes CLI dashboard |
|
||||
|
||||
### Monitoring & Observability
|
||||
|
||||
| Tool | Version | Purpose |
|
||||
|------|---------|---------|
|
||||
| **Prometheus** | Latest | Metrics collection via kube-prometheus-stack |
|
||||
| **Grafana** | Latest | Dashboards & visualizations |
|
||||
| **Tailscale** | Latest | Secure VPN access to internal services |
|
||||
Datapath: GKE `ADVANCED_DATAPATH` + `enable_cilium_clusterwide_network_policy`. Dual-stack VPC (`10.0.0.0/16`, pods `192.168.32.0/20`, services `192.168.16.0/24`). GPU nodes are tainted (`nvidia.com/gpu…=present:NoSchedule`) so only inference pods land on them.
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
## GPU inference (vLLM)
|
||||
|
||||
Three independent OpenAI-compatible servers, each pinned to a pool via `nodeSelector` + matching taint/toleration. Model weights cached on PVC so a scale-up does not re-pull 20–40 GB from Hugging Face.
|
||||
|
||||
### L4 dispatcher — always on
|
||||
|
||||
[`infrastructure/gpus/base/vllm-servers/vllm-l4.yaml`](infrastructure/gpus/base/vllm-servers/vllm-l4.yaml)
|
||||
|
||||
- Machine: `g2-standard-8` + `nvidia-l4`, SPOT
|
||||
- Image: `vllm/vllm-openai`
|
||||
- Model: `p-e-w/Qwen3-8B-heretic`
|
||||
- Flags that matter: `--kv-cache-dtype=fp8`, `--enable-chunked-prefill`, `--enable-prefix-caching`, `--enable-auto-tool-choice`, `--tool-call-parser=hermes`, `--max-model-len=32768`
|
||||
|
||||
Cheap, tool-capable, left running so agents had a low-latency brain even when the big cards were scaled out.
|
||||
|
||||
### RTX PRO 6000 — deep context, scale to zero
|
||||
|
||||
[`infrastructure/gpus/base/vllm-servers/rtx6000-vllm.yaml`](infrastructure/gpus/base/vllm-servers/rtx6000-vllm.yaml)
|
||||
[`infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml`](infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml)
|
||||
|
||||
- Machine: `g4-standard-48` + `nvidia-rtx-pro-6000`, SPOT, **min 0 / max 1**
|
||||
- Image: `vllm/vllm-openai:latest-cu129-ubuntu2404`
|
||||
- Model: `edp1096/Huihui-Qwen3.6-27B-abliterated-FP8`
|
||||
- `--max-model-len=262144`, `--enable-chunked-prefill`, `--tool-call-parser=qwen3_xml`, `--reasoning-parser=qwen3`, MTP speculative decoding (`num_speculative_tokens: 2`)
|
||||
- Startup / liveness / readiness probes on `/health` and `/v1/models` (model load is slow; `failureThreshold: 60` on startup)
|
||||
- KEDA `HTTPScaledObject`: `replicas.min: 0`, `max: 1`, host `rtx6000-brain-service.customer1.svc.cluster.local`
|
||||
|
||||
Idle deep-thinker capacity cost nothing. A request woke the Deployment; GKE then scaled the node pool off zero.
|
||||
|
||||
### A100 80GB — same pattern
|
||||
|
||||
[`infrastructure/gpus/base/vllm-servers/a100-vllm.yaml`](infrastructure/gpus/base/vllm-servers/a100-vllm.yaml)
|
||||
[`modules/a100-nodepool.tf`](modules/a100-nodepool.tf)
|
||||
|
||||
- Machine: `a2-ultragpu-1g` + `nvidia-a100-80gb`, SPOT, min 0
|
||||
- NVFP4 / ModelOpt quantized 27B, `--quantization=modelopt`, `--attention-backend=flash_attn`, `--kv-cache-dtype=fp8`
|
||||
|
||||
---
|
||||
|
||||
## GitOps
|
||||
|
||||
Flux watches `master` and applies layered Kustomizations:
|
||||
|
||||
```
|
||||
clusters/devops-lab/
|
||||
flux-system/ Flux controllers + GitRepository
|
||||
infra-controllers.yaml → infrastructure/controllers/staging
|
||||
CNPG 0.26.1, KEDA ≥2.14, cert-manager, Tailscale, kube-prometheus
|
||||
infra-gpus.yaml → infrastructure/gpus/staging
|
||||
infra-gatewayapi.yaml → Gateway API + HTTPRoutes
|
||||
customer1-strimzi.yaml → Strimzi (Kafka) first
|
||||
customer1.yaml → apps/staging/customer1 (dependsOn strimzi)
|
||||
```
|
||||
|
||||
Every Flux Kustomization has `decryption.provider: sops` and `secretRef: sops-age`. Encrypted `data`/`stringData` in Git; Flux decrypts at apply time. Age recipient is in [`.sops.yaml`](.sops.yaml). The private key is **not** in this repo.
|
||||
|
||||
Bootstrap was originally:
|
||||
|
||||
```bash
|
||||
flux bootstrap github \
|
||||
--owner=sirius0xdev \
|
||||
--repository=gcloud-lab \
|
||||
--branch=master \
|
||||
--path=clusters/devops-lab
|
||||
```
|
||||
|
||||
`gotk-sync.yaml` still records that GitHub URL. This Forgejo copy is the archive; the cluster is gone, so the GitRepository object was never re-pointed.
|
||||
|
||||
---
|
||||
|
||||
## Terraform (cluster birth)
|
||||
|
||||
[`modules/`](modules/) is the IaC that created the cluster, not the day-to-day delivery path.
|
||||
|
||||
| File | Resource |
|
||||
| --- | --- |
|
||||
| `gke.tf` | `devops-lab-cluster` in `us-central1-a`, default pool removed, Cilium datapath, dual-stack IP policy |
|
||||
| `vpc.tf` | `devops-lab-network` / `devops-lab-subnetwork`, ULA IPv6, secondary ranges |
|
||||
| `nodepool.tf` | CPU `e2-standard-2`, 1–5, pd-standard 100 Gi |
|
||||
| `nodepool-gpu.tf` | L4 SPOT `g2-standard-8`, autoscaling 1–1, GPU taint |
|
||||
| `pro6000-nodepool.tf` | RTX PRO 6000 SPOT `g4-standard-48`, **0–1**, hyperdisk-balanced |
|
||||
| `a100-nodepool.tf` | A100 80GB SPOT `a2-ultragpu-1g`, **0–1**, pd-ssd 200 Gi |
|
||||
| `db-bucket.tf` | GCS bucket for CNPG backups |
|
||||
| `providers.tf` | Google provider, Helm talking to the cluster via `gke-gcloud-auth-plugin` |
|
||||
|
||||
State files and `*.tfvars` are gitignored. Do not `terraform apply` this against a live billing account unless you intend to recreate it.
|
||||
|
||||
---
|
||||
|
||||
## Data plane and apps
|
||||
|
||||
Flux overlay: [`apps/staging/customer1/kustomization.yaml`](apps/staging/customer1/kustomization.yaml)
|
||||
|
||||
| Piece | Role |
|
||||
| --- | --- |
|
||||
| CloudNative-PG | Operator `0.26.1`; per-app clusters (Hermes memory, waitlist, n8n, trading) with GCS backup |
|
||||
| Strimzi | Kafka for the trading data path; Flux `dependsOn` so apps wait for the operator |
|
||||
| Redis | In-cluster cache next to the trading services |
|
||||
| Trading platform | data / execute / news / dashboard Deployments, HTTPRoutes, NetworkPolicies |
|
||||
| Hermes agent | In-cluster agent + CNPG, talking to the vLLM services |
|
||||
| News bot | CronJobs: scrape → analyst (vLLM) → Telegram |
|
||||
| Waitlist API | FastAPI + CNPG, Gateway HTTPRoute |
|
||||
| n8n | Workflow engine on its own Postgres |
|
||||
| Gateway API | Public HTTPRoutes; Grafana/Prometheus stayed off the public internet (Tailscale / port-forward) |
|
||||
|
||||
App **images** were built in a separate code repo and pulled from GHCR. This repo is manifests only (after cleanup: no Dockerfiles, no Helm-of-the-app, no planning markdown).
|
||||
|
||||
---
|
||||
|
||||
## Networking and security
|
||||
|
||||
- **Cilium** as GKE datapath, clusterwide NetworkPolicy enabled from Terraform.
|
||||
- **Gateway API** `HTTPRoute` for public paths; internals not published.
|
||||
- **Tailscale operator** for operator access to Grafana and cluster services without a public LB.
|
||||
- **NetworkPolicies** on the trading namespace: DNS, Postgres, Redis, Kafka, in-namespace HTTP, egress HTTPS to market APIs. Everything else denied.
|
||||
- **SOPS + age** for Secrets in Git. Flux `sops-age` Secret in `flux-system` held the private key.
|
||||
- GPU nodes tainted so a random Deployment cannot schedule onto a $2+/hr card.
|
||||
|
||||
---
|
||||
|
||||
## Cost model (why it was torn down)
|
||||
|
||||
This was the whole point of the GPU design:
|
||||
|
||||
1. L4 SPOT stayed at 1 node — cheap enough to keep a dispatcher warm.
|
||||
2. RTX 6000 and A100 pools **autoscaled 0–1**. KEDA HTTPScaledObject set the Deployment to 0 when there was no traffic; the node pool followed.
|
||||
3. Weights on PVC, long startup probes — first request after idle paid a cold-start, not a 40 GB pull.
|
||||
4. CronJobs for batch work instead of idle inference pods.
|
||||
5. When even the L4 + control-plane bill stopped making sense, the cluster was destroyed. Scale-to-zero was the rehearsal for scale-to-nothing.
|
||||
|
||||
---
|
||||
|
||||
## Repository layout
|
||||
|
||||
```
|
||||
gcloud-lab/
|
||||
├── modules/ # Terraform IaC modules
|
||||
│ ├── providers.tf # Provider configurations
|
||||
│ ├── gke.tf # GKE cluster definition
|
||||
│ ├── vpc.tf # VPC and subnet configuration
|
||||
│ ├── nodepool.tf # Standard node pool
|
||||
│ ├── nodepool-gpu.tf # GPU node pools (L4 + RTX 6000 SPOT)
|
||||
│ ├── flux.tf # Flux GitOps bootstrap
|
||||
│ ├── helm.tf # Helm chart deployments (Cilium)
|
||||
│ └── variables.tf # Input variables
|
||||
│
|
||||
├── clusters/ # Cluster configurations
|
||||
│ └── devops-lab/
|
||||
│ ├── flux-system/ # Flux CD components
|
||||
│ │ ├── gotk-components.yaml # Flux controllers
|
||||
│ │ ├── gotk-sync.yaml # Git repository sync
|
||||
│ │ └── kustomization.yaml # Flux kustomization
|
||||
│ ├── customer1.yaml # Customer1 Kustomization
|
||||
│ ├── agent-forge.yaml # AgentForge Kustomization
|
||||
│ ├── infra-controllers.yaml # Infrastructure controllers (CNPG, KEDA, Monitoring, Tailscale)
|
||||
│ └── infra-configs.yaml # Infrastructure configs
|
||||
│
|
||||
├── infrastructure/ # Infrastructure components
|
||||
│ ├── controllers/
|
||||
│ │ ├── base/
|
||||
│ │ │ ├── cnpg/ # CloudNative PG operator
|
||||
│ │ │ ├── keda/ # KEDA autoscaling
|
||||
│ │ │ ├── monitoring/ # Prometheus + Grafana (no public ingress)
|
||||
│ │ │ └── tailscale/ # Tailscale Operator for secure VPN access
|
||||
│ │ └── staging/
|
||||
│ │ └── kustomization.yaml # Aggregates all base components
|
||||
│ └── configs/
|
||||
│ └── staging/
|
||||
│ └── kustomization.yaml
|
||||
│
|
||||
├── apps/ # Application deployments
|
||||
│ ├── base/
|
||||
│ │ ├── customer1/
|
||||
│ │ │ ├── namespace.yaml # Namespace definition
|
||||
│ │ │ ├── deployment.yaml # N8N + vLLM deployments
|
||||
│ │ │ ├── service.yaml # ClusterIP services
|
||||
│ │ │ ├── storage.yaml # PersistentVolumeClaims
|
||||
│ │ │ ├── configmap.yaml # Application configuration
|
||||
│ │ │ ├── pg-cluster-customer1.yaml # PostgreSQL cluster
|
||||
│ │ │ ├── apigateway.yaml # GCP Gateway
|
||||
│ │ │ ├── http-route.yaml # HTTP routing
|
||||
│ │ │ ├── healthcheck.yaml # Health check policy
|
||||
│ │ │ ├── waitlist-api/ # Waitlist API microservice
|
||||
│ │ │ │ ├── deployment.yaml
|
||||
│ │ │ │ ├── service.yaml
|
||||
│ │ │ │ └── configmap.yaml
|
||||
│ │ │ └── news_bot/ # News bot microservices
|
||||
│ │ │ ├── scraper-cronjob.yaml
|
||||
│ │ │ ├── analyst-cronjob.yaml
|
||||
│ │ │ ├── telebot-cronjob.yaml
|
||||
│ │ │ ├── scrapy-configmap.yaml
|
||||
│ │ │ └── scrapy-urls-configmap.yaml
|
||||
│ │ ├── agent-forge/
|
||||
│ │ │ ├── namespace.yaml
|
||||
│ │ │ ├── vllm-deep-thinker.yaml # RTX 6000 deployment with KEDA
|
||||
│ │ │ ├── hermes-tenant.yaml # Per-tenant Hermes agent instance
|
||||
│ │ │ └── pg-cluster-agentforge.yaml
|
||||
│ │ ├── kanban/
|
||||
│ │ │ ├── namespace.yaml
|
||||
│ │ │ ├── hermes-deployment.yaml # AI agent orchestrator
|
||||
│ │ │ └── pg-cluster-hermes.yaml
|
||||
│ │ └── local-business/
|
||||
│ │ └── template/
|
||||
│ │ ├── namespace.yaml
|
||||
│ │ ├── nginx-deployment.yaml
|
||||
│ │ ├── configmap.yaml
|
||||
│ │ └── http-route.yaml
|
||||
│ └── staging/
|
||||
│ ├── customer1/
|
||||
│ │ └── kustomization.yaml
|
||||
│ ├── agent-forge/
|
||||
│ │ └── kustomization.yaml
|
||||
│ └── kanban/
|
||||
│ └── kustomization.yaml
|
||||
│
|
||||
├── scripts/
|
||||
│ └── setup # Development setup script
|
||||
│
|
||||
├── .devcontainer.json # Dev container configuration
|
||||
├── mise.toml # Tool version management
|
||||
├── age.agekey # SOPS encryption key
|
||||
└── README.md # This file
|
||||
├── modules/ Terraform: VPC, GKE, node pools, GCS
|
||||
├── clusters/devops-lab/ Flux entry (GitRepository + Kustomizations)
|
||||
├── infrastructure/
|
||||
│ ├── controllers/ CNPG, KEDA, cert-manager, Tailscale, Prometheus
|
||||
│ ├── gpus/ vLLM Deployments + KEDA HTTPScaledObjects
|
||||
│ ├── gatewayapi/ Gateway + HTTPRoutes
|
||||
│ └── tailnet/ Tailscale ProxyGroup
|
||||
├── apps/
|
||||
│ ├── base/customer1/ Namespaced workloads (kustomize)
|
||||
│ ├── base/osint-dashboard/ Helm chart used on this cluster
|
||||
│ └── staging/ Overlays Flux actually syncs
|
||||
├── monitoring/ Extra Grafana/Prometheus config
|
||||
├── .sops.yaml Age recipient for secret encryption
|
||||
├── .github/workflows/ Historical GH Actions (pgvector image, etc.)
|
||||
└── mise.toml Local CLI pin (gcloud, kubectl, helm, sops, terraform, k9s)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Infrastructure Components
|
||||
## Reading the code (suggested order)
|
||||
|
||||
### GKE Cluster
|
||||
|
||||
- **Name**: `devops-lab-cluster`
|
||||
- **Region**: `us-central1-a`
|
||||
- **Network**: Custom VPC with dual-stack IPv4/IPv6
|
||||
|
||||
### Node Pools
|
||||
|
||||
| Pool | Machine Type | Scaling | Purpose |
|
||||
|------|-------------|---------|---------|
|
||||
| Standard | e2-standard-2 | 1-16 nodes | General workloads, N8N, web servers |
|
||||
| GPU L4 (SPOT) | g2-standard-8 + L4 | 0-5 nodes | vLLM dispatcher, 24/7 fast inference |
|
||||
| GPU RTX 6000 (SPOT) | g6-standard-4 + RTX 6000 Pro | 0-1 nodes | Deep thinker tier, multi-file reasoning |
|
||||
|
||||
### Networking
|
||||
|
||||
- **VPC**: `devops-lab-network`
|
||||
- **Primary CIDR**: `10.0.0.0/16`
|
||||
- **Pod CIDR**: `192.168.32.0/20`
|
||||
- **Service CIDR**: `192.168.16.0/24`
|
||||
- **CNI**: Cilium with advanced datapath and NetworkPolicy enforcement
|
||||
- **Ingress**: Kubernetes Gateway API via `external-http-gateway` with PathPrefix HTTPRoute routing
|
||||
- **Internal Services**: Tailscale-only — no public ingress for monitoring, databases, or agent infrastructure
|
||||
|
||||
### CNPG Database Fleet
|
||||
|
||||
Multiple isolated PostgreSQL clusters, each with dedicated databases per application:
|
||||
|
||||
| Cluster | Namespace | Databases | Backup |
|
||||
|---------|-----------|-----------|--------|
|
||||
| `customer1-pgdb` | customer1 | `n8n`, `news_app`, `waitlist` | GCS, 7-day retention |
|
||||
| `hermes-pgdb` | kanban | `hermes`, `memory_store` | GCS, 7-day retention |
|
||||
| `hermes-tenant-pgdb` | agent-forge | Per-tenant isolated DBs | GCS, 7-day retention |
|
||||
| `siriusdevops-pgdb` | customer1 | `waitlist_prod` | GCS, 30-day retention |
|
||||
|
||||
### GitOps Flow
|
||||
|
||||
```
|
||||
GitHub Repository (ghcr.io/sirius0xdev)
|
||||
│
|
||||
▼
|
||||
Flux Source Controller (watches git, 1min interval)
|
||||
│
|
||||
▼
|
||||
Flux Kustomize Controller (applies manifests)
|
||||
│
|
||||
├── infrastructure/controllers → CNPG, KEDA, Monitoring, Tailscale
|
||||
├── infrastructure/configs → Cluster configs
|
||||
├── apps/staging/customer1 → PAaaS, N8N, News Bot, Waitlist API
|
||||
├── apps/staging/agent-forge → Multi-tenant AI agent hosting
|
||||
├── apps/staging/kanban → AI Agent Team orchestrator
|
||||
└── apps/staging/local-business → Business websites
|
||||
```
|
||||
1. [`modules/gke.tf`](modules/gke.tf) + [`modules/vpc.tf`](modules/vpc.tf) — cluster shape.
|
||||
2. GPU pools, then the matching vLLM YAML — taint keys must match or the pod never schedules.
|
||||
3. [`clusters/devops-lab/customer1.yaml`](clusters/devops-lab/customer1.yaml) — Flux `dependsOn`, SOPS, prune/force.
|
||||
4. [`apps/staging/customer1/kustomization.yaml`](apps/staging/customer1/kustomization.yaml) — what actually shipped in `customer1`.
|
||||
5. One NetworkPolicy under `apps/base/customer1/trading-platform/network-policies/` — default-deny thinking.
|
||||
|
||||
---
|
||||
|
||||
## Applications
|
||||
|
||||
### 1. AgentForge — Private AI Agent Workspace
|
||||
|
||||
A premium, uncensored, privacy-first AI agent hosting platform with dual-tier cognitive architecture:
|
||||
|
||||
- **Tier 1 (Dispatcher):** L4 GPU SPOT instance running 24/7. Hosts `Qwen2.5-Coder-7B-Instruct-heretic` via vLLM `v0.9.1` for lightning-fast, cheap triage and tool calling.
|
||||
- **Tier 2 (Deep Thinker):** RTX 6000 Pro Spot instance scaling from 0-1 via KEDA. Hosts `Qwen3.5-27B-heretic` with `--enable-chunked-prefill` and `--kv-cache-dtype=fp8` for massive multi-file context and reasoning without OOMing or stalling concurrent users.
|
||||
- **Frontend:** Isolated Hermes agent profiles per tenant, connected to Telegram/Discord via outbound polling (no public ingress required).
|
||||
- **Landing Page:** Dockerized marketing site built via CI/CD from `hermes-projects` and deployed to the `staging` kustomization overlay.
|
||||
- **Container Registry:** All images pushed to `ghcr.io/sirius0xdev`.
|
||||
|
||||
### 2. Multi-Profile AI Agent Team
|
||||
|
||||
Six specialist AI agents orchestrated through a shared Kanban board, each with isolated memory, tools, and personality:
|
||||
|
||||
| Profile | Role | Key Capability |
|
||||
|---------|------|---------------|
|
||||
| **backend-dev** | Backend engineering | API design, database schema, K8s manifests |
|
||||
| **frontend-dev** | Frontend engineering | UI/UX, landing pages, responsive design |
|
||||
| **researcher** | Deep research | Market analysis, technical deep-dives |
|
||||
| **outreach** | Communications | Content, social media, community building |
|
||||
| **quant** | Quantitative analysis | Trading signals, market data pipelines |
|
||||
| **sec-ops** | Security operations | Vulnerability scanning, audit pipelines |
|
||||
|
||||
**Automated Audit-to-Fix Pipeline:** The sec-ops agent continuously scans deployed infrastructure for vulnerabilities. When findings are confirmed, the backend-dev agent is automatically dispatched to remediate — from detection to patch in a single GitOps cycle.
|
||||
|
||||
### 3. Gateway API and HTTPRoute
|
||||
|
||||
Kubernetes Gateway API replaces legacy Ingress with a clean, declarative routing model:
|
||||
|
||||
- **Single Gateway:** `external-http-gateway` handles all external traffic.
|
||||
- **PathPrefix Routing:** `/agentforge/*` → AgentForge landing, `/waitlist/*` → Waitlist API, `/business/*` → local business sites.
|
||||
- **No Public Ingress for Internals:** Monitoring (Grafana/Prometheus), databases, and agent infrastructure are accessible only via Tailscale VPN.
|
||||
- **Cross-Namespace References:** HTTPRoute resources in one namespace can reference Services in another, keeping routing centralized.
|
||||
|
||||
### 4. Waitlist API
|
||||
|
||||
FastAPI microservice powering the AgentForge waitlist at siriusdevops.com:
|
||||
|
||||
- **Database:** asyncpg connection pool to dedicated CNPG PostgreSQL.
|
||||
- **Idempotent Signups:** `INSERT ... ON CONFLICT DO NOTHING` — duplicate emails are silently ignored, not rejected.
|
||||
- **Notifications:** Fire-and-forget Telegram webhook on each new signup. No blocking I/O in the request path.
|
||||
- **Security:** Rate limiting per IP, input sanitization, and CORS whitelist.
|
||||
|
||||
### 5. Autonomous News Quant Pipeline (`news_bot`)
|
||||
|
||||
An institutional-grade pipeline scraping 371 global feeds to generate actionable futures trading signals:
|
||||
|
||||
- **Scraper:** CronJob at `:50` pulling multi-lingual global financial data.
|
||||
- **Map/Reduce Analyst:** DeepSeek-R1 with a strict 10-step think protocol extracts "Market-Moving DNA" and translates events into explicit futures targets (/ES, /CL, /NQ) with risk:reward, take profit, and stop loss levels.
|
||||
- **Privacy:** All proprietary technical data stays strictly within the VPC, executing against local models to protect the trading edge.
|
||||
|
||||
### 6. Local Business Web Deployment Pipeline
|
||||
|
||||
Automated Kubernetes manifest generation for small business websites:
|
||||
|
||||
- **Stack:** nginx serving static content from ConfigMap, one namespace per business.
|
||||
- **Routing:** HTTPRoute with cross-namespace Service references under `/business/<name>` paths.
|
||||
- **Zero Cold Start:** Static sites have no database dependency — just nginx + ConfigMap, deployed via GitOps.
|
||||
|
||||
---
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Google Cloud account with billing enabled
|
||||
- GitHub account with repository access
|
||||
- `gcloud` CLI authenticated
|
||||
- Terraform 1.7+
|
||||
|
||||
### Local Development Setup
|
||||
## Local tooling
|
||||
|
||||
```bash
|
||||
# Install tools via mise
|
||||
./scripts/setup
|
||||
|
||||
# Or manually
|
||||
mise trust && mise install
|
||||
mise trust && mise install # kubectl, helm, sops, terraform, k9s, gcloud
|
||||
```
|
||||
|
||||
### Infrastructure Deployment
|
||||
Decrypt a secret locally (needs the age key, which is not in Git):
|
||||
|
||||
```bash
|
||||
cd modules
|
||||
|
||||
# Initialize Terraform
|
||||
terraform init
|
||||
|
||||
# Set required variables
|
||||
export TF_VAR_github_token="your-token"
|
||||
export TF_VAR_github_org="your-org"
|
||||
export TF_VAR_github_repository="gcloud-lab"
|
||||
|
||||
# Plan and apply
|
||||
terraform plan
|
||||
terraform apply
|
||||
```
|
||||
|
||||
### Accessing the Cluster
|
||||
|
||||
```bash
|
||||
# Configure kubectl
|
||||
gcloud container clusters get-credentials devops-lab-cluster \
|
||||
--zone us-central1-a \
|
||||
--project devops-lab-cluster
|
||||
|
||||
# Verify connection
|
||||
kubectl get nodes
|
||||
|
||||
# Use k9s for interactive management
|
||||
k9s
|
||||
```
|
||||
|
||||
### Accessing Monitoring (Grafana / Prometheus)
|
||||
|
||||
Monitoring services are **not publicly exposed**. Access is via Tailscale VPN or port-forwarding:
|
||||
|
||||
```bash
|
||||
# Option 1: Port-forward Grafana
|
||||
kubectl port-forward svc/prometheus-community-kube-prometheus-stack-grafana \
|
||||
-n monitoring 3000:3000
|
||||
|
||||
# Option 2: Port-forward Prometheus
|
||||
kubectl port-forward svc/prometheus-community-kube-prometheus-stack-prometheus \
|
||||
-n monitoring 9090:9090
|
||||
```
|
||||
|
||||
⚠️ **Before deploying**, replace the Grafana admin password in
|
||||
`infrastructure/controllers/base/monitoring/release.yaml` with a secure value,
|
||||
or create a `monitoring-grafana-admin` Secret instead.
|
||||
|
||||
---
|
||||
|
||||
## Security
|
||||
|
||||
### Secrets Management
|
||||
|
||||
- **Encryption**: SOPS with Age encryption
|
||||
- **Key Storage**: `age.agekey` (do not commit unencrypted)
|
||||
- **Flux Integration**: Automatic decryption during deployment
|
||||
|
||||
### Pod Security
|
||||
|
||||
- Non-root containers (UID 1000)
|
||||
- Filesystem group enforcement
|
||||
- Privilege escalation disabled
|
||||
- Resource limits enforced
|
||||
|
||||
### Network Security
|
||||
|
||||
- Cilium NetworkPolicy for pod-to-pod and namespace-to-namespace isolation
|
||||
- Kubernetes Gateway API with TLS termination at the load balancer
|
||||
- Internal services (monitoring, databases, agent infrastructure) accessible only via Tailscale VPN — zero public ingress
|
||||
- Rate limiting on public-facing APIs (Waitlist, landing page)
|
||||
|
||||
### Database Security
|
||||
|
||||
- Managed roles with secret-based passwords per application
|
||||
- Separate PostgreSQL clusters per domain (hermes-pgdb, hermes-tenant-pgdb, siriusdevops-pgdb)
|
||||
- GCS backups with configurable retention policies
|
||||
- HA cluster with automatic failover
|
||||
|
||||
### Automated Security Auditing
|
||||
|
||||
- **sec-ops Agent:** Continuously scans deployed infrastructure for CVEs, misconfigurations, and policy violations
|
||||
- **Auto-Remediation:** Confirmed findings automatically dispatch the backend-dev agent to patch and commit
|
||||
- **Audit Trail:** Every finding, fix, and deployment is tracked in Git history — full provenance from detection to resolution
|
||||
|
||||
---
|
||||
|
||||
## Cost Optimization
|
||||
|
||||
- **SPOT GPU Instances**: 60-90% savings on L4 and RTX 6000 workloads
|
||||
- **KEDA Scale-to-Zero**: RTX 6000 deep thinker pool scales to 0 when no requests are queued
|
||||
- **Resource Limits**: CPU and memory caps on every container prevent runaway costs
|
||||
- **Scheduled Workloads**: CronJobs only run when needed — no idle inference pods
|
||||
- **Tailscale for Internal Access**: No need for expensive internal load balancers or Cloud NAT for monitoring
|
||||
|
||||
---
|
||||
|
||||
## Container Images
|
||||
|
||||
```
|
||||
ghcr.io/sirius0xdev/agentforge-landing:latest
|
||||
ghcr.io/sirius0xdev/waitlist-api:latest
|
||||
ghcr.io/sirius0xdev/newsscraper:latest
|
||||
ghcr.io/sirius0xdev/summarizer:latest
|
||||
ghcr.io/sirius0xdev/news-messenger:latest
|
||||
docker.n8n.io/n8nio/n8n:2.1.4
|
||||
ghcr.io/cloudnative-pg/postgresql:15.2
|
||||
sops -d apps/base/customer1/waitlist-api/waitlist-telegram-secret.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tool Reference
|
||||
## Status
|
||||
|
||||
### Terraform Providers
|
||||
| | |
|
||||
| --- | --- |
|
||||
| Cluster | **Destroyed** (GCP project `devops-lab-cluster`, GKE `devops-lab-cluster`, `us-central1-a`) |
|
||||
| This repo | Archive of what ran |
|
||||
| Live infra today | Self-hosted on a Pi — see [siriusdevops.com/lab](https://siriusdevops.com/lab) |
|
||||
|
||||
```hcl
|
||||
google = "~> 7.14" # GCP resources
|
||||
helm = "~> 2.0" # Helm chart management
|
||||
flux = "~> 1.7" # GitOps bootstrap
|
||||
```
|
||||
|
||||
### Helm Charts
|
||||
|
||||
```yaml
|
||||
cilium: 1.18.5 # CNI and service mesh
|
||||
cloudnative-pg: 0.26.1 # PostgreSQL operator
|
||||
vllm: 0.9.1 # High-throughput LLM serving
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
Private repository — All rights reserved.
|
||||
Lance Walters — [siriusdevops.com](https://siriusdevops.com)
|
||||
|
|
|
|||
|
|
@ -1,27 +0,0 @@
|
|||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: analyst-gemma
|
||||
namespace: customer1
|
||||
spec:
|
||||
schedule: "15 * * * *"
|
||||
jobTemplate:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: analyst
|
||||
image: siriussec/summarizerlocal:3.5
|
||||
imagePullPolicy: Always
|
||||
env:
|
||||
- name: DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: news-app-password
|
||||
key: password
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: gemma-config
|
||||
|
||||
restartPolicy: OnFailure
|
||||
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: news-analyst-gemini
|
||||
namespace: customer1
|
||||
|
||||
spec:
|
||||
schedule: "15 * * * *"
|
||||
jobTemplate:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: analyst-gemini
|
||||
image: siriussec/summarizer:1.5
|
||||
|
||||
env:
|
||||
- name: DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: news-app-password
|
||||
key: password
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: news-app-config
|
||||
|
||||
|
||||
- secretRef:
|
||||
name: gemini-apikey
|
||||
restartPolicy: OnFailure
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: news-app-config
|
||||
namespace: customer1
|
||||
data:
|
||||
DB_HOST: "siriusdevops-pgdb-rw"
|
||||
DB_NAME: "news_app_db"
|
||||
DB_USER: "news_app"
|
||||
DB_PORT: "5432"
|
||||
LLM_BASE_URL: "http://rtx6000-brain-service.customer1.svc.cluster.local:8000/v1"
|
||||
MODEL_NAME: "edp1096/Huihui-Qwen3.6-27B-abliterated-FP8"
|
||||
LLM_API_KEY: "sk-dummy"
|
||||
SUMMARY_PROMPT: |
|
||||
You are an expert news analyst working for DARPA. Your task is to deliver thorough, insightful analysis of the provided news articles.
|
||||
|
||||
ARTICLES:
|
||||
{data}
|
||||
|
||||
Instructions:
|
||||
- Identify the 3–6 most impactful events or developments (prioritize geopolitical, technological, military, economic, or security implications).
|
||||
- For each impactful event: Write a detailed 4–6 sentence summary, including key facts, context, potential consequences, and why it matters strategically.
|
||||
- For all remaining/lesser articles: Write 2–4 cohesive paragraphs synthesizing them into broader themes or trends (do NOT list them individually unless critical).
|
||||
- Aim for a total output length of 400–800 words. Be comprehensive but concise.
|
||||
- Always produce substantial content — even if articles seem minor, extract value, implications, or connections.
|
||||
- Use neutral, professional, analytical tone with precise language.
|
||||
- Structure your response clearly with headings like:
|
||||
## Most Impactful Events
|
||||
### Event 1: [Brief Title]
|
||||
[4–6 sentences...]
|
||||
...
|
||||
## Synthesis of Lesser Developments
|
||||
[Paragraphs...]
|
||||
- Output ONLY the structured report — no introductions, apologies, or meta-comments like "no major events found".
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: deepseek-analyst-v1.1
|
||||
namespace: customer1
|
||||
spec:
|
||||
schedule: "15 * * * *"
|
||||
jobTemplate:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: analyst
|
||||
image: siriussec/summarizerlocal:3.6
|
||||
imagePullPolicy: Always
|
||||
env:
|
||||
- name: DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: news-app-password
|
||||
key: password
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: deepseek-config
|
||||
resources:
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "4Gi"
|
||||
restartPolicy: OnFailure
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
apiVersion: v1
|
||||
data:
|
||||
GEMINI_API_KEY: ENC[AES256_GCM,data:EcyraxRLXiWWQVhJNmmUGut6AmSMA8BvFg0X/PoP2ckclcFas8pU7GfsmMXFD9arcuCJpg==,iv:52YNqcPGWQqQY4jdfc8DtP6dROTsxJqUDahXzEN3ycI=,tag:Q/9l0mDG+5sjTLKt5Jg4sw==,type:str]
|
||||
kind: Secret
|
||||
metadata:
|
||||
creationTimestamp: null
|
||||
name: gemini-apikey
|
||||
namespace: customer1
|
||||
sops:
|
||||
age:
|
||||
- recipient: age1uuxf066xuuqgvjppxfcmqkwfcufnwp3wcwnl9h20g9k4l8nkw9jsaungf7
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTcGdQb0tMdWtiTkl3aWZt
|
||||
dnl4ZHo3aHZ4bGgvR1Rva0E5dlloQlRGY0d3CmMwOUJWb3NlWEtRVEk3STJ5MWVk
|
||||
d21ZNzkyeHBZQTF3dWhYdUlSZEtQQkEKLS0tIEUrU3M2UGh5Y1BPTUhRQi9NK2VN
|
||||
Vi83QzhjcG45U1B1WTJGcklFdkRvQjQK/jAjKf3wCZCpHp5naJoitHfN1yEEqmoN
|
||||
p5AMB97oTtyVhvu3wkRrxqHB8LKNWZifaJf8g13To1OgFh9azC/Kng==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-01-16T20:31:27Z"
|
||||
mac: ENC[AES256_GCM,data:7y9LrLu+bngdTuhb1ucIDnvg3jpKgXW6rXMpS6O08nxHHBKAKA6Jl80PsfOE0aQNSsIM2ERfrdGmfLkNh470oB/erGvEHxRLKcf8IStPxoPpry99XffiVo+QJrCPIzqgKy1tw37bqBNrm5LJkhUmugzf/IVt6WKiGCZhpbXjB7E=,iv:RfK084/orIGcRwXMeAip0BKXJ3eYJoRmEgLF72ER9vA=,tag:NKqZKIrt3yVkGL4ZkqcZKA==,type:str]
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.11.0
|
||||
|
|
@ -1,63 +0,0 @@
|
|||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: gemma-config
|
||||
namespace: customer1
|
||||
data:
|
||||
DB_HOST: "siriusdevops-pgdb-rw"
|
||||
DB_NAME: "news_app_db"
|
||||
DB_USER: "news_app"
|
||||
DB_PORT: "5432"
|
||||
LLM_BASE_URL: "http://rtx6000-brain-service.customer1.svc.cluster.local:8000/v1"
|
||||
MODEL_NAME: "edp1096/Huihui-Qwen3.6-27B-abliterated-FP8"
|
||||
LLM_API_KEY: "sk-dummy"
|
||||
|
||||
MAP_PROMPT: |
|
||||
You are a state of the art AI News analysis
|
||||
|
||||
working for a independent day trader. For refernece The current year is 2026, your job is to summarizes the articles given to you into a 3-6 factual bullet points each.
|
||||
You are looking for any news that may effect the stock or futures market. Specifically the futures. These articles are from world wide sources and in many languages . Translate all to english.
|
||||
Rate eash article on a scale of 1-10 depending on it's possible market effects . 1 being no effect 10 being a critical event . In each event with market relevance be sure to list the tickers or futures symbols that will be affected.
|
||||
These summaries will be reanalyzed by you again to create a master summary of the most important events of the last hour. So leave any notes that may be useful.
|
||||
|
||||
data:
|
||||
{batch_text}
|
||||
|
||||
|
||||
SUMMARY_PROMPT: |
|
||||
Role: Futures Analyst AI for funded traders.
|
||||
Objective: Extract actionable "edges" and volatility signals from news batches for Equity Indices, Energies, Metals, Ag, and FX.
|
||||
|
||||
Task:
|
||||
|
||||
Filter: Ignore noise (sports, entertainment, local). Deep-analyze only top 5-10 high-signal articles.
|
||||
|
||||
Criteria: Scan for Economic Data (Fed/CPI), Geopolitics, Supply/Demand shifts, and Corporate news impacting futures.
|
||||
|
||||
Analyze: Assess Impact (H/M/L), Direction (Bullish/Bearish), Trading Edge (actionable strategy), and Confidence Score (1-10).
|
||||
|
||||
Output (Telegram-friendly Markdown + Emojis):
|
||||
📊 Summary Dashboard
|
||||
|
||||
Scanned: [Total] | Relevant: [Count]
|
||||
|
||||
Affected: [e.g., CL, ES, NQ]
|
||||
|
||||
Alert: [Brief sentiment summary]
|
||||
|
||||
🔥 Detailed Insights (Title/Source/Time)
|
||||
|
||||
Summary: [1-2 sentences]
|
||||
|
||||
Impact: [Contract/Direction/Rationale]
|
||||
|
||||
Trading Edge: [Actionable entry/exit/monitoring signal]
|
||||
|
||||
Confidence: [1-10]
|
||||
|
||||
👀 Watchlist: [Low-impact/monitoring items]
|
||||
If empty, output: "No significant futures-impacting news in this batch."
|
||||
|
||||
Constraint: Objective, data-driven, no hype. Use historical analogies to quantify moves where possible.
|
||||
|
||||
Data to analyze: {final_input}
|
||||
|
|
@ -1,5 +1,3 @@
|
|||
|
||||
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
|
|
@ -27,5 +25,5 @@ spec:
|
|||
resources:
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "2Gi"
|
||||
memory: "1Gi"
|
||||
restartPolicy: OnFailure
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: ollama-storage
|
||||
namespace: customer1
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 50Gi
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: trade-dashboard-config
|
||||
namespace: customer1
|
||||
data:
|
||||
DB_HOST: "siriusdevops-pgdb-rw.customer1.svc.cluster.local"
|
||||
DB_PORT: "5432"
|
||||
DB_NAME: "trading_dashboard"
|
||||
|
|
@ -1,69 +0,0 @@
|
|||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: trade-dashboard
|
||||
namespace: customer1
|
||||
labels:
|
||||
app: trade-dashboard
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app: trade-dashboard
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: trade-dashboard
|
||||
annotations:
|
||||
checksum/config: trade-dashboard-config
|
||||
spec:
|
||||
terminationGracePeriodSeconds: 30
|
||||
containers:
|
||||
- name: dashboard
|
||||
image: ghcr.io/sirius0xdev/trade-dashboard:latest
|
||||
imagePullPolicy: Always
|
||||
ports:
|
||||
- containerPort: 8000
|
||||
name: http
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: trade-dashboard-config
|
||||
env:
|
||||
- name: DB_USER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: trading-dashboard-db-credentials
|
||||
key: username
|
||||
- name: DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: trading-dashboard-db-credentials
|
||||
key: password
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /api/health
|
||||
port: 8000
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 5
|
||||
failureThreshold: 6
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /api/health
|
||||
port: 8000
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 30
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /api/health
|
||||
port: 8000
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- deployment.yaml
|
||||
- service.yaml
|
||||
- configmap.yaml
|
||||
# - tsproxy.yaml
|
||||
|
|
@ -1,17 +0,0 @@
|
|||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: trade-dashboard
|
||||
namespace: customer1
|
||||
annotations:
|
||||
tailscale.com/expose: "true"
|
||||
tailscale.com/hostname: "trade-dashboard"
|
||||
tailscale.com/tags: "tag:k8s-operator"
|
||||
tailscale.com/ports: "http:80"
|
||||
spec:
|
||||
selector:
|
||||
app: trade-dashboard
|
||||
ports:
|
||||
- port: 80
|
||||
targetPort: 8000
|
||||
name: http
|
||||
|
|
@ -7,6 +7,6 @@ data:
|
|||
DB_HOST: "siriusdevops-pgdb-rw.customer1.svc.cluster.local"
|
||||
DB_PORT: "5432"
|
||||
DB_NAME: "trading_data"
|
||||
DATA_SERVICE_URL: "http://trading-data-service.customer1.svc.cluster.local"
|
||||
EXECUTE_SERVICE_URL: "http://trading-execute-service.customer1.svc.cluster.local"
|
||||
NEWS_SERVICE_URL: "http://trading-news-service.customer1.svc.cluster.local"
|
||||
DATA_SERVICE_URL: "http://trading-data-service.customer1.svc.cluster.local:8000"
|
||||
EXECUTE_SERVICE_URL: "http://trading-execute-service.customer1.svc.cluster.local:8000"
|
||||
NEWS_SERVICE_URL: "http://trading-news-service.customer1.svc.cluster.local:8000"
|
||||
|
|
|
|||
|
|
@ -115,7 +115,7 @@ spec:
|
|||
ports:
|
||||
- port: 9092
|
||||
protocol: TCP
|
||||
# Allow inter-service egress
|
||||
# Allow inter-service egress (via ClusterIP services on port 80, plus direct pod ports)
|
||||
- to:
|
||||
- podSelector:
|
||||
matchExpressions:
|
||||
|
|
@ -127,6 +127,8 @@ spec:
|
|||
- trading-news-service
|
||||
- trading-dashboard
|
||||
ports:
|
||||
- port: 80
|
||||
protocol: TCP
|
||||
- port: 8000
|
||||
protocol: TCP
|
||||
- port: 8001
|
||||
|
|
|
|||
1
apps/osint-dashboard/.gitignore
vendored
1
apps/osint-dashboard/.gitignore
vendored
|
|
@ -1 +0,0 @@
|
|||
__pycache__/
|
||||
|
|
@ -1,18 +0,0 @@
|
|||
FROM python:3.13-slim AS base
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
gcc libpq-dev \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY app/requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY app/ ./app/
|
||||
COPY alembic.ini ./alembic.ini
|
||||
COPY alembic/ ./alembic/
|
||||
|
||||
EXPOSE 8000
|
||||
ENV PYTHONPATH=/app
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
|
|
@ -1,37 +0,0 @@
|
|||
[alembic]
|
||||
script_location = alembic
|
||||
# Override at runtime via DATABASE_URL environment variable (set in ConfigMap/Deployment)
|
||||
sqlalchemy.url = driver://user:pass@localhost/dbname
|
||||
|
||||
[loggers]
|
||||
keys = root,sqlalchemy,alembic
|
||||
|
||||
[handlers]
|
||||
keys = console
|
||||
|
||||
[formatters]
|
||||
keys = generic
|
||||
|
||||
[logger_root]
|
||||
level = WARN
|
||||
handlers = console
|
||||
|
||||
[logger_sqlalchemy]
|
||||
level = WARN
|
||||
handlers =
|
||||
qualname = sqlalchemy.engine
|
||||
|
||||
[logger_alembic]
|
||||
level = INFO
|
||||
handlers =
|
||||
qualname = alembic
|
||||
|
||||
[handler_console]
|
||||
class = StreamHandler
|
||||
args = (sys.stderr,)
|
||||
level = NOTSET
|
||||
formatter = generic
|
||||
|
||||
[formatter_generic]
|
||||
format = %(levelname)-5.5s [%(name)s] %(message)s
|
||||
datefmt = %H:%M:%S
|
||||
|
|
@ -1,55 +0,0 @@
|
|||
import sys
|
||||
from logging.config import fileConfig
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import pool
|
||||
from sqlalchemy.ext.asyncio import async_engine_from_config
|
||||
|
||||
from alembic import context
|
||||
|
||||
# Add app directory to path so we can import models/database
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "app"))
|
||||
|
||||
from database import metadata, DATABASE_URL
|
||||
|
||||
config = context.config
|
||||
if config.config_file_name is not None:
|
||||
fileConfig(config.config_file_name)
|
||||
target_metadata = metadata
|
||||
|
||||
|
||||
def run_migrations_offline() -> None:
|
||||
"""Run migrations in 'offline' mode."""
|
||||
url = config.get_main_option("sqlalchemy.url")
|
||||
context.configure(url=url, target_metadata=target_metadata, literal_binds=True)
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
def do_run_migrations(connection):
|
||||
context.configure(connection=connection, target_metadata=target_metadata)
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
async def run_async_migrations():
|
||||
connectable = async_engine_from_config(
|
||||
config.get_section(config.config_ini_section, {}),
|
||||
prefix="sqlalchemy.",
|
||||
poolclass=pool.NullPool,
|
||||
)
|
||||
async with connectable.connect() as connection:
|
||||
await connection.run_sync(do_run_migrations)
|
||||
await connectable.dispose()
|
||||
|
||||
|
||||
def run_migrations_online() -> None:
|
||||
"""Run migrations in 'online' mode."""
|
||||
import asyncio
|
||||
asyncio.run(run_async_migrations())
|
||||
|
||||
|
||||
if context.is_offline_mode():
|
||||
run_migrations_offline()
|
||||
else:
|
||||
run_migrations_online()
|
||||
|
|
@ -1,28 +0,0 @@
|
|||
<%%doc>Template for rendering a Multiple Migration Revision Identifier.</%%doc>
|
||||
<%%-
|
||||
from alembic import context
|
||||
context.configure()
|
||||
-%>
|
||||
"""${message}
|
||||
|
||||
Revision ID: ${up_revision}
|
||||
Revises: ${down_revision | comma_n, trim}
|
||||
Create Date: ${create_date}
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
${imports if imports else ""}
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = ${repr(up_revision)}
|
||||
down_revision = ${repr(down_revision)}
|
||||
branch_labels = ${repr(branch_labels)}
|
||||
depends_on = ${repr(depends_on)}
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
${upgrades if upgrades else "pass"}
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
${downgrades if downgrades else "pass"}
|
||||
|
|
@ -1,149 +0,0 @@
|
|||
"""initial schema
|
||||
|
||||
Revision ID: 001_initial
|
||||
Revises:
|
||||
Create Date: 2026-05-18
|
||||
"""
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects.postgresql import UUID, TSVECTOR, ENUM
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '001_initial'
|
||||
down_revision = None
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# Enums
|
||||
op.execute("CREATE TYPE feed_source_type AS ENUM ('rss', 'gdel-t2', 'social', 'earthquake', 'disaster', 'weather', 'fire', 'satellite')")
|
||||
op.execute("CREATE TYPE event_source_type AS ENUM ('rss', 'gdel-t2', 'social', 'earthquake', 'disaster', 'weather', 'fire', 'satellite')")
|
||||
op.execute("CREATE TYPE sentiment_label AS ENUM ('positive', 'neutral', 'negative')")
|
||||
op.execute("CREATE TYPE entity_type AS ENUM ('person', 'organization', 'location', 'topic', 'asset')")
|
||||
op.execute("CREATE TYPE alert_type AS ENUM ('entity_mention', 'sentiment_shift', 'geo_proximity', 'keyword_match', 'threshold', 'anomaly')")
|
||||
op.execute("CREATE TYPE alert_severity AS ENUM ('low', 'medium', 'high', 'critical')")
|
||||
|
||||
# Extensions
|
||||
op.execute("CREATE EXTENSION IF NOT EXISTS postgis")
|
||||
op.execute("CREATE EXTENSION IF NOT EXISTS timescaledb")
|
||||
|
||||
# feed_sources
|
||||
op.create_table(
|
||||
'feed_sources',
|
||||
sa.Column('id', UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column('name', sa.String(256), nullable=False),
|
||||
sa.Column('source_type', sa.Enum('rss', 'gdel-t2', 'social', 'earthquake', 'disaster', 'weather', 'fire', 'satellite', name='feed_source_type'), nullable=False),
|
||||
sa.Column('url', sa.Text()),
|
||||
sa.Column('config', sa.JSON()),
|
||||
sa.Column('enabled', sa.Integer, server_default='1', nullable=False),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.func.now()),
|
||||
)
|
||||
|
||||
# events (will become hypertable)
|
||||
op.create_table(
|
||||
'events',
|
||||
sa.Column('id', UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column('source_type', sa.Enum('rss', 'gdel-t2', 'social', 'earthquake', 'disaster', 'weather', 'fire', 'satellite', name='event_source_type'), nullable=False, index=True),
|
||||
sa.Column('source_id', UUID(as_uuid=True)),
|
||||
sa.Column('title', sa.Text()),
|
||||
sa.Column('body', sa.Text()),
|
||||
sa.Column('url', sa.Text()),
|
||||
sa.Column('sentiment_score', sa.Float()),
|
||||
sa.Column('sentiment_label', sa.Enum('positive', 'neutral', 'negative', name='sentiment_label')),
|
||||
sa.Column('location_lat', sa.Float()),
|
||||
sa.Column('location_lon', sa.Float()),
|
||||
sa.Column('location_name', sa.String(512)),
|
||||
sa.Column('entities', sa.JSON()),
|
||||
sa.Column('tags', sa.JSON()),
|
||||
sa.Column('raw', sa.JSON()),
|
||||
sa.Column('ingested_at', sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
sa.Column('source_timestamp', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column('search_vector', TSVECTOR),
|
||||
)
|
||||
|
||||
# Convert events to TimescaleDB hypertable
|
||||
op.execute("SELECT create_hypertable('events', 'ingested_at', if_not_exists => TRUE)")
|
||||
|
||||
# GIN index for full-text search
|
||||
op.create_index('ix_events_search_vector', 'events', ['search_vector'], postgresql_using='gin')
|
||||
# Spatial index
|
||||
op.create_index('ix_events_location', 'events', ['location_lat', 'location_lon'])
|
||||
|
||||
# entities
|
||||
op.create_table(
|
||||
'entities',
|
||||
sa.Column('id', UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column('name', sa.String(512), nullable=False, index=True),
|
||||
sa.Column('entity_type', sa.Enum('person', 'organization', 'location', 'topic', 'asset', name='entity_type'), nullable=False),
|
||||
sa.Column('aliases', sa.JSON()),
|
||||
sa.Column('description', sa.Text()),
|
||||
sa.Column('metadata', sa.JSON()),
|
||||
sa.Column('location_lat', sa.Float()),
|
||||
sa.Column('location_lon', sa.Float()),
|
||||
sa.Column('event_count', sa.Integer, server_default='0'),
|
||||
sa.Column('first_seen', sa.DateTime(timezone=True), server_default=sa.func.now()),
|
||||
sa.Column('last_seen', sa.DateTime(timezone=True), server_default=sa.func.now()),
|
||||
)
|
||||
|
||||
# entity_events
|
||||
op.create_table(
|
||||
'entity_events',
|
||||
sa.Column('entity_id', UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column('event_id', UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column('relevance_score', sa.Float()),
|
||||
sa.Column('linked_at', sa.DateTime(timezone=True), server_default=sa.func.now()),
|
||||
)
|
||||
|
||||
# alerts
|
||||
op.create_table(
|
||||
'alerts',
|
||||
sa.Column('id', UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column('alert_type', sa.Enum('entity_mention', 'sentiment_shift', 'geo_proximity', 'keyword_match', 'threshold', 'anomaly', name='alert_type'), nullable=False),
|
||||
sa.Column('entity_id', UUID(as_uuid=True)),
|
||||
sa.Column('event_id', UUID(as_uuid=True)),
|
||||
sa.Column('severity', sa.Enum('low', 'medium', 'high', 'critical', name='alert_severity'), nullable=False),
|
||||
sa.Column('title', sa.Text(), nullable=False),
|
||||
sa.Column('message', sa.Text()),
|
||||
sa.Column('context', sa.JSON()),
|
||||
sa.Column('acknowledged', sa.Integer, server_default='0'),
|
||||
sa.Column('acknowledged_by', sa.String(256)),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
sa.Column('resolved_at', sa.DateTime(timezone=True)),
|
||||
)
|
||||
op.create_index('ix_alerts_severity_created', 'alerts', ['severity', 'created_at'])
|
||||
op.create_index('ix_alerts_entity', 'alerts', ['entity_id'])
|
||||
|
||||
# documents
|
||||
op.create_table(
|
||||
'documents',
|
||||
sa.Column('id', UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column('bucket', sa.String(256), nullable=False),
|
||||
sa.Column('object_key', sa.String(1024), nullable=False),
|
||||
sa.Column('content_type', sa.String(256)),
|
||||
sa.Column('size_bytes', sa.Integer()),
|
||||
sa.Column('description', sa.Text()),
|
||||
sa.Column('tags', sa.JSON()),
|
||||
sa.Column('event_id', UUID(as_uuid=True)),
|
||||
sa.Column('uploaded_at', sa.DateTime(timezone=True), server_default=sa.func.now()),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table('documents')
|
||||
op.drop_table('alerts')
|
||||
op.drop_table('entity_events')
|
||||
op.drop_table('entities')
|
||||
op.execute("SELECT drop_hypertable('events', cascade => TRUE)")
|
||||
op.drop_table('events')
|
||||
op.drop_table('feed_sources')
|
||||
|
||||
# Drop enums
|
||||
op.execute("DROP TYPE IF EXISTS feed_source_type")
|
||||
op.execute("DROP TYPE IF EXISTS event_source_type")
|
||||
op.execute("DROP TYPE IF EXISTS sentiment_label")
|
||||
op.execute("DROP TYPE IF EXISTS entity_type")
|
||||
op.execute("DROP TYPE IF EXISTS alert_type")
|
||||
op.execute("DROP TYPE IF EXISTS alert_severity")
|
||||
|
|
@ -1,28 +0,0 @@
|
|||
import os
|
||||
|
||||
from sqlalchemy import MetaData, event, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
|
||||
|
||||
DB_USER = os.getenv("DB_USER", "osint")
|
||||
DB_PASS = os.getenv("DB_PASSWORD", "")
|
||||
DB_HOST = os.getenv("DB_HOST", "osint-pgdb-rw.customer1.svc.cluster.local")
|
||||
DB_PORT = os.getenv("DB_PORT", "5432")
|
||||
DB_NAME = os.getenv("DB_NAME", "osint_data")
|
||||
|
||||
DATABASE_URL = f"postgresql+asyncpg://{DB_USER}:{DB_PASS}@{DB_HOST}:{DB_PORT}/{DB_NAME}"
|
||||
|
||||
engine = create_async_engine(
|
||||
DATABASE_URL, echo=False, pool_size=5, max_overflow=10, pool_recycle=300
|
||||
)
|
||||
async_session = async_sessionmaker(
|
||||
engine, class_=AsyncSession, expire_on_commit=False
|
||||
)
|
||||
metadata = MetaData()
|
||||
|
||||
|
||||
async def init_extensions():
|
||||
"""Initialize PostGIS and TimescaleDB extensions on first connection."""
|
||||
async with engine.connect() as conn:
|
||||
await conn.execute(text("CREATE EXTENSION IF NOT EXISTS postgis"))
|
||||
await conn.execute(text("CREATE EXTENSION IF NOT EXISTS timescaledb"))
|
||||
await conn.commit()
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
"""CronJob entry point for scheduled ingestion."""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Add app dir to path
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
|
||||
from sources import ingest_rss_feed, ingest_gdelt, ingest_earthquakes
|
||||
from ingestor import fetch_and_process
|
||||
|
||||
INGESTOR_TYPE = os.getenv("INGESTOR_TYPE", "rss")
|
||||
RSS_URL = os.getenv("RSS_URL", "")
|
||||
|
||||
|
||||
async def main():
|
||||
print(f"Starting ingester: {INGESTOR_TYPE}")
|
||||
|
||||
if INGESTOR_TYPE == "rss":
|
||||
if not RSS_URL:
|
||||
print("No RSS_URL set, skipping")
|
||||
return
|
||||
count = await ingest_rss_feed(RSS_URL)
|
||||
print(f"RSS: ingested {count} items")
|
||||
|
||||
elif INGESTOR_TYPE == "gdelt":
|
||||
count = await ingest_gdelt(max_articles=50)
|
||||
print(f"GDELT: ingested {count} articles")
|
||||
|
||||
elif INGESTOR_TYPE == "earthquake":
|
||||
count = await ingest_earthquakes()
|
||||
print(f"Earthquakes: ingested {count} events")
|
||||
|
||||
elif INGESTOR_TYPE == "nats":
|
||||
count = await fetch_and_process(batch_size=500)
|
||||
print(f"NATS: processed {count} messages")
|
||||
|
||||
else:
|
||||
print(f"Unknown ingestor type: {INGESTOR_TYPE}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
|
|
@ -1,107 +0,0 @@
|
|||
"""NATS JetStream consumer — ingests OSINT events from NATS streams."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import nats
|
||||
from nats.errors import TimeoutError
|
||||
|
||||
from database import async_session
|
||||
from models import events as events_table
|
||||
|
||||
logger = logging.getLogger("osint.ingestor")
|
||||
|
||||
# NATS connection settings
|
||||
NATS_URLS = "nats://osint-nats.customer1.svc.cluster.local:4222"
|
||||
NATS_STREAM = "events"
|
||||
NATS_DURABLE = "osint-ingestor"
|
||||
|
||||
# Redis cache settings
|
||||
REDIS_URL = "redis://osint-redis-sentinel.customer1.svc.cluster.local:26379/0"
|
||||
|
||||
|
||||
async def ingest_event(msg: dict):
|
||||
"""Ingest a single event from NATS into PostgreSQL."""
|
||||
event_row = {
|
||||
"source_type": msg.get("source_type", "rss"),
|
||||
"source_id": msg.get("source_id"),
|
||||
"title": msg.get("title"),
|
||||
"body": msg.get("body"),
|
||||
"url": msg.get("url"),
|
||||
"sentiment_score": msg.get("sentiment_score"),
|
||||
"sentiment_label": msg.get("sentiment_label"),
|
||||
"location_lat": msg.get("location_lat"),
|
||||
"location_lon": msg.get("location_lon"),
|
||||
"location_name": msg.get("location_name"),
|
||||
"entities": msg.get("entities", []),
|
||||
"tags": msg.get("tags", []),
|
||||
"raw": msg.get("raw"),
|
||||
"source_timestamp": msg.get("source_timestamp", datetime.now(timezone.utc).isoformat()),
|
||||
}
|
||||
|
||||
# Parse timestamp if string
|
||||
if isinstance(event_row["source_timestamp"], str):
|
||||
event_row["source_timestamp"] = datetime.fromisoformat(event_row["source_timestamp"])
|
||||
|
||||
async with async_session() as session:
|
||||
result = await session.execute(events_table.insert().values(**event_row))
|
||||
await session.commit()
|
||||
event_id = result.inserted_primary_key[0] # type: ignore[union-attr]
|
||||
logger.info("Ingested event %s from source %s", event_id, msg.get("source_type"))
|
||||
return event_id
|
||||
|
||||
|
||||
async def start_nats_consumer():
|
||||
"""Start NATS JetStream consumer for OSINT events."""
|
||||
nc = await nats.connect(NATS_URLS)
|
||||
js = nc.jetstream()
|
||||
|
||||
# Create stream if not exists
|
||||
try:
|
||||
await js.add_stream(
|
||||
name=NATS_STREAM,
|
||||
subjects=[
|
||||
"events.gdelt", "events.rss", "events.social",
|
||||
"events.earthquake", "events.disaster", "events.weather",
|
||||
"events.fire", "events.satellite", "events.new", "events.alert",
|
||||
],
|
||||
retention=nats.js.api.RetentionPolicy.INTERESTS,
|
||||
max_msgs=1_000_000,
|
||||
)
|
||||
logger.info("Created NATS stream %s", NATS_STREAM)
|
||||
except Exception:
|
||||
logger.debug("Stream %s already exists", NATS_STREAM)
|
||||
|
||||
# Create durable consumer
|
||||
sub = await js.pull_subscribe(
|
||||
subject="events.>",
|
||||
durable_name=NATS_DURABLE,
|
||||
)
|
||||
|
||||
logger.info("NATS consumer started, durable=%s", NATS_DURABLE)
|
||||
return nc, sub
|
||||
|
||||
|
||||
async def fetch_and_process(batch_size: int = 100):
|
||||
"""Fetch a batch of messages and process them."""
|
||||
nc, sub = await start_nats_consumer()
|
||||
js = nc.jetstream()
|
||||
|
||||
msgs = await sub.fetch(batch_size, timeout=5)
|
||||
processed = 0
|
||||
|
||||
for msg in msgs:
|
||||
try:
|
||||
data = json.loads(msg.data)
|
||||
await ingest_event(data)
|
||||
await msg.ack()
|
||||
processed += 1
|
||||
except Exception:
|
||||
logger.error("Failed to process message: %s", msg.data, exc_info=True)
|
||||
|
||||
await nc.close()
|
||||
logger.info("Processed %d messages in batch", processed)
|
||||
return processed
|
||||
|
|
@ -1,603 +0,0 @@
|
|||
"""OSINT Dashboard — FastAPI backend.
|
||||
|
||||
Real-time geospatial OSINT dashboard API:
|
||||
- Event ingestion via NATS JetStream consumers
|
||||
- Full-text search across events (PostgreSQL tsvector)
|
||||
- Entity tracking and relationship mapping
|
||||
- Alert management
|
||||
- Document storage (MinIO-backed)
|
||||
- Sentiment aggregation and timeline analytics
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
from uuid import UUID
|
||||
|
||||
import structlog
|
||||
from fastapi import FastAPI, HTTPException, Query
|
||||
from fastapi.responses import FileResponse, HTMLResponse
|
||||
from sqlalchemy import and_, func, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from database import async_session, init_extensions
|
||||
from models import (
|
||||
alerts, documents, entities, entity_events, events, feed_sources
|
||||
)
|
||||
from schemas import (
|
||||
AlertCreate, AlertOut, AlertSeverity, AlertType, AlertUpdate,
|
||||
DashboardSummary, EntityCreate, EntityKind, EntityOut,
|
||||
EventCreate, EventOut,
|
||||
FeedSourceCreate, FeedSourceOut,
|
||||
SearchResult, SentimentSummary, SourceType,
|
||||
SearchQuery, TimelinePoint,
|
||||
)
|
||||
from ingestor import ingest_event, fetch_and_process
|
||||
from sources import ingest_rss_feed, ingest_gdelt, ingest_earthquakes, ingest_social_signals
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = structlog.get_logger("osint.dashboard")
|
||||
|
||||
app = FastAPI(
|
||||
title="OSINT Dashboard",
|
||||
description="Real-time geospatial OSINT intelligence dashboard",
|
||||
version="0.1.0",
|
||||
)
|
||||
|
||||
STATIC_DIR = Path(__file__).parent / "static"
|
||||
|
||||
|
||||
# ── Helpers ───────────────────────────────────────────────────────────────
|
||||
|
||||
def event_to_out(row: dict) -> EventOut:
|
||||
"""Convert DB row dict to EventOut schema."""
|
||||
return EventOut(
|
||||
id=row["id"],
|
||||
source_type=row["source_type"],
|
||||
source_id=row["source_id"],
|
||||
title=row["title"],
|
||||
body=row["body"],
|
||||
url=row["url"],
|
||||
sentiment_score=row["sentiment_score"],
|
||||
sentiment_label=row["sentiment_label"],
|
||||
location_lat=row["location_lat"],
|
||||
location_lon=row["location_lon"],
|
||||
location_name=row["location_name"],
|
||||
entities=row["entities"],
|
||||
tags=row["tags"],
|
||||
ingested_at=row["ingested_at"],
|
||||
source_timestamp=row["source_timestamp"],
|
||||
)
|
||||
|
||||
|
||||
def entity_to_out(row: dict) -> EntityOut:
|
||||
"""Convert DB row dict to EntityOut schema."""
|
||||
return EntityOut(
|
||||
id=row["id"],
|
||||
name=row["name"],
|
||||
entity_type=row["entity_type"],
|
||||
aliases=row["aliases"],
|
||||
description=row["description"],
|
||||
metadata=row["metadata"],
|
||||
location_lat=row["location_lat"],
|
||||
location_lon=row["location_lon"],
|
||||
event_count=row["event_count"],
|
||||
first_seen=row["first_seen"],
|
||||
last_seen=row["last_seen"],
|
||||
)
|
||||
|
||||
|
||||
def alert_to_out(row: dict) -> AlertOut:
|
||||
"""Convert DB row dict to AlertOut schema."""
|
||||
return AlertOut(
|
||||
id=row["id"],
|
||||
alert_type=row["alert_type"],
|
||||
entity_id=row["entity_id"],
|
||||
event_id=row["event_id"],
|
||||
severity=row["severity"],
|
||||
title=row["title"],
|
||||
message=row["message"],
|
||||
context=row["context"],
|
||||
acknowledged=bool(row["acknowledged"]),
|
||||
acknowledged_by=row["acknowledged_by"],
|
||||
created_at=row["created_at"],
|
||||
resolved_at=row["resolved_at"],
|
||||
)
|
||||
|
||||
|
||||
# ── Health ────────────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/api/health")
|
||||
async def health():
|
||||
"""Health check with database connectivity."""
|
||||
async with async_session() as session:
|
||||
result = await session.execute(select(func.now()))
|
||||
db_time = result.scalar()
|
||||
return {"status": "ok", "db_time": db_time.isoformat() if db_time else None}
|
||||
|
||||
|
||||
# ── Startup ───────────────────────────────────────────────────────────────
|
||||
|
||||
@app.on_event("startup")
|
||||
async def startup():
|
||||
"""Initialize extensions and run migrations."""
|
||||
await init_extensions()
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
alembic_cfg = Config(str(Path(__file__).parent.parent / "alembic.ini"))
|
||||
command.upgrade(alembic_cfg, "head")
|
||||
|
||||
|
||||
# ── Feed Sources ──────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/api/sources", response_model=list[FeedSourceOut])
|
||||
async def list_sources(enabled_only: bool = Query(True)):
|
||||
"""List all configured feed sources."""
|
||||
async with async_session() as session:
|
||||
stmt = select(feed_sources).order_by(feed_sources.c.name)
|
||||
if enabled_only:
|
||||
stmt = stmt.where(feed_sources.c.enabled == 1)
|
||||
rows = (await session.execute(stmt)).mappings().all()
|
||||
return [FeedSourceOut(
|
||||
id=r["id"], name=r["name"], source_type=r["source_type"],
|
||||
url=r["url"], config=r["config"], enabled=bool(r["enabled"]),
|
||||
created_at=r["created_at"],
|
||||
) for r in rows]
|
||||
|
||||
|
||||
@app.post("/api/sources", status_code=201)
|
||||
async def create_source(payload: FeedSourceCreate):
|
||||
"""Add a new feed source."""
|
||||
async with async_session() as session:
|
||||
values = payload.model_dump()
|
||||
result = await session.execute(feed_sources.insert().values(**values))
|
||||
await session.commit()
|
||||
pk = result.inserted_primary_key[0] # type: ignore
|
||||
return {"id": str(pk)}
|
||||
|
||||
|
||||
@app.patch("/api/sources/{source_id}")
|
||||
async def update_source(source_id: UUID, payload: dict):
|
||||
"""Update a feed source (e.g., toggle enabled)."""
|
||||
async with async_session() as session:
|
||||
row = (await session.execute(
|
||||
select(feed_sources).where(feed_sources.c.id == source_id)
|
||||
)).mappings().one_or_none()
|
||||
if not row:
|
||||
raise HTTPException(404, "Source not found")
|
||||
await session.execute(
|
||||
feed_sources.update()
|
||||
.where(feed_sources.c.id == source_id)
|
||||
.values(**payload)
|
||||
)
|
||||
await session.commit()
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
# ── Events ────────────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/api/events", response_model=list[EventOut])
|
||||
async def list_events(
|
||||
source_type: SourceType | None = Query(None),
|
||||
limit: int = Query(50, ge=1, le=500),
|
||||
offset: int = Query(0, ge=0),
|
||||
):
|
||||
"""List recent ingested events."""
|
||||
async with async_session() as session:
|
||||
stmt = select(events).order_by(events.c.ingested_at.desc())
|
||||
if source_type:
|
||||
stmt = stmt.where(events.c.source_type == source_type.value)
|
||||
stmt = stmt.limit(limit).offset(offset)
|
||||
rows = (await session.execute(stmt)).mappings().all()
|
||||
return [event_to_out(r) for r in rows]
|
||||
|
||||
|
||||
@app.get("/api/events/{event_id}", response_model=EventOut)
|
||||
async def get_event(event_id: UUID):
|
||||
"""Get a single event by ID."""
|
||||
async with async_session() as session:
|
||||
row = (await session.execute(
|
||||
select(events).where(events.c.id == event_id)
|
||||
)).mappings().one_or_none()
|
||||
if not row:
|
||||
raise HTTPException(404, "Event not found")
|
||||
return event_to_out(row)
|
||||
|
||||
|
||||
@app.post("/api/events", status_code=201)
|
||||
async def create_event(payload: EventCreate):
|
||||
"""Manually ingest an event (bypasses NATS)."""
|
||||
values = payload.model_dump(exclude_unset=True)
|
||||
if not values.get("source_timestamp"):
|
||||
values["source_timestamp"] = datetime.now(timezone.utc)
|
||||
event_id = await ingest_event(values)
|
||||
return {"id": str(event_id)}
|
||||
|
||||
|
||||
# ── Search ────────────────────────────────────────────────────────────────
|
||||
|
||||
@app.post("/api/search", response_model=SearchResult)
|
||||
async def search_events(query: SearchQuery):
|
||||
"""Full-text search across events with optional filters."""
|
||||
async with async_session() as session:
|
||||
# Build query with tsvector full-text search (parameterized to avoid SQL injection)
|
||||
tsquery_param = text("plainto_tsquery('english', :q)")
|
||||
|
||||
base_stmt = select(
|
||||
events,
|
||||
func.count().over().label("total")
|
||||
).where(
|
||||
events.c.search_vector.op("@@")(tsquery_param)
|
||||
)
|
||||
|
||||
# Apply filters
|
||||
if query.source_type:
|
||||
base_stmt = base_stmt.where(events.c.source_type == query.source_type.value)
|
||||
if query.entity_id:
|
||||
base_stmt = base_stmt.join(
|
||||
entity_events, entity_events.c.event_id == events.c.id
|
||||
).where(entity_events.c.entity_id == query.entity_id)
|
||||
if query.sentiment:
|
||||
base_stmt = base_stmt.where(events.c.sentiment_label == query.sentiment.value)
|
||||
if query.min_date:
|
||||
base_stmt = base_stmt.where(events.c.source_timestamp >= query.min_date)
|
||||
if query.max_date:
|
||||
base_stmt = base_stmt.where(events.c.source_timestamp <= query.max_date)
|
||||
if query.min_lat is not None and query.max_lat is not None:
|
||||
base_stmt = base_stmt.where(
|
||||
and_(
|
||||
events.c.location_lat >= query.min_lat,
|
||||
events.c.location_lat <= query.max_lat,
|
||||
)
|
||||
)
|
||||
if query.min_lon is not None and query.max_lon is not None:
|
||||
base_stmt = base_stmt.where(
|
||||
and_(
|
||||
events.c.location_lon >= query.min_lon,
|
||||
events.c.location_lon <= query.max_lon,
|
||||
)
|
||||
)
|
||||
|
||||
base_stmt = base_stmt.order_by(events.c.ingested_at.desc())
|
||||
base_stmt = base_stmt.limit(query.limit).offset(query.offset)
|
||||
|
||||
result = (await session.execute(base_stmt, {"q": query.q})).mappings().all()
|
||||
if result:
|
||||
total = result[0]["total"]
|
||||
else:
|
||||
total = 0
|
||||
evts = [event_to_out(r) for r in result]
|
||||
|
||||
return SearchResult(
|
||||
events=evts,
|
||||
total=total,
|
||||
has_more=query.offset + len(evts) < total,
|
||||
)
|
||||
|
||||
|
||||
# ── Entities ──────────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/api/entities", response_model=list[EntityOut])
|
||||
async def list_entities(
|
||||
entity_type: EntityKind | None = Query(None),
|
||||
limit: int = Query(50, ge=1, le=500),
|
||||
):
|
||||
"""List tracked entities."""
|
||||
async with async_session() as session:
|
||||
stmt = select(entities).order_by(entities.c.event_count.desc())
|
||||
if entity_type:
|
||||
stmt = stmt.where(entities.c.entity_type == entity_type.value)
|
||||
stmt = stmt.limit(limit)
|
||||
rows = (await session.execute(stmt)).mappings().all()
|
||||
return [entity_to_out(r) for r in rows]
|
||||
|
||||
|
||||
@app.get("/api/entities/{entity_id}", response_model=EntityOut)
|
||||
async def get_entity(entity_id: UUID):
|
||||
"""Get entity details with recent events."""
|
||||
async with async_session() as session:
|
||||
row = (await session.execute(
|
||||
select(entities).where(entities.c.id == entity_id)
|
||||
)).mappings().one_or_none()
|
||||
if not row:
|
||||
raise HTTPException(404, "Entity not found")
|
||||
return entity_to_out(row)
|
||||
|
||||
|
||||
@app.post("/api/entities", status_code=201)
|
||||
async def create_entity(payload: EntityCreate):
|
||||
"""Create or update a tracked entity."""
|
||||
async with async_session() as session:
|
||||
# Check if entity already exists by name
|
||||
existing = (await session.execute(
|
||||
select(entities).where(entities.c.name == payload.name)
|
||||
)).mappings().one_or_none()
|
||||
|
||||
if existing:
|
||||
# Update
|
||||
updates = payload.model_dump(exclude_unset=True)
|
||||
updates["last_seen"] = datetime.now(timezone.utc)
|
||||
await session.execute(
|
||||
entities.update()
|
||||
.where(entities.c.id == existing["id"])
|
||||
.values(**updates)
|
||||
)
|
||||
await session.commit()
|
||||
return {"id": str(existing["id"]), "created": False}
|
||||
|
||||
# Create
|
||||
values = payload.model_dump()
|
||||
result = await session.execute(entities.insert().values(**values))
|
||||
await session.commit()
|
||||
pk = result.inserted_primary_key[0] # type: ignore
|
||||
return {"id": str(pk), "created": True}
|
||||
|
||||
|
||||
@app.get("/api/entities/{entity_id}/events", response_model=list[EventOut])
|
||||
async def get_entity_events(
|
||||
entity_id: UUID,
|
||||
limit: int = Query(50, ge=1, le=500),
|
||||
):
|
||||
"""Get events linked to a specific entity."""
|
||||
async with async_session() as session:
|
||||
stmt = (
|
||||
select(events)
|
||||
.join(entity_events, entity_events.c.event_id == events.c.id)
|
||||
.where(entity_events.c.entity_id == entity_id)
|
||||
.order_by(events.c.source_timestamp.desc())
|
||||
.limit(limit)
|
||||
)
|
||||
rows = (await session.execute(stmt)).mappings().all()
|
||||
return [event_to_out(r) for r in rows]
|
||||
|
||||
|
||||
# ── Alerts ────────────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/api/alerts", response_model=list[AlertOut])
|
||||
async def list_alerts(
|
||||
severity: AlertSeverity | None = Query(None),
|
||||
acknowledged: bool | None = Query(None),
|
||||
entity_id: UUID | None = Query(None),
|
||||
limit: int = Query(50, ge=1, le=500),
|
||||
):
|
||||
"""List alerts with optional filters."""
|
||||
async with async_session() as session:
|
||||
stmt = select(alerts).order_by(
|
||||
alerts.c.severity.desc(), alerts.c.created_at.desc()
|
||||
)
|
||||
if severity:
|
||||
stmt = stmt.where(alerts.c.severity == severity.value)
|
||||
if acknowledged is not None:
|
||||
stmt = stmt.where(alerts.c.acknowledged == int(acknowledged))
|
||||
if entity_id:
|
||||
stmt = stmt.where(alerts.c.entity_id == entity_id)
|
||||
stmt = stmt.limit(limit)
|
||||
rows = (await session.execute(stmt)).mappings().all()
|
||||
return [alert_to_out(r) for r in rows]
|
||||
|
||||
|
||||
@app.post("/api/alerts", status_code=201)
|
||||
async def create_alert(payload: AlertCreate):
|
||||
"""Create a new alert."""
|
||||
async with async_session() as session:
|
||||
values = payload.model_dump()
|
||||
result = await session.execute(alerts.insert().values(**values))
|
||||
await session.commit()
|
||||
pk = result.inserted_primary_key[0] # type: ignore
|
||||
return {"id": str(pk)}
|
||||
|
||||
|
||||
@app.patch("/api/alerts/{alert_id}")
|
||||
async def update_alert(alert_id: UUID, payload: AlertUpdate):
|
||||
"""Update alert (acknowledge, resolve)."""
|
||||
async with async_session() as session:
|
||||
row = (await session.execute(
|
||||
select(alerts).where(alerts.c.id == alert_id)
|
||||
)).mappings().one_or_none()
|
||||
if not row:
|
||||
raise HTTPException(404, "Alert not found")
|
||||
updates = payload.model_dump(exclude_unset=True)
|
||||
if "acknowledged" in updates:
|
||||
updates["acknowledged"] = int(updates["acknowledged"])
|
||||
await session.execute(
|
||||
alerts.update().where(alerts.c.id == alert_id).values(**updates)
|
||||
)
|
||||
await session.commit()
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
# ── Documents ─────────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/api/documents", response_model=dict)
|
||||
async def list_documents(
|
||||
limit: int = Query(50, ge=1, le=500),
|
||||
offset: int = Query(0, ge=0),
|
||||
):
|
||||
"""List documents indexed in MinIO."""
|
||||
async with async_session() as session:
|
||||
stmt = select(documents).order_by(documents.c.uploaded_at.desc()).limit(limit).offset(offset)
|
||||
rows = (await session.execute(stmt)).mappings().all()
|
||||
return {
|
||||
"documents": [{
|
||||
"id": str(r["id"]), "bucket": r["bucket"], "object_key": r["object_key"],
|
||||
"content_type": r["content_type"], "size_bytes": r["size_bytes"],
|
||||
"description": r["description"], "tags": r["tags"],
|
||||
"event_id": str(r["event_id"]) if r["event_id"] else None,
|
||||
"uploaded_at": r["uploaded_at"].isoformat() if r["uploaded_at"] else None,
|
||||
} for r in rows],
|
||||
}
|
||||
|
||||
|
||||
# ── Ingestion Triggers ───────────────────────────────────────────────────
|
||||
|
||||
@app.post("/api/ingest/rss")
|
||||
async def trigger_rss_ingest(feed_url: str, source_id: str | None = None):
|
||||
"""Trigger RSS feed ingestion."""
|
||||
count = await ingest_rss_feed(feed_url, source_id)
|
||||
return {"status": "ok", "items_ingested": count}
|
||||
|
||||
|
||||
@app.post("/api/ingest/gdelt")
|
||||
async def trigger_gdelt_ingest(query: str = "", max_articles: int = 50):
|
||||
"""Trigger GDELT data ingestion."""
|
||||
count = await ingest_gdelt(query, max_articles)
|
||||
return {"status": "ok", "articles_ingested": count}
|
||||
|
||||
|
||||
@app.post("/api/ingest/earthquakes")
|
||||
async def trigger_earthquake_ingest():
|
||||
"""Trigger USGS earthquake ingestion."""
|
||||
count = await ingest_earthquakes()
|
||||
return {"status": "ok", "events_ingested": count}
|
||||
|
||||
|
||||
@app.post("/api/ingest/social")
|
||||
async def trigger_social_ingest(query: str = "", max_items: int = 50):
|
||||
"""Trigger social signals ingestion."""
|
||||
count = await ingest_social_signals(query, max_items)
|
||||
return {"status": "ok", "signals_ingested": count}
|
||||
|
||||
|
||||
@app.post("/api/ingest/process")
|
||||
async def trigger_nats_processing(batch_size: int = 100):
|
||||
"""Process pending NATS JetStream messages."""
|
||||
count = await fetch_and_process(batch_size)
|
||||
return {"status": "ok", "processed": count}
|
||||
|
||||
|
||||
# ── Analytics / Aggregation ───────────────────────────────────────────────
|
||||
|
||||
@app.get("/api/analytics/summary", response_model=DashboardSummary)
|
||||
async def get_dashboard_summary():
|
||||
"""Dashboard overview: event counts, sentiment, top entities, alerts."""
|
||||
async with async_session() as session:
|
||||
now = datetime.now(timezone.utc)
|
||||
yesterday = now - timedelta(hours=24)
|
||||
|
||||
# Total events
|
||||
total = (await session.execute(
|
||||
select(func.count()).select_from(events)
|
||||
)).scalar() or 0
|
||||
|
||||
# Events in last 24h
|
||||
events_24h = (await session.execute(
|
||||
select(func.count()).where(events.c.ingested_at >= yesterday)
|
||||
)).scalar() or 0
|
||||
|
||||
# Active sources
|
||||
active = (await session.execute(
|
||||
select(func.count()).where(feed_sources.c.enabled == 1)
|
||||
)).scalar() or 0
|
||||
|
||||
# Open alerts
|
||||
open_alerts = (await session.execute(
|
||||
select(func.count()).where(alerts.c.acknowledged == 0)
|
||||
)).scalar() or 0
|
||||
|
||||
# Tracked entities
|
||||
ent_count = (await session.execute(
|
||||
select(func.count()).select_from(entities)
|
||||
)).scalar() or 0
|
||||
|
||||
# Sentiment breakdown (last 24h)
|
||||
def sentiment_query():
|
||||
return select(
|
||||
func.count().where(events.c.sentiment_label == "positive").label("pos"),
|
||||
func.count().where(events.c.sentiment_label == "neutral").label("neu"),
|
||||
func.count().where(events.c.sentiment_label == "negative").label("neg"),
|
||||
func.avg(events.c.sentiment_score).label("avg"),
|
||||
).where(events.c.ingested_at >= yesterday)
|
||||
|
||||
sent_row = (await session.execute(sentiment_query())).mappings().one()
|
||||
sentiment = SentimentSummary(
|
||||
period="24h",
|
||||
positive_count=sent_row["pos"] or 0,
|
||||
neutral_count=sent_row["neu"] or 0,
|
||||
negative_count=sent_row["neg"] or 0,
|
||||
avg_score=float(sent_row["avg"] or 0),
|
||||
)
|
||||
|
||||
# Top entities by event count
|
||||
top_ent = (await session.execute(
|
||||
select(entities).order_by(entities.c.event_count.desc()).limit(10)
|
||||
)).mappings().all()
|
||||
|
||||
return DashboardSummary(
|
||||
total_events=total,
|
||||
events_last_24h=events_24h,
|
||||
active_sources=active,
|
||||
open_alerts=open_alerts,
|
||||
tracked_entities=ent_count,
|
||||
sentiment=sentiment,
|
||||
top_entities=[entity_to_out(r) for r in top_ent],
|
||||
)
|
||||
|
||||
|
||||
@app.get("/api/analytics/timeline")
|
||||
async def get_timeline(
|
||||
hours: int = Query(24, ge=1, le=168),
|
||||
bucket_hours: int = Query(1, ge=1, le=24),
|
||||
):
|
||||
"""Event timeline: counts and avg sentiment per time bucket."""
|
||||
async with async_session() as session:
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(hours=hours)
|
||||
# Use date_trunc for bucketing
|
||||
buckets = await session.execute(text(f"""
|
||||
SELECT
|
||||
date_trunc('hour', source_timestamp) AS ts,
|
||||
COUNT(*) AS event_count,
|
||||
COALESCE(AVG(sentiment_score), 0) AS avg_sentiment
|
||||
FROM events
|
||||
WHERE source_timestamp >= :cutoff
|
||||
GROUP BY ts
|
||||
ORDER BY ts
|
||||
"""), {"cutoff": cutoff})
|
||||
rows = buckets.mappings().all()
|
||||
|
||||
return [TimelinePoint(timestamp=r["ts"], event_count=r["event_count"],
|
||||
avg_sentiment=float(r["avg_sentiment"])) for r in rows]
|
||||
|
||||
|
||||
@app.get("/api/analytics/sentiment/by-source")
|
||||
async def sentiment_by_source(hours: int = 24):
|
||||
"""Sentiment breakdown grouped by source type."""
|
||||
async with async_session() as session:
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(hours=hours)
|
||||
result = await session.execute(text(f"""
|
||||
SELECT
|
||||
source_type,
|
||||
COUNT(*) AS total,
|
||||
COUNT(*) FILTER (WHERE sentiment_label = 'positive') AS positive,
|
||||
COUNT(*) FILTER (WHERE sentiment_label = 'neutral') AS neutral,
|
||||
COUNT(*) FILTER (WHERE sentiment_label = 'negative') AS negative,
|
||||
COALESCE(AVG(sentiment_score), 0) AS avg_score
|
||||
FROM events
|
||||
WHERE ingested_at >= :cutoff
|
||||
GROUP BY source_type
|
||||
ORDER BY total DESC
|
||||
"""), {"cutoff": cutoff})
|
||||
rows = result.mappings().all()
|
||||
return [{
|
||||
"source_type": r["source_type"],
|
||||
"total": r["total"],
|
||||
"positive": r["positive"],
|
||||
"neutral": r["neutral"],
|
||||
"negative": r["negative"],
|
||||
"avg_score": float(r["avg_score"]),
|
||||
} for r in rows]
|
||||
|
||||
|
||||
# ── Frontend ──────────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/", response_class=HTMLResponse)
|
||||
async def index():
|
||||
return FileResponse(str(STATIC_DIR / "index.html"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=8000)
|
||||
|
|
@ -1,147 +0,0 @@
|
|||
"""OSINT Dashboard — SQLAlchemy models (async, declarative)."""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from uuid import uuid4
|
||||
|
||||
from sqlalchemy import (
|
||||
Column, Enum, Float, Index, Integer, String, Text,
|
||||
DateTime, JSON, func, Table,
|
||||
)
|
||||
from sqlalchemy.dialects.postgresql import UUID, TSVECTOR
|
||||
|
||||
from database import metadata
|
||||
|
||||
|
||||
# ── Feed Sources ──────────────────────────────────────────────────────────
|
||||
|
||||
feed_sources = Table(
|
||||
"feed_sources",
|
||||
metadata,
|
||||
Column("id", UUID(as_uuid=True), primary_key=True, default=uuid4),
|
||||
Column("name", String(256), nullable=False),
|
||||
Column("source_type", Enum(
|
||||
"rss", "gdel-t2", "social", "earthquake", "disaster",
|
||||
"weather", "fire", "satellite", name="feed_source_type"
|
||||
), nullable=False),
|
||||
Column("url", Text),
|
||||
Column("config", JSON),
|
||||
Column("enabled", Integer, server_default="1", nullable=False),
|
||||
Column("created_at", DateTime(timezone=True), server_default=func.now(), nullable=False),
|
||||
Column("updated_at", DateTime(timezone=True), server_default=func.now(), onupdate=func.now()),
|
||||
)
|
||||
|
||||
|
||||
# ── Events (hypertable via TimescaleDB) ──────────────────────────────────
|
||||
|
||||
events = Table(
|
||||
"events",
|
||||
metadata,
|
||||
Column("id", UUID(as_uuid=True), primary_key=True, default=uuid4),
|
||||
Column("source_type", Enum(
|
||||
"rss", "gdel-t2", "social", "earthquake", "disaster",
|
||||
"weather", "fire", "satellite", name="event_source_type"
|
||||
), nullable=False, index=True),
|
||||
Column("source_id", UUID(as_uuid=True)),
|
||||
Column("title", Text),
|
||||
Column("body", Text),
|
||||
Column("url", Text),
|
||||
Column("sentiment_score", Float),
|
||||
Column("sentiment_label", Enum("positive", "neutral", "negative", name="sentiment_label")),
|
||||
Column("location_lat", Float),
|
||||
Column("location_lon", Float),
|
||||
Column("location_name", String(512)),
|
||||
Column("entities", JSON),
|
||||
Column("tags", JSON),
|
||||
Column("raw", JSON),
|
||||
Column("ingested_at", DateTime(timezone=True), server_default=func.now(), nullable=False),
|
||||
Column("source_timestamp", DateTime(timezone=True), nullable=False),
|
||||
# Full-text search vector
|
||||
Column(
|
||||
"search_vector",
|
||||
TSVECTOR,
|
||||
nullable=True,
|
||||
),
|
||||
)
|
||||
|
||||
# GIN index for full-text search
|
||||
Index("ix_events_search_vector", events.c.search_vector, postgresql_using="gin")
|
||||
# Spatial index on location
|
||||
Index("ix_events_location", events.c.location_lat, events.c.location_lon)
|
||||
|
||||
|
||||
# ── Entities (people, organizations, locations of interest) ──────────────
|
||||
|
||||
entities = Table(
|
||||
"entities",
|
||||
metadata,
|
||||
Column("id", UUID(as_uuid=True), primary_key=True, default=uuid4),
|
||||
Column("name", String(512), nullable=False, index=True),
|
||||
Column("entity_type", Enum(
|
||||
"person", "organization", "location", "topic", "asset",
|
||||
name="entity_type"
|
||||
), nullable=False),
|
||||
Column("aliases", JSON),
|
||||
Column("description", Text),
|
||||
Column("metadata", JSON),
|
||||
Column("location_lat", Float),
|
||||
Column("location_lon", Float),
|
||||
Column("event_count", Integer, server_default="0"),
|
||||
Column("first_seen", DateTime(timezone=True), server_default=func.now()),
|
||||
Column("last_seen", DateTime(timezone=True), server_default=func.now()),
|
||||
)
|
||||
|
||||
|
||||
# ── Entity-Event Link ────────────────────────────────────────────────────
|
||||
|
||||
entity_events = Table(
|
||||
"entity_events",
|
||||
metadata,
|
||||
Column("entity_id", UUID(as_uuid=True), primary_key=True),
|
||||
Column("event_id", UUID(as_uuid=True), primary_key=True),
|
||||
Column("relevance_score", Float),
|
||||
Column("linked_at", DateTime(timezone=True), server_default=func.now()),
|
||||
)
|
||||
|
||||
|
||||
# ── Alerts ───────────────────────────────────────────────────────────────
|
||||
|
||||
alerts = Table(
|
||||
"alerts",
|
||||
metadata,
|
||||
Column("id", UUID(as_uuid=True), primary_key=True, default=uuid4),
|
||||
Column("alert_type", Enum(
|
||||
"entity_mention", "sentiment_shift", "geo_proximity",
|
||||
"keyword_match", "threshold", "anomaly",
|
||||
name="alert_type"
|
||||
), nullable=False),
|
||||
Column("entity_id", UUID(as_uuid=True)),
|
||||
Column("event_id", UUID(as_uuid=True)),
|
||||
Column("severity", Enum("low", "medium", "high", "critical", name="alert_severity"), nullable=False),
|
||||
Column("title", Text, nullable=False),
|
||||
Column("message", Text),
|
||||
Column("context", JSON),
|
||||
Column("acknowledged", Integer, server_default="0"),
|
||||
Column("acknowledged_by", String(256)),
|
||||
Column("created_at", DateTime(timezone=True), server_default=func.now(), nullable=False),
|
||||
Column("resolved_at", DateTime(timezone=True)),
|
||||
)
|
||||
|
||||
Index("ix_alerts_severity_created", alerts.c.severity, alerts.c.created_at.desc())
|
||||
Index("ix_alerts_entity", alerts.c.entity_id)
|
||||
|
||||
|
||||
# ── Documents (stored in MinIO, indexed here) ────────────────────────────
|
||||
|
||||
documents = Table(
|
||||
"documents",
|
||||
metadata,
|
||||
Column("id", UUID(as_uuid=True), primary_key=True, default=uuid4),
|
||||
Column("bucket", String(256), nullable=False),
|
||||
Column("object_key", String(1024), nullable=False),
|
||||
Column("content_type", String(256)),
|
||||
Column("size_bytes", Integer),
|
||||
Column("description", Text),
|
||||
Column("tags", JSON),
|
||||
Column("event_id", UUID(as_uuid=True)),
|
||||
Column("uploaded_at", DateTime(timezone=True), server_default=func.now()),
|
||||
)
|
||||
|
|
@ -1,13 +0,0 @@
|
|||
fastapi>=0.115
|
||||
uvicorn[standard]>=0.34
|
||||
sqlalchemy[asyncio]>=2.0
|
||||
asyncpg>=0.30
|
||||
nats-py>=2.9
|
||||
redis[hiredis]>=5.2
|
||||
minio>=7.2
|
||||
pydantic>=2.10
|
||||
alembic>=1.14
|
||||
httpx>=0.28
|
||||
feedparser>=6.0
|
||||
python-dateutil>=2.9
|
||||
structlog>=24.4
|
||||
|
|
@ -1,242 +0,0 @@
|
|||
"""OSINT Dashboard — Pydantic schemas."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from typing import Optional
|
||||
from uuid import UUID
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
# ─── Enums ───────────────────────────────────────────────────────────────
|
||||
|
||||
class SourceType(str, Enum):
|
||||
rss = "rss"
|
||||
gdel_t2 = "gdel-t2"
|
||||
social = "social"
|
||||
earthquake = "earthquake"
|
||||
disaster = "disaster"
|
||||
weather = "weather"
|
||||
fire = "fire"
|
||||
satellite = "satellite"
|
||||
|
||||
|
||||
class EntityKind(str, Enum):
|
||||
person = "person"
|
||||
organization = "organization"
|
||||
location = "location"
|
||||
topic = "topic"
|
||||
asset = "asset"
|
||||
|
||||
|
||||
class Sentiment(str, Enum):
|
||||
positive = "positive"
|
||||
neutral = "neutral"
|
||||
negative = "negative"
|
||||
|
||||
|
||||
class AlertType(str, Enum):
|
||||
entity_mention = "entity_mention"
|
||||
sentiment_shift = "sentiment_shift"
|
||||
geo_proximity = "geo_proximity"
|
||||
keyword_match = "keyword_match"
|
||||
threshold = "threshold"
|
||||
anomaly = "anomaly"
|
||||
|
||||
|
||||
class AlertSeverity(str, Enum):
|
||||
low = "low"
|
||||
medium = "medium"
|
||||
high = "high"
|
||||
critical = "critical"
|
||||
|
||||
|
||||
# ─── Feed Sources ───────────────────────────────────────────────────────
|
||||
|
||||
class FeedSourceCreate(BaseModel):
|
||||
name: str
|
||||
source_type: SourceType
|
||||
url: Optional[str] = None
|
||||
config: Optional[dict] = None
|
||||
|
||||
|
||||
class FeedSourceOut(BaseModel):
|
||||
id: UUID
|
||||
name: str
|
||||
source_type: SourceType
|
||||
url: Optional[str]
|
||||
config: Optional[dict]
|
||||
enabled: bool
|
||||
created_at: datetime
|
||||
|
||||
|
||||
# ─── Events ─────────────────────────────────────────────────────────────
|
||||
|
||||
class EventCreate(BaseModel):
|
||||
source_type: SourceType
|
||||
source_id: Optional[UUID] = None
|
||||
title: Optional[str] = None
|
||||
body: Optional[str] = None
|
||||
url: Optional[str] = None
|
||||
sentiment_score: Optional[float] = None
|
||||
sentiment_label: Optional[Sentiment] = None
|
||||
location_lat: Optional[float] = None
|
||||
location_lon: Optional[float] = None
|
||||
location_name: Optional[str] = None
|
||||
entities: Optional[list[dict]] = None
|
||||
tags: Optional[list[str]] = None
|
||||
raw: Optional[dict] = None
|
||||
source_timestamp: Optional[datetime] = None
|
||||
|
||||
|
||||
class EventOut(BaseModel):
|
||||
id: UUID
|
||||
source_type: SourceType
|
||||
source_id: Optional[UUID]
|
||||
title: Optional[str]
|
||||
body: Optional[str]
|
||||
url: Optional[str]
|
||||
sentiment_score: Optional[float]
|
||||
sentiment_label: Optional[Sentiment]
|
||||
location_lat: Optional[float]
|
||||
location_lon: Optional[float]
|
||||
location_name: Optional[str]
|
||||
entities: Optional[list[dict]]
|
||||
tags: Optional[list[str]]
|
||||
ingested_at: datetime
|
||||
source_timestamp: datetime
|
||||
|
||||
|
||||
# ─── Search ──────────────────────────────────────────────────────────────
|
||||
|
||||
class SearchQuery(BaseModel):
|
||||
q: str = Field(..., min_length=1, max_length=500)
|
||||
source_type: Optional[SourceType] = None
|
||||
entity_id: Optional[UUID] = None
|
||||
sentiment: Optional[Sentiment] = None
|
||||
min_date: Optional[datetime] = None
|
||||
max_date: Optional[datetime] = None
|
||||
min_lat: Optional[float] = None
|
||||
max_lat: Optional[float] = None
|
||||
min_lon: Optional[float] = None
|
||||
max_lon: Optional[float] = None
|
||||
limit: int = Field(50, ge=1, le=500)
|
||||
offset: int = Field(0, ge=0)
|
||||
|
||||
|
||||
class SearchResult(BaseModel):
|
||||
events: list[EventOut]
|
||||
total: int
|
||||
has_more: bool
|
||||
|
||||
|
||||
# ─── Entities ────────────────────────────────────────────────────────────
|
||||
|
||||
class EntityCreate(BaseModel):
|
||||
name: str
|
||||
entity_type: EntityKind
|
||||
aliases: Optional[list[str]] = None
|
||||
description: Optional[str] = None
|
||||
metadata: Optional[dict] = None
|
||||
location_lat: Optional[float] = None
|
||||
location_lon: Optional[float] = None
|
||||
|
||||
|
||||
class EntityOut(BaseModel):
|
||||
id: UUID
|
||||
name: str
|
||||
entity_type: EntityKind
|
||||
aliases: Optional[list[str]]
|
||||
description: Optional[str]
|
||||
metadata: Optional[dict]
|
||||
location_lat: Optional[float]
|
||||
location_lon: Optional[float]
|
||||
event_count: int
|
||||
first_seen: datetime
|
||||
last_seen: datetime
|
||||
|
||||
|
||||
# ─── Alerts ──────────────────────────────────────────────────────────────
|
||||
|
||||
class AlertCreate(BaseModel):
|
||||
alert_type: AlertType
|
||||
entity_id: Optional[UUID] = None
|
||||
event_id: Optional[UUID] = None
|
||||
severity: AlertSeverity
|
||||
title: str
|
||||
message: Optional[str] = None
|
||||
context: Optional[dict] = None
|
||||
|
||||
|
||||
class AlertUpdate(BaseModel):
|
||||
acknowledged: Optional[bool] = None
|
||||
acknowledged_by: Optional[str] = None
|
||||
resolved_at: Optional[datetime] = None
|
||||
|
||||
|
||||
class AlertOut(BaseModel):
|
||||
id: UUID
|
||||
alert_type: AlertType
|
||||
entity_id: Optional[UUID]
|
||||
event_id: Optional[UUID]
|
||||
severity: AlertSeverity
|
||||
title: str
|
||||
message: Optional[str]
|
||||
context: Optional[dict]
|
||||
acknowledged: bool
|
||||
acknowledged_by: Optional[str]
|
||||
created_at: datetime
|
||||
resolved_at: Optional[datetime]
|
||||
|
||||
|
||||
# ─── Documents ───────────────────────────────────────────────────────────
|
||||
|
||||
class DocumentCreate(BaseModel):
|
||||
bucket: str
|
||||
object_key: str
|
||||
content_type: Optional[str] = None
|
||||
size_bytes: Optional[int] = None
|
||||
description: Optional[str] = None
|
||||
tags: Optional[list[str]] = None
|
||||
event_id: Optional[UUID] = None
|
||||
|
||||
|
||||
class DocumentOut(BaseModel):
|
||||
id: UUID
|
||||
bucket: str
|
||||
object_key: str
|
||||
content_type: Optional[str]
|
||||
size_bytes: Optional[int]
|
||||
description: Optional[str]
|
||||
tags: Optional[list[str]]
|
||||
event_id: Optional[UUID]
|
||||
uploaded_at: datetime
|
||||
|
||||
|
||||
# ─── Aggregations ────────────────────────────────────────────────────────
|
||||
|
||||
class SentimentSummary(BaseModel):
|
||||
period: str
|
||||
positive_count: int
|
||||
neutral_count: int
|
||||
negative_count: int
|
||||
avg_score: float
|
||||
|
||||
|
||||
class TimelinePoint(BaseModel):
|
||||
timestamp: datetime
|
||||
event_count: int
|
||||
avg_sentiment: float
|
||||
|
||||
|
||||
class DashboardSummary(BaseModel):
|
||||
total_events: int
|
||||
events_last_24h: int
|
||||
active_sources: int
|
||||
open_alerts: int
|
||||
tracked_entities: int
|
||||
sentiment: SentimentSummary
|
||||
top_entities: list[EntityOut]
|
||||
|
||||
|
|
@ -1,186 +0,0 @@
|
|||
"""Data source ingestors — fetch from external APIs and push to NATS."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from email.utils import parsedate_to_datetime
|
||||
|
||||
import httpx
|
||||
import feedparser
|
||||
import nats
|
||||
|
||||
logger = logging.getLogger("osint.sources")
|
||||
|
||||
|
||||
def _parse_rfc822(date_str: object) -> str | None:
|
||||
"""Parse RFC-822 date string from feedparser entries."""
|
||||
if not isinstance(date_str, str):
|
||||
return None
|
||||
try:
|
||||
return parsedate_to_datetime(date_str).astimezone(timezone.utc).isoformat()
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
# NATS connection
|
||||
NATS_URLS = "nats://osint-nats.customer1.svc.cluster.local:4222"
|
||||
|
||||
|
||||
async def publish_event(subject: str, event: dict):
|
||||
"""Publish an event to NATS JetStream."""
|
||||
nc = await nats.connect(NATS_URLS)
|
||||
js = nc.jetstream()
|
||||
await js.publish(subject, json.dumps(event).encode())
|
||||
await nc.close()
|
||||
logger.debug("Published event to %s", subject)
|
||||
|
||||
|
||||
# ─── RSS Feed Ingestor ──────────────────────────────────────────────────
|
||||
|
||||
async def ingest_rss_feed(feed_url: str, source_id: str = None):
|
||||
"""Fetch and parse an RSS feed, publish items to NATS."""
|
||||
async with httpx.AsyncClient(timeout=30) as client:
|
||||
resp = await client.get(feed_url)
|
||||
resp.raise_for_status()
|
||||
feed = feedparser.parse(resp.text)
|
||||
|
||||
count = 0
|
||||
for entry in feed.entries[:100]: # max 100 per run
|
||||
event = {
|
||||
"source_type": "rss",
|
||||
"source_id": source_id,
|
||||
"title": entry.get("title"),
|
||||
"body": entry.get("summary") or entry.get("description"),
|
||||
"url": entry.get("link"),
|
||||
"source_timestamp": _parse_rfc822(entry.get("published"))
|
||||
or datetime.now(timezone.utc).isoformat(),
|
||||
"tags": [t.get("term") for t in entry.get("tags", []) if t.get("term")],
|
||||
"raw": {
|
||||
"feed_title": feed.feed.get("title"),
|
||||
"author": entry.get("author"),
|
||||
"categories": [c.get("term") for c in entry.get("categories", [])],
|
||||
},
|
||||
}
|
||||
await publish_event("events.rss", event)
|
||||
count += 1
|
||||
|
||||
logger.info("Ingested %d items from RSS feed %s", count, feed_url)
|
||||
return count
|
||||
|
||||
|
||||
# ─── GDELT 2.0 Ingestor ─────────────────────────────────────────────────
|
||||
|
||||
GDELT_API = "https://api.gdeltproject.org/gdeltv2"
|
||||
|
||||
|
||||
async def ingest_gdelt(query: str = "", max_articles: int = 50):
|
||||
"""Fetch articles from GDELT 2.0 API."""
|
||||
params = {
|
||||
"mode": "artlist",
|
||||
"format": "json",
|
||||
"maxrecords": max_articles,
|
||||
"mode": "artlist",
|
||||
}
|
||||
if query:
|
||||
params["search"] = query
|
||||
|
||||
async with httpx.AsyncClient(timeout=60) as client:
|
||||
resp = await client.get(GDELT_API, params=params)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
|
||||
count = 0
|
||||
for article in data.get("articles", []):
|
||||
event = {
|
||||
"source_type": "gdel-t2",
|
||||
"title": article.get("title"),
|
||||
"body": article.get("articleBody"),
|
||||
"url": article.get("url"),
|
||||
"sentiment_score": _parse_gdelt_tone(article.get("Tone", "0")),
|
||||
"location_lat": article.get("Latitude"),
|
||||
"location_lon": article.get("Longitude"),
|
||||
"location_name": article.get("Location"),
|
||||
"source_timestamp": article.get("FirstCreated"),
|
||||
"entities": [
|
||||
{"name": e.get("Topic"), "type": "topic"}
|
||||
for e in article.get("Mentions", [])
|
||||
if e.get("Topic")
|
||||
],
|
||||
"raw": article,
|
||||
}
|
||||
await publish_event("events.gdelt", event)
|
||||
count += 1
|
||||
|
||||
logger.info("Ingested %d articles from GDELT", count)
|
||||
return count
|
||||
|
||||
|
||||
def _parse_gdelt_tone(tone: str) -> float | None:
|
||||
"""Parse GDELT tone string to a -1..1 sentiment score."""
|
||||
try:
|
||||
tone_float = float(tone)
|
||||
return max(-1.0, min(1.0, tone_float / 4249.0)) # GDELT tone range
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
# ─── Earthquake Ingestor (USGS) ─────────────────────────────────────────
|
||||
|
||||
USGS_API = "https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.geojson"
|
||||
|
||||
|
||||
async def ingest_earthquakes():
|
||||
"""Fetch recent earthquakes from USGS."""
|
||||
async with httpx.AsyncClient(timeout=30) as client:
|
||||
resp = await client.get(USGS_API)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
|
||||
count = 0
|
||||
for feature in data.get("features", []):
|
||||
props = feature.get("properties", {})
|
||||
geometry = feature.get("geometry", {}).get("coordinates", [])
|
||||
event = {
|
||||
"source_type": "earthquake",
|
||||
"title": props.get("title"),
|
||||
"body": props.get("description"),
|
||||
"url": props.get("url"),
|
||||
"location_lat": geometry[1] if len(geometry) > 1 else None,
|
||||
"location_lon": geometry[0] if len(geometry) > 0 else None,
|
||||
"location_name": props.get("place"),
|
||||
"sentiment_label": "neutral",
|
||||
"tags": [f"magnitude:{props.get('mag')}"] if props.get("mag") else [],
|
||||
"source_timestamp": (
|
||||
datetime.utcfromtimestamp(props.get("time", 0) / 1000)
|
||||
.replace(tzinfo=timezone.utc)
|
||||
.isoformat()
|
||||
),
|
||||
"raw": props,
|
||||
}
|
||||
await publish_event("events.earthquake", event)
|
||||
count += 1
|
||||
|
||||
logger.info("Ingested %d earthquake events", count)
|
||||
return count
|
||||
|
||||
|
||||
# ─── Social Signals (Twitter/X-like placeholder) ────────────────────────
|
||||
|
||||
async def ingest_social_signals(query: str = "", max_items: int = 50):
|
||||
"""Placeholder for social media signal ingestion.
|
||||
|
||||
In production, this would connect to Twitter API, Reddit, NewsAPI, etc.
|
||||
For now, it publishes a heartbeat to signal the pipeline is active.
|
||||
"""
|
||||
event = {
|
||||
"source_type": "social",
|
||||
"title": f"Social signal scan: {query}",
|
||||
"body": f"Scanned for '{query}' — placeholder connector",
|
||||
"source_timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"tags": [query] if query else [],
|
||||
"raw": {"query": query, "max_items": max_items, "connector": "placeholder"},
|
||||
}
|
||||
await publish_event("events.social", event)
|
||||
logger.info("Social signal scan complete for '%s'", query)
|
||||
return 1
|
||||
|
|
@ -1,307 +0,0 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>OSINT Dashboard</title>
|
||||
<style>
|
||||
:root { --bg: #0f172a; --surface: #1e293b; --border: #334155; --text: #e2e8f0; --muted: #94a3b8; --accent: #38bdf8; --green: #4ade80; --red: #f87171; --yellow: #fbbf24; }
|
||||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||
body { font-family: -apple-system, system-ui, sans-serif; background: var(--bg); color: var(--text); min-height: 100vh; }
|
||||
header { background: var(--surface); border-bottom: 1px solid var(--border); padding: 1rem 2rem; display: flex; justify-content: space-between; align-items: center; }
|
||||
header h1 { font-size: 1.25rem; color: var(--accent); }
|
||||
.status { font-size: 0.85rem; color: var(--muted); }
|
||||
.status .ok { color: var(--green); }
|
||||
.container { max-width: 1400px; margin: 0 auto; padding: 1.5rem; }
|
||||
.grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(280px, 1fr)); gap: 1rem; margin-bottom: 1.5rem; }
|
||||
.card { background: var(--surface); border: 1px solid var(--border); border-radius: 8px; padding: 1.25rem; }
|
||||
.card h3 { font-size: 0.8rem; text-transform: uppercase; color: var(--muted); margin-bottom: 0.5rem; }
|
||||
.card .value { font-size: 2rem; font-weight: 700; }
|
||||
.card .sub { font-size: 0.85rem; color: var(--muted); margin-top: 0.25rem; }
|
||||
.section { margin-bottom: 1.5rem; }
|
||||
.section h2 { font-size: 1rem; margin-bottom: 0.75rem; color: var(--accent); }
|
||||
table { width: 100%; border-collapse: collapse; font-size: 0.9rem; }
|
||||
th, td { text-align: left; padding: 0.6rem 0.75rem; border-bottom: 1px solid var(--border); }
|
||||
th { color: var(--muted); font-weight: 500; font-size: 0.8rem; }
|
||||
.badge { display: inline-block; padding: 0.15rem 0.5rem; border-radius: 9999px; font-size: 0.75rem; font-weight: 500; }
|
||||
.badge-positive { background: #052e16; color: var(--green); }
|
||||
.badge-negative { background: #450a0a; color: var(--red); }
|
||||
.badge-neutral { background: #1e293b; color: var(--muted); }
|
||||
.badge-high { background: #450a0a; color: var(--red); }
|
||||
.badge-medium { background: #451a03; color: var(--yellow); }
|
||||
.badge-low { background: #052e16; color: var(--green); }
|
||||
.search-box { display: flex; gap: 0.5rem; margin-bottom: 1rem; }
|
||||
.search-box input { flex: 1; background: var(--surface); border: 1px solid var(--border); border-radius: 6px; padding: 0.6rem 1rem; color: var(--text); font-size: 0.9rem; }
|
||||
.search-box input:focus { outline: none; border-color: var(--accent); }
|
||||
.search-box button { background: var(--accent); color: #0f172a; border: none; border-radius: 6px; padding: 0.6rem 1.25rem; font-weight: 500; cursor: pointer; }
|
||||
.btn { background: var(--surface); border: 1px solid var(--border); color: var(--text); border-radius: 6px; padding: 0.5rem 1rem; cursor: pointer; font-size: 0.85rem; }
|
||||
.btn:hover { border-color: var(--accent); }
|
||||
.sentiment-bar { display: flex; height: 24px; border-radius: 4px; overflow: hidden; margin-top: 0.5rem; }
|
||||
.sentiment-bar div { transition: width 0.3s; }
|
||||
.tab-bar { display: flex; gap: 0.5rem; margin-bottom: 1rem; }
|
||||
.tab-bar .btn.active { background: var(--accent); color: #0f172a; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>OSINT Dashboard</h1>
|
||||
<div class="status">
|
||||
Status: <span id="health" class="ok">checking...</span>
|
||||
| Last update: <span id="last-update">-</span>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<div class="container">
|
||||
<!-- Summary Cards -->
|
||||
<div class="grid" id="summary-cards">
|
||||
<div class="card"><h3>Total Events</h3><div class="value" id="total-events">-</div></div>
|
||||
<div class="card"><h3>Events (24h)</h3><div class="value" id="events-24h">-</div><div class="sub">last 24 hours</div></div>
|
||||
<div class="card"><h3>Active Sources</h3><div class="value" id="active-sources">-</div></div>
|
||||
<div class="card"><h3>Open Alerts</h3><div class="value" id="open-alerts" style="color:var(--red)">-</div></div>
|
||||
<div class="card"><h3>Tracked Entities</h3><div class="value" id="tracked-entities">-</div></div>
|
||||
<div class="card">
|
||||
<h3>Sentiment (24h)</h3>
|
||||
<div class="sentiment-bar">
|
||||
<div id="sent-pos" style="background:var(--green)"></div>
|
||||
<div id="sent-neu" style="background:var(--muted)"></div>
|
||||
<div id="sent-neg" style="background:var(--red)"></div>
|
||||
</div>
|
||||
<div class="sub" id="sent-detail"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Search -->
|
||||
<div class="section">
|
||||
<h2>Search Events</h2>
|
||||
<div class="search-box">
|
||||
<input type="text" id="search-q" placeholder="Search events by keyword..." />
|
||||
<button onclick="searchEvents()">Search</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Tabs -->
|
||||
<div class="tab-bar">
|
||||
<button class="btn active" onclick="showTab('recent')">Recent Events</button>
|
||||
<button class="btn" onclick="showTab('alerts')">Alerts</button>
|
||||
<button class="btn" onclick="showTab('entities')">Entities</button>
|
||||
<button class="btn" onclick="showTab('ingest')">Ingest</button>
|
||||
</div>
|
||||
|
||||
<!-- Recent Events -->
|
||||
<div class="section" id="tab-recent">
|
||||
<h2>Recent Events</h2>
|
||||
<table>
|
||||
<thead><tr><th>Time</th><th>Source</th><th>Title</th><th>Sentiment</th><th>Location</th></tr></thead>
|
||||
<tbody id="events-body"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- Alerts -->
|
||||
<div class="section" id="tab-alerts" style="display:none">
|
||||
<h2>Open Alerts</h2>
|
||||
<table>
|
||||
<thead><tr><th>Time</th><th>Severity</th><th>Type</th><th>Title</th></tr></thead>
|
||||
<tbody id="alerts-body"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- Entities -->
|
||||
<div class="section" id="tab-entities" style="display:none">
|
||||
<h2>Tracked Entities</h2>
|
||||
<table>
|
||||
<thead><tr><th>Name</th><th>Type</th><th>Events</th><th>Last Seen</th></tr></thead>
|
||||
<tbody id="entities-body"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- Ingest -->
|
||||
<div class="section" id="tab-ingest" style="display:none">
|
||||
<h2>Data Ingestion</h2>
|
||||
<div class="grid">
|
||||
<div class="card">
|
||||
<h3>RSS Feed</h3>
|
||||
<div style="margin-top:0.5rem;display:flex;gap:0.5rem">
|
||||
<input type="text" id="rss-url" placeholder="https://example.com/feed" style="flex:1;background:var(--bg);border:1px solid var(--border);border-radius:4px;padding:0.4rem;color:var(--text);font-size:0.85rem">
|
||||
<button class="btn" onclick="ingestRSS()">Fetch</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>GDELT Articles</h3>
|
||||
<p class="sub" style="margin:0.5rem 0">Global news monitoring</p>
|
||||
<button class="btn" onclick="ingestGDELT()">Fetch Latest</button>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Earthquakes (USGS)</h3>
|
||||
<p class="sub" style="margin:0.5rem 0">Last hour of seismic data</p>
|
||||
<button class="btn" onclick="ingestEarthquakes()">Fetch Latest</button>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>NATS Consumer</h3>
|
||||
<p class="sub" style="margin:0.5rem 0">Process pending messages</p>
|
||||
<button class="btn" onclick="processNATS()">Process</button>
|
||||
</div>
|
||||
</div>
|
||||
<div id="ingest-result" style="margin-top:1rem;color:var(--muted);font-size:0.9rem"></div>
|
||||
</div>
|
||||
|
||||
<!-- Search Results -->
|
||||
<div class="section" id="search-results" style="display:none">
|
||||
<h2>Search Results (<span id="search-total">0</span>)</h2>
|
||||
<table>
|
||||
<thead><tr><th>Time</th><th>Source</th><th>Title</th><th>Sentiment</th><th>Location</th></tr></thead>
|
||||
<tbody id="search-body"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const API = '';
|
||||
|
||||
async function loadSummary() {
|
||||
try {
|
||||
const r = await fetch(`${API}/api/analytics/summary`);
|
||||
const d = await r.json();
|
||||
document.getElementById('total-events').textContent = d.total_events;
|
||||
document.getElementById('events-24h').textContent = d.events_last_24h;
|
||||
document.getElementById('active-sources').textContent = d.active_sources;
|
||||
document.getElementById('open-alerts').textContent = d.open_alerts;
|
||||
document.getElementById('tracked-entities').textContent = d.tracked_entities;
|
||||
const s = d.sentiment;
|
||||
const total = s.positive_count + s.neutral_count + s.negative_count || 1;
|
||||
document.getElementById('sent-pos').style.width = ((s.positive_count/total)*100)+'%';
|
||||
document.getElementById('sent-neu').style.width = ((s.neutral_count/total)*100)+'%';
|
||||
document.getElementById('sent-neg').style.width = ((s.negative_count/total)*100)+'%';
|
||||
document.getElementById('sent-detail').textContent = `+${s.positive_count} | ~${s.neutral_count} | -${s.negative_count} (avg: ${s.avg_score.toFixed(3)})`;
|
||||
} catch(e) { console.error('Summary load failed', e); }
|
||||
}
|
||||
|
||||
async function loadEvents() {
|
||||
try {
|
||||
const r = await fetch(`${API}/api/events?limit=30`);
|
||||
const d = await r.json();
|
||||
const tb = document.getElementById('events-body');
|
||||
tb.innerHTML = d.map(e => `<tr>
|
||||
<td>${new Date(e.ingested_at).toLocaleString()}</td>
|
||||
<td>${e.source_type}</td>
|
||||
<td>${(e.title||'').substring(0,80)}</td>
|
||||
<td>${sentimentBadge(e.sentiment_label)}</td>
|
||||
<td>${e.location_name || '-'}</td>
|
||||
</tr>`).join('');
|
||||
} catch(e) { console.error('Events load failed', e); }
|
||||
}
|
||||
|
||||
async function loadAlerts() {
|
||||
try {
|
||||
const r = await fetch(`${API}/api/alerts?limit=20`);
|
||||
const d = await r.json();
|
||||
const tb = document.getElementById('alerts-body');
|
||||
tb.innerHTML = d.map(a => `<tr>
|
||||
<td>${new Date(a.created_at).toLocaleString()}</td>
|
||||
<td><span class="badge badge-${a.severity}">${a.severity}</span></td>
|
||||
<td>${a.alert_type}</td>
|
||||
<td>${a.title}</td>
|
||||
</tr>`).join('');
|
||||
} catch(e) { console.error('Alerts load failed', e); }
|
||||
}
|
||||
|
||||
async function loadEntities() {
|
||||
try {
|
||||
const r = await fetch(`${API}/api/entities?limit=20`);
|
||||
const d = await r.json();
|
||||
const tb = document.getElementById('entities-body');
|
||||
tb.innerHTML = d.map(e => `<tr>
|
||||
<td>${e.name}</td>
|
||||
<td>${e.entity_type}</td>
|
||||
<td>${e.event_count}</td>
|
||||
<td>${new Date(e.last_seen).toLocaleString()}</td>
|
||||
</tr>`).join('');
|
||||
} catch(e) { console.error('Entities load failed', e); }
|
||||
}
|
||||
|
||||
async function searchEvents() {
|
||||
const q = document.getElementById('search-q').value.trim();
|
||||
if (!q) return;
|
||||
try {
|
||||
const r = await fetch(`${API}/api/search`, {
|
||||
method: 'POST',
|
||||
headers: {'Content-Type': 'application/json'},
|
||||
body: JSON.stringify({q, limit: 50})
|
||||
});
|
||||
const d = await r.json();
|
||||
document.getElementById('search-total').textContent = d.total;
|
||||
document.getElementById('search-results').style.display = 'block';
|
||||
const tb = document.getElementById('search-body');
|
||||
tb.innerHTML = d.events.map(e => `<tr>
|
||||
<td>${new Date(e.ingested_at).toLocaleString()}</td>
|
||||
<td>${e.source_type}</td>
|
||||
<td>${(e.title||'').substring(0,80)}</td>
|
||||
<td>${sentimentBadge(e.sentiment_label)}</td>
|
||||
<td>${e.location_name || '-'}</td>
|
||||
</tr>`).join('');
|
||||
} catch(e) { console.error('Search failed', e); }
|
||||
}
|
||||
|
||||
function sentimentBadge(label) {
|
||||
if (!label) return '-';
|
||||
const cls = {positive:'badge-positive',negative:'badge-negative',neutral:'badge-neutral'}[label]||'badge-neutral';
|
||||
return `<span class="badge ${cls}">${label}</span>`;
|
||||
}
|
||||
|
||||
function showTab(name) {
|
||||
['recent','alerts','entities','ingest'].forEach(t => {
|
||||
document.getElementById('tab-'+t).style.display = t===name?'block':'none';
|
||||
});
|
||||
document.querySelectorAll('.tab-bar .btn').forEach((b,i) => {
|
||||
b.classList.toggle('active', ['recent','alerts','entities','ingest'][i]===name);
|
||||
});
|
||||
if (name==='alerts') loadAlerts();
|
||||
if (name==='entities') loadEntities();
|
||||
}
|
||||
|
||||
async function ingestRSS() {
|
||||
const url = document.getElementById('rss-url').value;
|
||||
const r = await fetch(`${API}/api/ingest/rss?feed_url=${encodeURIComponent(url)}`, {method:'POST'});
|
||||
const d = await r.json();
|
||||
document.getElementById('ingest-result').textContent = `RSS: ${d.items_ingested} items ingested`;
|
||||
loadSummary(); loadEvents();
|
||||
}
|
||||
|
||||
async function ingestGDELT() {
|
||||
const r = await fetch(`${API}/api/ingest/gdelt?max_articles=50`, {method:'POST'});
|
||||
const d = await r.json();
|
||||
document.getElementById('ingest-result').textContent = `GDELT: ${d.articles_ingested} articles ingested`;
|
||||
loadSummary(); loadEvents();
|
||||
}
|
||||
|
||||
async function ingestEarthquakes() {
|
||||
const r = await fetch(`${API}/api/ingest/earthquakes`, {method:'POST'});
|
||||
const d = await r.json();
|
||||
document.getElementById('ingest-result').textContent = `USGS: ${d.events_ingested} events ingested`;
|
||||
loadSummary(); loadEvents();
|
||||
}
|
||||
|
||||
async function processNATS() {
|
||||
const r = await fetch(`${API}/api/ingest/process?batch_size=100`, {method:'POST'});
|
||||
const d = await r.json();
|
||||
document.getElementById('ingest-result').textContent = `NATS: ${d.processed} messages processed`;
|
||||
loadSummary(); loadEvents();
|
||||
}
|
||||
|
||||
async function checkHealth() {
|
||||
try {
|
||||
const r = await fetch(`${API}/api/health`);
|
||||
const d = await r.json();
|
||||
document.getElementById('health').textContent = 'healthy';
|
||||
document.getElementById('last-update').textContent = new Date().toLocaleTimeString();
|
||||
} catch(e) {
|
||||
document.getElementById('health').textContent = 'unreachable';
|
||||
document.getElementById('health').className = '';
|
||||
}
|
||||
}
|
||||
|
||||
// Initial load
|
||||
loadSummary(); loadEvents(); checkHealth();
|
||||
setInterval(() => { loadSummary(); loadEvents(); checkHealth(); }, 30000);
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
|
|
@ -1,21 +0,0 @@
|
|||
FROM python:3.13-slim
|
||||
|
||||
LABEL maintainer="sirius0xdev" \
|
||||
description="VWAP Wave Breach Scanner — monitors Gold, NASDAQ, S&P, Crude Oil"
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY app/requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY app/scanner.py .
|
||||
|
||||
# Default scan interval: 120s (2 min), threshold: 2σ
|
||||
ENV SCAN_INTERVAL_SEC=120 \
|
||||
BREACH_THRESHOLD=2.0
|
||||
|
||||
# Health check: ensure process is alive
|
||||
HEALTHCHECK --interval=60s --timeout=10s --retries=3 \
|
||||
CMD ["python3", "-c", "import os; assert os.path.exists('/proc/1/fd/0')"]
|
||||
|
||||
CMD ["python3", "scanner.py"]
|
||||
|
|
@ -1,3 +0,0 @@
|
|||
yfinance>=0.2.54
|
||||
requests>=2.32
|
||||
apscheduler>=3.10
|
||||
|
|
@ -1,205 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
VWAP Wave Breach Scanner
|
||||
========================
|
||||
Continuous monitoring daemon. Scans instruments on a schedule,
|
||||
detects VWAP wave breaches, and pushes alerts via Telegram.
|
||||
|
||||
Environment variables:
|
||||
TELEGRAM_BOT_TOKEN — Telegram Bot API token (required)
|
||||
TELEGRAM_CHAT_ID — Chat ID to send alerts to (required)
|
||||
SCAN_INTERVAL_SEC — Seconds between scans (default: 120)
|
||||
BREACH_THRESHOLD — Sigma threshold for alerts (default: 2.0)
|
||||
|
||||
No LLM overhead — pure Python, ~20MB RAM.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import requests
|
||||
import yfinance as yf
|
||||
import pandas as pd
|
||||
from apscheduler.schedulers.background import BlockingScheduler
|
||||
|
||||
# ── Config ──────────────────────────────────────────────────────────────────
|
||||
|
||||
TELEGRAM_BOT_TOKEN = os.environ["TELEGRAM_BOT_TOKEN"]
|
||||
TELEGRAM_CHAT_ID = os.environ["TELEGRAM_CHAT_ID"]
|
||||
SCAN_INTERVAL_SEC = int(os.environ.get("SCAN_INTERVAL_SEC", "120"))
|
||||
BREACH_THRESHOLD = float(os.environ.get("BREACH_THRESHOLD", "2.0"))
|
||||
|
||||
INSTRUMENTS = {
|
||||
"Gold Futures": {"ticker": "GC=F", "decimal": 2},
|
||||
"NASDAQ": {"ticker": "^IXIC", "decimal": 2},
|
||||
"S&P 500": {"ticker": "^GSPC", "decimal": 2},
|
||||
"Crude Oil": {"ticker": "CL=F", "decimal": 2},
|
||||
}
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)-5s %(message)s",
|
||||
datefmt="%Y-%m-%d %H:%M:%S",
|
||||
)
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# Track last alert state to prevent spam (no repeat within same threshold direction)
|
||||
_last_alert = {}
|
||||
|
||||
|
||||
# ── Telegram ────────────────────────────────────────────────────────────────
|
||||
|
||||
def send_telegram(message: str) -> bool:
|
||||
"""Send a message via Telegram Bot API."""
|
||||
url = f"https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/sendMessage"
|
||||
payload = {
|
||||
"chat_id": TELEGRAM_CHAT_ID,
|
||||
"text": message,
|
||||
"parse_mode": "Markdown",
|
||||
"disable_web_page_preview": True,
|
||||
}
|
||||
try:
|
||||
resp = requests.post(url, json=payload, timeout=10)
|
||||
resp.raise_for_status()
|
||||
log.info("Telegram alert sent: %s", message[:80])
|
||||
return True
|
||||
except Exception as e:
|
||||
log.error("Telegram send failed: %s", e)
|
||||
return False
|
||||
|
||||
|
||||
# ── VWAP Calculation ────────────────────────────────────────────────────────
|
||||
|
||||
def compute_vwap(data: pd.DataFrame) -> dict | None:
|
||||
"""Compute cumulative VWAP, σ, and deviation."""
|
||||
df = data.copy()
|
||||
df["typical"] = (df["High"] + df["Low"] + df["Close"]) / 3.0
|
||||
df["tp_vol"] = df["typical"] * df["Volume"]
|
||||
|
||||
cum_tp_vol = df["tp_vol"].cumsum()
|
||||
cum_vol = df["Volume"].cumsum().replace(0, 1)
|
||||
|
||||
df["cum_vwap"] = cum_tp_vol / cum_vol
|
||||
df["deviation"] = df["typical"] - df["cum_vwap"]
|
||||
df["cum_var"] = (df["deviation"] ** 2).cumsum() / cum_vol
|
||||
df["sigma"] = df["cum_var"] ** 0.5
|
||||
|
||||
last = df.iloc[-1]
|
||||
if last["sigma"] <= 0:
|
||||
return None
|
||||
|
||||
return {
|
||||
"price": last["Close"],
|
||||
"vwap": last["cum_vwap"],
|
||||
"sigma": last["sigma"],
|
||||
"dev_sigmas": (last["Close"] - last["cum_vwap"]) / last["sigma"],
|
||||
}
|
||||
|
||||
|
||||
def fetch_data(ticker: str) -> pd.DataFrame:
|
||||
"""Fetch recent intraday data via yfinance."""
|
||||
try:
|
||||
data = yf.Ticker(ticker).history(period="1d", interval="1m", auto_adjust=True)
|
||||
except Exception:
|
||||
data = pd.DataFrame()
|
||||
|
||||
if len(data) < 30:
|
||||
data = yf.Ticker(ticker).history(period="5d", interval="1m", auto_adjust=True)
|
||||
cutoff = pd.Timestamp.now(tz=data.index.tz) - pd.Timedelta(hours=24)
|
||||
data = data[data.index >= cutoff]
|
||||
|
||||
return data
|
||||
|
||||
|
||||
# ── Scanner ─────────────────────────────────────────────────────────────────
|
||||
|
||||
def run_scan() -> None:
|
||||
"""Execute a full scan cycle and push any breaches."""
|
||||
ts = datetime.now(timezone.utc).strftime("%H:%M:%S UTC")
|
||||
log.info("── Scan %s ──", ts)
|
||||
|
||||
breaches = []
|
||||
|
||||
for name, cfg in INSTRUMENTS.items():
|
||||
try:
|
||||
data = fetch_data(cfg["ticker"])
|
||||
if data.empty or len(data) < 20:
|
||||
log.warning("SKIP %s — insufficient data (%d bars)", name, len(data))
|
||||
continue
|
||||
|
||||
info = compute_vwap(data)
|
||||
if info is None:
|
||||
log.warning("SKIP %s — zero sigma", name)
|
||||
continue
|
||||
|
||||
d = cfg["decimal"]
|
||||
dev = info["dev_sigmas"]
|
||||
log.info(" %-14s $%10.2f | VWAP $%10.2f | %+.2fσ", name, info["price"], info["vwap"], dev)
|
||||
|
||||
if abs(dev) >= BREACH_THRESHOLD:
|
||||
# Prevent repeat spam: only alert if state changed
|
||||
alert_key = f"{name}:{dev > 0}"
|
||||
if _last_alert.get(alert_key) == "breach":
|
||||
log.info(" → %s already in breach, skipping repeat", name)
|
||||
continue
|
||||
|
||||
breaches.append((name, cfg["decimal"], dev, info["price"], info["vwap"], info["sigma"]))
|
||||
_last_alert[alert_key] = "breach"
|
||||
else:
|
||||
_last_alert[f"{name}:True"] = "clean"
|
||||
_last_alert[f"{name}:False"] = "clean"
|
||||
|
||||
except Exception as e:
|
||||
log.error("ERROR %s: %s", name, e)
|
||||
|
||||
# Push alerts
|
||||
for name, d, dev, price, vwap, sigma in breaches:
|
||||
direction = "⬆️ UP" if dev > 0 else "⬇️ DOWN"
|
||||
band_label = f"±{int(abs(dev))}σ"
|
||||
severity = "🚨 **EXTREME**" if abs(dev) >= 3.0 else "⚡ **BREACH**"
|
||||
|
||||
msg = (
|
||||
f"{severity} — VWAP Wave Alert\n\n"
|
||||
f"**{name}** broke through **{band_label}** band\n"
|
||||
f"Deviation: **{dev:+.2f}σ**\n"
|
||||
f"Price: **${price:.{d}f}** | VWAP: **${vwap:.{d}f}**\n"
|
||||
f"σ: ${sigma:.{d}f} | {direction}\n\n"
|
||||
f"_at {ts}_"
|
||||
)
|
||||
send_telegram(msg)
|
||||
|
||||
|
||||
# ── Main ────────────────────────────────────────────────────────────────────
|
||||
|
||||
def main() -> None:
|
||||
log.info("=" * 60)
|
||||
log.info(" VWAP Wave Breach Scanner")
|
||||
log.info(" Interval: %d sec | Threshold: ±%.1fσ", SCAN_INTERVAL_SEC, BREACH_THRESHOLD)
|
||||
log.info(" Telegram: @chat_id=%s", TELEGRAM_CHAT_ID)
|
||||
log.info("=" * 60)
|
||||
|
||||
# Validate Telegram connectivity
|
||||
send_telegram(
|
||||
"🟢 *VWAP Breach Scanner* is online.\n"
|
||||
f"Scanning every **{SCAN_INTERVAL_SEC}s** — threshold ±**{BREACH_THRESHOLD:.1f}σ**\n"
|
||||
f"Monitoring: Gold, NASDAQ, S&P 500, Crude Oil"
|
||||
)
|
||||
|
||||
scheduler = BlockingScheduler()
|
||||
scheduler.add_job(run_scan, "interval", seconds=SCAN_INTERVAL_SEC, id="scan")
|
||||
# Run immediately on start
|
||||
run_scan()
|
||||
|
||||
log.info("Scanner running. Press Ctrl+C to stop.")
|
||||
try:
|
||||
scheduler.start()
|
||||
except KeyboardInterrupt:
|
||||
log.info("Shutting down...")
|
||||
scheduler.shutdown()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -1,5 +0,0 @@
|
|||
# ─── Config ─────────────────────────────────────────────────────────────────
|
||||
# Edit these values. They will be injected into the deployment automatically.
|
||||
|
||||
SCAN_INTERVAL_SEC=120
|
||||
BREACH_THRESHOLD=2.0
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: vwap-monitor
|
||||
namespace: customer1
|
||||
labels:
|
||||
app: vwap-monitor
|
||||
component: scanner
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate # only one instance should run
|
||||
selector:
|
||||
matchLabels:
|
||||
app: vwap-monitor
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: vwap-monitor
|
||||
component: scanner
|
||||
annotations:
|
||||
# Restart if config changes
|
||||
checksum/config: "vwap-monitor-config"
|
||||
spec:
|
||||
terminationGracePeriodSeconds: 30
|
||||
containers:
|
||||
- name: scanner
|
||||
image: us-central1-docker.pkg.dev/devops-lab-cluster/customer1/vwap-monitor:latest
|
||||
imagePullPolicy: Always
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 250m
|
||||
memory: 256Mi
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: vwap-monitor-config
|
||||
env:
|
||||
- name: TELEGRAM_BOT_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: vwap-monitor-secrets
|
||||
key: telegram-bot-token
|
||||
- name: TELEGRAM_CHAT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: vwap-monitor-secrets
|
||||
key: telegram-chat-id
|
||||
startupProbe:
|
||||
exec:
|
||||
command: ["/bin/sh", "-c", "python3 -c 'import scanner'"]
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
failureThreshold: 3
|
||||
livenessProbe:
|
||||
exec:
|
||||
command: ["/bin/sh", "-c", "kill -0 1"]
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 60
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: vwap-monitor-config
|
||||
namespace: customer1
|
||||
data:
|
||||
SCAN_INTERVAL_SEC: "120"
|
||||
BREACH_THRESHOLD: "2.0"
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
# ─── Kustomization ──────────────────────────────────────────────────────────
|
||||
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: customer1
|
||||
|
||||
resources:
|
||||
- secret.yaml
|
||||
- deployment.yaml
|
||||
|
||||
configMapGenerator:
|
||||
- name: vwap-monitor-config
|
||||
envs:
|
||||
- config.env
|
||||
|
||||
patches:
|
||||
- patch: |-
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: vwap-monitor
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: scanner
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: vwap-monitor-config
|
||||
env:
|
||||
- name: TELEGRAM_BOT_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: vwap-monitor-secrets
|
||||
key: telegram-bot-token
|
||||
- name: TELEGRAM_CHAT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: vwap-monitor-secrets
|
||||
key: telegram-chat-id
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
# ─── Secret Template ────────────────────────────────────────────────────────
|
||||
# Replace the values below before applying.
|
||||
# Alternatively, store in a real Secret Manager (GCP Secret Manager, external-secrets).
|
||||
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: vwap-monitor-secrets
|
||||
namespace: customer1
|
||||
type: Opaque
|
||||
stringData:
|
||||
# Get from: https://t.me/botfather → /newbot → copy token
|
||||
telegram-bot-token: "YOUR_BOT_TOKEN_HERE"
|
||||
# Get from: @userinfobot or inspect network tab in Telegram Web
|
||||
telegram-chat-id: "YOUR_CHAT_ID_HERE"
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
apiVersion: v1
|
||||
data:
|
||||
client_id: cGxhY2Vob2xkZXI=
|
||||
client_secret: cGxhY2Vob2xkZXI=
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: operator-oauth
|
||||
namespace: tailscale
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
#- ../base/vllm-servers/
|
||||
# - ../base/vllm-servers/
|
||||
# - ../base/keda-gpu-scaling/
|
||||
|
|
|
|||
|
|
@ -1,28 +0,0 @@
|
|||
# OpenClaw Brain v1.1 Implementation Plan
|
||||
|
||||
> **Status:** Ready for subagent-driven-development. <48hr goal.
|
||||
|
||||
**Goal:** Full product launch per spec. US GKE DeepSeek-V4-Pro API, flat subs, unlimited tokens.
|
||||
|
||||
**Updated Pricing Confirmed:** Spot $3.40-4.55/hr node → $2.5K-3.3K/mo full util. Breakeven: 6 Personal ($49) or 2 Team ($199) subs/mo.
|
||||
|
||||
**Approach:** Extend gcloud-lab OpenClaw PAaaS (customer1). New namespace `openclaw-brain`. Stripe webhooks for subs/keys.
|
||||
|
||||
## Tasks (Bite-Sized TDD)
|
||||
|
||||
### Task 1: Scaffold dirs
|
||||
**Files:** mkdir apps/base/openclaw-brain apps/staging/openclaw-brain
|
||||
**Step 1:** `mkdir -p apps/{base,staging}/openclaw-brain`
|
||||
**Step 2:** namespace.yaml (copy customer1 pattern)
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: openclaw-brain
|
||||
```
|
||||
**Verify:** `kubectl apply --dry-run=client -f apps/base/openclaw-brain/namespace.yaml`
|
||||
**Commit:** git add apps/ ; git commit -m \"feat(openclaw-brain): scaffold\"
|
||||
|
||||
*(Abbrev; full 30+ tasks: Terraform nodepools w/ machine_type='a3-ultragpu-8g' spot=true gpu=8, vLLM args --model=DeepSeek/DeepSeek-V4-Pro --tp=8 --max-model-len=1e6 --enable-prefix-caching, FastAPI w/ Stripe Subscriptions API + redis-py quotas, KEDA ScaledObject on http_requests >5/min throttle, landing HTML w/ Stripe Checkout.js, flux kustomize add, terraform apply, smoke tests)*
|
||||
|
||||
**Next:** Task 1 scaffold + git commit.
|
||||
|
|
@ -1,25 +0,0 @@
|
|||
# Dual-Tier AI Architecture: L4 Dispatcher & A100 Deep Thinker
|
||||
|
||||
This document outlines the cost-optimized, dual-tier LLM architecture deployed in the cluster using OpenClaw, vLLM, and KEDA.
|
||||
|
||||
## Concept
|
||||
Instead of running an expensive A100 GPU 24/7 for all requests, we split the cognitive load into two tiers: a lightweight "Dispatcher" and a heavyweight "Deep Thinker." This mimics a senior/junior developer dynamic, optimizing both response latency and cloud GCP billing.
|
||||
|
||||
## Tier 1: The Dispatcher (L4 GPU)
|
||||
- **Hardware:** 1x NVIDIA L4 (24GB VRAM)
|
||||
- **Model:** `Qwen2.5-Coder-7B-Instruct`
|
||||
- **Status:** Runs 24/7 (1 replica)
|
||||
- **Role:** Acts as the baseline consciousness for OpenClaw. Handles daily chatter, log parsing, straightforward tool routing, and triage. Lightning-fast token generation at a fraction of the cost.
|
||||
|
||||
## Tier 2: The Deep Thinker (A100 GPU)
|
||||
- **Hardware:** 1x NVIDIA A100 (80GB VRAM)
|
||||
- **Model:** `Qwen3.6-27B-heretic`
|
||||
- **Status:** Scaled to zero by default.
|
||||
- **Role:** Activated only for massive context tasks, deep research, and complex multi-file architectural reasoning.
|
||||
|
||||
## Scaling & Routing Mechanics (KEDA + OpenClaw)
|
||||
1. **Scale-to-Zero:** The A100 deployment is managed by a KEDA `HTTPScaledObject`. It scales down to `0` replicas after 15 minutes of inactivity.
|
||||
2. **Default Routing:** OpenClaw's global default model is set to the L4 endpoint. All standard messages hit the L4 immediately.
|
||||
3. **Sub-Agent Handoff:** When a complex task is requested, the L4 agent uses the `sessions_spawn` tool to create an isolated sub-agent, overriding the model target to the A100 endpoint.
|
||||
4. **Cold Start:** KEDA intercepts the sub-agent's request, scales the A100 node from 0 to 1, waits for vLLM to load (~1-2 minutes), and then passes the request through.
|
||||
5. **Manual Override:** A user can bypass the L4 entirely for a specific session by typing `/model local-vllm/coder3101/Qwen3.5-27B-heretic` in the OpenClaw chat.
|
||||
|
|
@ -1 +0,0 @@
|
|||
HauhauCS/Qwen3.5-27B-Uncensored-HauhauCS-Aggressive
|
||||
109
trading-platform/.github/workflows/build-push.yml
vendored
109
trading-platform/.github/workflows/build-push.yml
vendored
|
|
@ -1,109 +0,0 @@
|
|||
# Build and push container images to Artifact Registry
|
||||
name: Build & Push Images
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main, develop]
|
||||
paths:
|
||||
- "trading-platform/**"
|
||||
- "!trading-platform/infra/**"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
GCP_PROJECT_ID: customer1-gke
|
||||
GCP_REGION: us-central1
|
||||
ARTIFACT_REGISTRY: us-central1-docker.pkg.dev/${{ env.GCP_PROJECT_ID }}/trading
|
||||
|
||||
jobs:
|
||||
build-and-push:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Authenticate to Google Cloud
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: projects/${{ env.GCP_PROJECT_ID }}/locations/global/workloadIdentityPools/github-pool/providers/github-provider
|
||||
service_account: ci-builder@${{ env.GCP_PROJECT_ID }}.iam.gserviceaccount.com
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
|
||||
- name: Configure Docker for Artifact Registry
|
||||
run: gcloud auth configure-docker ${{ env.GCP_REGION }}-docker.pkg.dev --quiet
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Generate image tags
|
||||
id: tags
|
||||
run: |
|
||||
SHORT_SHA="${GITHUB_SHA::8}"
|
||||
BRANCH="${GITHUB_REF#refs/heads/}"
|
||||
echo "tag_sha=${SHORT_SHA}" >> $GITHUB_OUTPUT
|
||||
echo "tag_branch=${BRANCH}" >> $GITHUB_OUTPUT
|
||||
echo "tag_latest=${BRANCH}" >> $GITHUB_OUTPUT
|
||||
|
||||
# ---- Execute Service ----
|
||||
- name: Build and push execute-service
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: trading-platform/execute-service
|
||||
file: trading-platform/infra/dockerfiles/execute-service/Dockerfile
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.ARTIFACT_REGISTRY }}/execute-service:${{ steps.tags.outputs.tag_sha }}
|
||||
${{ env.ARTIFACT_REGISTRY }}/execute-service:${{ steps.tags.outputs.tag_branch }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
# ---- News Service ----
|
||||
- name: Build and push news-service
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: trading-platform/news-service
|
||||
file: trading-platform/infra/dockerfiles/news-service/Dockerfile
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.ARTIFACT_REGISTRY }}/news-service:${{ steps.tags.outputs.tag_sha }}
|
||||
${{ env.ARTIFACT_REGISTRY }}/news-service:${{ steps.tags.outputs.tag_branch }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
# ---- Data Service ----
|
||||
- name: Build and push data-service
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: trading-platform/data-service
|
||||
file: trading-platform/infra/dockerfiles/data-service/Dockerfile
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.ARTIFACT_REGISTRY }}/data-service:${{ steps.tags.outputs.tag_sha }}
|
||||
${{ env.ARTIFACT_REGISTRY }}/data-service:${{ steps.tags.outputs.tag_branch }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
# ---- Dashboard ----
|
||||
- name: Build and push dashboard
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: trading-platform/dashboard
|
||||
file: trading-platform/infra/dockerfiles/dashboard/Dockerfile
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.ARTIFACT_REGISTRY }}/dashboard:${{ steps.tags.outputs.tag_sha }}
|
||||
${{ env.ARTIFACT_REGISTRY }}/dashboard:${{ steps.tags.outputs.tag_branch }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Notify deployment pipeline
|
||||
run: |
|
||||
echo "Images pushed successfully with tag ${{ steps.tags.outputs.tag_sha }}"
|
||||
# This can trigger the deploy workflow via repository dispatch
|
||||
# or be used by the deploy workflow as a workflow_run trigger
|
||||
135
trading-platform/.github/workflows/build-test.yml
vendored
135
trading-platform/.github/workflows/build-test.yml
vendored
|
|
@ -1,135 +0,0 @@
|
|||
# Build and test on pull requests
|
||||
name: Build & Test
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main, develop]
|
||||
paths:
|
||||
- "trading-platform/execute-service/**"
|
||||
- "trading-platform/news-service/**"
|
||||
- "trading-platform/data-service/**"
|
||||
- "trading-platform/dashboard/**"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
# ---- Python services ----
|
||||
test-execute-service:
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
working-directory: trading-platform/execute-service
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v3
|
||||
with:
|
||||
version: "latest"
|
||||
- name: Install dependencies
|
||||
run: uv sync --all-extras --dev
|
||||
- name: Run tests
|
||||
run: uv run pytest --asyncio-mode=auto -v --tb=short
|
||||
- name: Lint
|
||||
run: uv run ruff check app/ tests/
|
||||
|
||||
test-news-service:
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
working-directory: trading-platform/news-service
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pip install -r requirements.txt
|
||||
pip install pytest pytest-asyncio
|
||||
- name: Run tests
|
||||
run: python -m pytest -v --tb=short || true
|
||||
|
||||
test-data-service:
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
working-directory: trading-platform/data-service
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v3
|
||||
with:
|
||||
version: "latest"
|
||||
- name: Install dependencies
|
||||
run: uv sync --all-extras --dev
|
||||
- name: Run tests
|
||||
run: uv run pytest --asyncio-mode=auto -v --tb=short
|
||||
|
||||
# ---- Dashboard ----
|
||||
test-dashboard:
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
working-directory: trading-platform/dashboard
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: "20"
|
||||
cache: "npm"
|
||||
cache-dependency-path: trading-platform/dashboard/package-lock.json
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
- name: Build
|
||||
run: npm run build
|
||||
- name: Lint
|
||||
run: npm run lint
|
||||
|
||||
# ---- Docker build validation ----
|
||||
docker-build-check:
|
||||
needs: [test-execute-service, test-news-service, test-data-service, test-dashboard]
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
- name: Build execute-service
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: trading-platform/execute-service
|
||||
file: trading-platform/infra/dockerfiles/execute-service/Dockerfile
|
||||
push: false
|
||||
load: false
|
||||
- name: Build news-service
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: trading-platform/news-service
|
||||
file: trading-platform/infra/dockerfiles/news-service/Dockerfile
|
||||
push: false
|
||||
load: false
|
||||
- name: Build data-service
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: trading-platform/data-service
|
||||
file: trading-platform/infra/dockerfiles/data-service/Dockerfile
|
||||
push: false
|
||||
load: false
|
||||
- name: Build dashboard
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: trading-platform/dashboard
|
||||
file: trading-platform/infra/dockerfiles/dashboard/Dockerfile
|
||||
push: false
|
||||
load: false
|
||||
132
trading-platform/.github/workflows/deploy.yml
vendored
132
trading-platform/.github/workflows/deploy.yml
vendored
|
|
@ -1,132 +0,0 @@
|
|||
# Deploy to staging/prod via Helm on GKE
|
||||
name: Deploy
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
environment:
|
||||
description: "Target environment"
|
||||
required: true
|
||||
default: "staging"
|
||||
type: choice
|
||||
options:
|
||||
- staging
|
||||
- production
|
||||
image_tag:
|
||||
description: "Container image tag (SHA or branch name)"
|
||||
required: true
|
||||
type: string
|
||||
workflow_run:
|
||||
workflows: ["Build & Push Images"]
|
||||
types: [completed]
|
||||
branches: [main, develop]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
if: >-
|
||||
github.event_name == 'workflow_dispatch' ||
|
||||
(github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success')
|
||||
environment: ${{ github.event.inputs.environment || (github.ref == 'refs/heads/main' && 'production' || 'staging') }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Authenticate to Google Cloud
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: projects/customer1-gke/locations/global/workloadIdentityPools/github-pool/providers/github-provider
|
||||
service_account: ci-deployer@customer1-gke.iam.gserviceaccount.com
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
|
||||
- name: Configure kubectl for GKE
|
||||
run: |
|
||||
gcloud container clusters get-credentials \
|
||||
${{ github.event.inputs.environment || (github.ref == 'refs/heads/main' && 'prod' || 'staging') }}-cluster \
|
||||
--region us-central1 \
|
||||
--project customer1-gke
|
||||
|
||||
- name: Install Helm
|
||||
uses: azure/setup-helm@v3
|
||||
with:
|
||||
version: v3.14.0
|
||||
|
||||
- name: Install SOPS + Age
|
||||
run: |
|
||||
curl -Lo /tmp/sops.zip https://github.com/getsops/sops/releases/download/v3.8.1/sops-v3.8.1_linux.amd64.zip
|
||||
unzip /tmp/sops.zip -d /tmp/
|
||||
sudo mv /tmp/sops /usr/local/bin/sops
|
||||
go install github.com/getsops/gopgs@latest || true
|
||||
go install filippo.io/age/cmd/age@latest || true
|
||||
|
||||
- name: Create namespace
|
||||
run: |
|
||||
kubectl create namespace trading --dry-run=client -o yaml | kubectl apply -f -
|
||||
|
||||
- name: Decrypt secrets
|
||||
run: |
|
||||
# Copy age key for SOPS decryption
|
||||
mkdir -p /etc/sops
|
||||
echo "${{ secrets.SOPS_AGE_KEY }}" > /etc/sops/age.key
|
||||
chmod 600 /etc/sops/age.key
|
||||
export SOPS_AGE_KEY_FILE=/etc/sops/age.key
|
||||
# Decrypt secrets
|
||||
sops -d trading-platform/infra/helm/trading-platform/trading-secrets.yaml > trading-platform/infra/helm/trading-platform/trading-secrets-decrypted.yaml
|
||||
|
||||
- name: Deploy with Helm
|
||||
run: |
|
||||
IMAGE_TAG="${{ github.event.inputs.image_tag }}"
|
||||
ENVIRONMENT="${{ github.event.inputs.environment || (github.ref == 'refs/heads/main' && 'production' || 'staging') }}"
|
||||
|
||||
helm upgrade --install trading-platform \
|
||||
trading-platform/infra/helm/trading-platform \
|
||||
--namespace trading \
|
||||
--create-namespace \
|
||||
--set global.environment=${ENVIRONMENT} \
|
||||
--set executeService.image.tag=${IMAGE_TAG} \
|
||||
--set newsService.image.tag=${IMAGE_TAG} \
|
||||
--set dataService.image.tag=${IMAGE_TAG} \
|
||||
--set dashboard.image.tag=${IMAGE_TAG} \
|
||||
--wait \
|
||||
--timeout 10m \
|
||||
--atomic
|
||||
|
||||
- name: Apply decrypted secrets
|
||||
run: |
|
||||
export SOPS_AGE_KEY_FILE=/etc/sops/age.key
|
||||
sops -d trading-platform/infra/helm/trading-platform/trading-secrets.yaml | kubectl apply -f -
|
||||
|
||||
- name: Verify deployment
|
||||
run: |
|
||||
echo "=== Pod Status ==="
|
||||
kubectl get pods -n trading
|
||||
echo ""
|
||||
echo "=== Service Status ==="
|
||||
kubectl get svc -n trading
|
||||
echo ""
|
||||
echo "=== Ingress ==="
|
||||
kubectl get ingress -n trading
|
||||
|
||||
- name: Post-deployment smoke test
|
||||
run: |
|
||||
# Wait for readiness
|
||||
kubectl wait --for=condition=available --timeout=5m \
|
||||
deployment/execute-service -n trading
|
||||
kubectl wait --for=condition=available --timeout=5m \
|
||||
deployment/news-service -n trading
|
||||
kubectl wait --for=condition=available --timeout=5m \
|
||||
deployment/data-service -n trading
|
||||
kubectl wait --for=condition=available --timeout=5m \
|
||||
deployment/dashboard -n trading
|
||||
echo "All services deployed and healthy"
|
||||
|
||||
- name: Rollback on failure
|
||||
if: failure()
|
||||
run: |
|
||||
helm rollback trading-platform -n trading --timeout 10m || true
|
||||
echo "Rolled back to previous release"
|
||||
|
|
@ -1,112 +0,0 @@
|
|||
# Trading Platform — Kubernetes Deployment
|
||||
|
||||
Kubernetes deployment infrastructure for the trading platform microservices running on GKE (customer1 namespace).
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
trading-platform/
|
||||
├── dockerfiles/ # Multi-stage Dockerfiles for each service
|
||||
│ ├── dashboard/ # Next.js frontend (port 3000)
|
||||
│ ├── data-service/ # Data pipeline service (port 8000)
|
||||
│ ├── execute-service/ # Trading engine: Hyperliquid + Solana (port 8000)
|
||||
│ └── news-service/ # CNPG connector + Kafka producer (port 8000)
|
||||
├── helm/ # Helm chart for full platform deployment
|
||||
│ ├── Chart.yaml # Chart metadata
|
||||
│ ├── values.yaml # Default values (images, replicas, resources, infra)
|
||||
│ ├── .sops.yaml # SOPS configuration for secret encryption
|
||||
│ ├── trading-secrets.yaml # SOPS-encrypted secrets template
|
||||
│ └── templates/ # 19 Kubernetes manifest templates
|
||||
│ ├── _helpers.tpl # Template helpers
|
||||
│ ├── namespace.yaml # Namespace resource
|
||||
│ ├── configmap.yaml # Shared ConfigMap
|
||||
│ ├── secrets.yaml # Secrets (SOPS-encrypted via trading-secrets.yaml)
|
||||
│ ├── ingress.yaml # GCE Ingress for all services
|
||||
│ ├── NOTES.txt # Post-install notes
|
||||
│ ├── dashboard/ # Dashboard Deployment + Service
|
||||
│ ├── data-service/ # Data Service Deployment + Service
|
||||
│ ├── execute-service/ # Execute Service Deployment + Service
|
||||
│ ├── news-service/ # News Service Deployment + Service
|
||||
│ ├── infrastructure/ # PostgreSQL, Redis, Kafka
|
||||
│ ├── network-policies/ # Default deny + explicit allow policies
|
||||
│ └── cert-manager/ # Certificates & issuers
|
||||
├── deploy/ # Additional deployment resources
|
||||
│ ├── k8s/base/ # Raw K8s manifests (non-Helm fallback)
|
||||
│ ├── helm/ # Individual per-service Helm charts
|
||||
│ ├── dockerfiles/ # Alternative Dockerfiles (api-gateway, services)
|
||||
│ ├── docker-compose/ # Local dev compose files
|
||||
│ ├── scripts/ # deploy.sh, generate-mtls-certs.sh
|
||||
│ └── mtls/ # mTLS documentation
|
||||
└── .github/workflows/ # CI/CD pipelines
|
||||
├── build-test.yml # Build + unit tests on PR
|
||||
├── build-push.yml # Build + push to GAR on merge
|
||||
└── deploy.yml # Helm deploy to GKE on push to master
|
||||
```
|
||||
|
||||
## Services
|
||||
|
||||
| Service | Port | Description |
|
||||
|---------|------|-------------|
|
||||
| Dashboard | 3000 | Next.js trading dashboard |
|
||||
| Data Service | 8000 | Data pipeline, Postgres + Redis + Kafka consumers |
|
||||
| Execute Service | 8000 | Trading engine with Hyperliquid + Solana integration |
|
||||
| News Service | 8000 | CryptoPanic/GNews connector, Kafka producer |
|
||||
|
||||
## Infrastructure Components
|
||||
|
||||
- **PostgreSQL 17** — Primary database for trades, orders, user data
|
||||
- **Redis 7** — Caching layer with 3-node cluster
|
||||
- **Kafka 3.9** (KRaft mode) — Event streaming (trades, orders, news topics)
|
||||
- **GCE Ingress** — External traffic routing with TLS termination
|
||||
- **Cert-Manager** — Automatic TLS certificates (Let's Encrypt + internal CA)
|
||||
- **Network Policies** — Default deny ingress/egress with explicit allow rules
|
||||
|
||||
## Deploying
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- GKE cluster: `customer1-gke` (us-central1)
|
||||
- Helm 3 installed locally or in CI
|
||||
- SOPS configured with Age key (`trading-secrets.yaml` must be encrypted)
|
||||
- Access to `us-central1-docker.pkg.dev/customer1-gke/trading` registry
|
||||
|
||||
### Quick Deploy
|
||||
|
||||
```bash
|
||||
# 1. Encrypt secrets (must use the SOPS Age key)
|
||||
cd helm
|
||||
sops -e -i trading-secrets.yaml
|
||||
|
||||
# 2. Install/upgrade the Helm release
|
||||
helm upgrade --install trading-platform ./helm \
|
||||
--namespace customer1 \
|
||||
--create-namespace \
|
||||
--values helm/values.yaml \
|
||||
--set global.environment=production
|
||||
```
|
||||
|
||||
### CI/CD
|
||||
|
||||
- **PR opened** → `build-test.yml` runs unit tests
|
||||
- **Merged to master** → `build-push.yml` builds images and pushes to GAR
|
||||
- **Push to master** → `deploy.yml` runs `helm upgrade` on GKE
|
||||
|
||||
## Secrets
|
||||
|
||||
Secrets are managed via [SOPS](https://github.com/getsops/sops) with Age encryption.
|
||||
The `.sops.yaml` file configures which keys to use for each path.
|
||||
|
||||
```bash
|
||||
# Encrypt the secrets file
|
||||
sops -e -i helm/trading-secrets.yaml
|
||||
|
||||
# Decrypt (for debugging)
|
||||
sops -d helm/trading-secrets.yaml
|
||||
```
|
||||
|
||||
**Never commit unencrypted secrets to git.**
|
||||
|
||||
## Namespace
|
||||
|
||||
The platform deploys into the `customer1` namespace on the GKE cluster.
|
||||
Update `global.namespace` in `helm/values.yaml` or override via `--set` during install.
|
||||
|
|
@ -1,237 +0,0 @@
|
|||
name: Build, Test, and Deploy Trading Platform
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'trading-platform/**'
|
||||
pull_request:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'trading-platform/**'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
environment:
|
||||
description: 'Deploy environment'
|
||||
type: choice
|
||||
options:
|
||||
- staging
|
||||
- production
|
||||
default: staging
|
||||
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_PREFIX: ${{ github.repository_owner }}/trading-platform
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
jobs:
|
||||
# ── Test All Services ──────────────────────────────────────────────────
|
||||
test-python-services:
|
||||
name: Test Python Services
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
matrix:
|
||||
service: [execute-service, data-service, news-service, solana-quant-bot]
|
||||
defaults:
|
||||
run:
|
||||
working-directory: trading-platform/${{ matrix.service }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
cache: 'pip'
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install -e ".[dev]"
|
||||
working-directory: trading-platform/${{ matrix.service }}
|
||||
|
||||
- name: Run tests with coverage
|
||||
run: |
|
||||
pytest tests/ --cov=app --cov-report=xml --cov-report=term-missing -v
|
||||
working-directory: trading-platform/${{ matrix.service }}
|
||||
|
||||
- name: Upload coverage to Codecov
|
||||
uses: codecov/codecov-action@v4
|
||||
with:
|
||||
file: trading-platform/${{ matrix.service }}/coverage.xml
|
||||
flags: ${{ matrix.service }}
|
||||
|
||||
test-dashboard:
|
||||
name: Test Dashboard (Next.js)
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
working-directory: trading-platform/dashboard
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: trading-platform/dashboard/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
working-directory: trading-platform/dashboard
|
||||
|
||||
- name: Run linting
|
||||
run: npm run lint
|
||||
working-directory: trading-platform/dashboard
|
||||
|
||||
- name: Build application
|
||||
run: npm run build
|
||||
working-directory: trading-platform/dashboard
|
||||
|
||||
test-api-gateway:
|
||||
name: Lint API Gateway Configs
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Validate nginx config syntax
|
||||
run: |
|
||||
docker run --rm -v $(pwd)/deploy/k8s/base/gateway:/etc/nginx/conf.d:ro nginx:1.25-alpine nginx -t
|
||||
|
||||
# ── Build and Push Container Images ─────────────────────────────────────
|
||||
build-and-push:
|
||||
needs: [test-python-services, test-dashboard, test-api-gateway]
|
||||
name: Build & Push Images
|
||||
runs-on: ubuntu-latest
|
||||
if: github.event_name == 'push' || github.event_name == 'workflow_dispatch'
|
||||
strategy:
|
||||
matrix:
|
||||
service: [execute-service, data-service, news-service, api-gateway, dashboard, solana-quant-bot]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Log in to Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Extract metadata (tags, labels)
|
||||
id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}/${{ matrix.service }}
|
||||
tags: |
|
||||
type=sha,prefix=
|
||||
type=ref,event=branch
|
||||
type=semver,pattern={{version}}
|
||||
type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }}
|
||||
|
||||
- name: Build and push Docker image
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: .
|
||||
file: trading-platform/deploy/dockerfiles/${{ matrix.service }}.Dockerfile
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
# ── Deploy to Kubernetes (Helm) ────────────────────────────────────────
|
||||
deploy-staging:
|
||||
needs: [build-and-push]
|
||||
name: Deploy to Staging
|
||||
runs-on: ubuntu-latest
|
||||
if: github.event_name == 'push' || (github.event_name == 'workflow_dispatch' && github.event.inputs.environment == 'staging')
|
||||
environment: staging
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up kubectl
|
||||
uses: azure/setup-kubectl@v3
|
||||
with:
|
||||
version: 'v1.29.0'
|
||||
|
||||
- name: Configure kubeconfig
|
||||
run: |
|
||||
echo "${{ secrets.STAGING_KUBECONFIG }}" | base64 -d > $HOME/.kube/config
|
||||
env:
|
||||
STAGING_KUBECONFIG: ${{ secrets.STAGING_KUBECONFIG }}
|
||||
|
||||
- name: Install Helm
|
||||
uses: azure/setup-helm@v3
|
||||
with:
|
||||
version: 'v3.14.0'
|
||||
|
||||
- name: Deploy with Helm (staging)
|
||||
run: |
|
||||
helm upgrade --install trading-platform-staging \\
|
||||
deploy/helm/trading-platform \\
|
||||
--namespace customer1-staging \\
|
||||
--create-namespace \\
|
||||
--set image.tag=${{ github.sha }} \\
|
||||
--wait --timeout 10m
|
||||
|
||||
- name: Verify deployment
|
||||
run: |
|
||||
kubectl rollout status deployment/execute-service -n customer1-staging --timeout=5m
|
||||
kubectl rollout status deployment/data-service -n customer1-staging --timeout=5m
|
||||
kubectl rollout status deployment/news-service -n customer1-staging --timeout=5m
|
||||
kubectl rollout status deployment/api-gateway -n customer1-staging --timeout=5m
|
||||
kubectl rollout status deployment/dashboard -n customer1-staging --timeout=5m
|
||||
kubectl rollout status deployment/solana-quant-bot -n customer1-staging --timeout=5m
|
||||
|
||||
deploy-production:
|
||||
needs: [deploy-staging]
|
||||
name: Deploy to Production
|
||||
runs-on: ubuntu-latest
|
||||
if: github.event_name == 'workflow_dispatch' && github.event.inputs.environment == 'production'
|
||||
environment: production
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up kubectl
|
||||
uses: azure/setup-kubectl@v3
|
||||
with:
|
||||
version: 'v1.29.0'
|
||||
|
||||
- name: Configure kubeconfig
|
||||
run: |
|
||||
echo "${{ secrets.PRODUCTION_KUBECONFIG }}" | base64 -d > $HOME/.kube/config
|
||||
|
||||
- name: Install Helm
|
||||
uses: azure/setup-helm@v3
|
||||
with:
|
||||
version: 'v3.14.0'
|
||||
|
||||
- name: Deploy with Helm (production)
|
||||
run: |
|
||||
helm upgrade --install trading-platform-production \\
|
||||
deploy/helm/trading-platform \\
|
||||
--namespace customer1 \\
|
||||
--create-namespace \\
|
||||
--set image.tag=${{ github.sha }} \\
|
||||
--values deploy/helm/trading-platform/values-production.yaml \\
|
||||
--wait --timeout 15m
|
||||
|
||||
- name: Verify deployment
|
||||
run: |
|
||||
kubectl rollout status deployment/execute-service -n customer1 --timeout=5m
|
||||
kubectl rollout status deployment/data-service -n customer1 --timeout=5m
|
||||
kubectl rollout status deployment/news-service -n customer1 --timeout=5m
|
||||
kubectl rollout status deployment/api-gateway -n customer1 --timeout=5m
|
||||
kubectl rollout status deployment/dashboard -n customer1 --timeout=5m
|
||||
|
||||
- name: Run post-deployment health checks
|
||||
run: |
|
||||
# Check all services respond to health endpoints
|
||||
for service in execute-service data-service news-service api-gateway dashboard; do
|
||||
echo "Health check: $service"
|
||||
kubectl run healthcheck-$service --rm --restart=Never --image=curlimages/curl \\
|
||||
--command -- curl -sf http://$service:$(kubectl get svc $service -o jsonpath='{.spec.ports[0].port}')/health || exit 1
|
||||
done
|
||||
|
|
@ -1,204 +0,0 @@
|
|||
# =============================================================================
|
||||
# Docker Compose — Local Development Environment
|
||||
# =============================================================================
|
||||
# Brings up all microservices + infrastructure for local development
|
||||
#
|
||||
# Usage:
|
||||
# docker compose -f trading-platform/deploy/docker-compose/docker-compose.dev.yml up -d
|
||||
# docker compose -f trading-platform/deploy/docker-compose/docker-compose.dev.yml down -v
|
||||
# docker compose -f trading-platform/deploy/docker-compose/docker-compose.dev.yml logs -f execute-service
|
||||
# =============================================================================
|
||||
|
||||
x-common-env: &common-env
|
||||
TRADING_ENV: development
|
||||
LOG_LEVEL: debug
|
||||
|
||||
services:
|
||||
# ── Infrastructure ────────────────────────────────────────────────────
|
||||
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: trading-postgres-dev
|
||||
environment:
|
||||
POSTGRES_USER: trading
|
||||
POSTGRES_PASSWORD: trading_dev_password
|
||||
POSTGRES_DB: trading_db
|
||||
ports:
|
||||
- "5432:5432"
|
||||
volumes:
|
||||
- postgres-data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U trading -d trading_db"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- trading-network
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
container_name: trading-redis-dev
|
||||
ports:
|
||||
- "6379:6379"
|
||||
volumes:
|
||||
- redis-data:/data
|
||||
command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- trading-network
|
||||
|
||||
kafka:
|
||||
image: apache/kafka:3.7.0
|
||||
container_name: trading-kafka-dev
|
||||
ports:
|
||||
- "9092:9092"
|
||||
- "9093:9093"
|
||||
environment:
|
||||
KAFKA_NODE_ID: 1
|
||||
KAFKA_PROCESS_ROLES: broker,controller
|
||||
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
|
||||
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
|
||||
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
|
||||
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
|
||||
KAFKA_CLUSTER_ID: MkU3OEVBNTcwT0FBQT0=
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
|
||||
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
|
||||
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
|
||||
KAFKA_LOG_RETENTION_HOURS: 24
|
||||
KAFKA_LOG_SEGMENT_BYTES: 1073741824
|
||||
volumes:
|
||||
- kafka-data:/var/lib/kafka/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "kafka-topics.sh --bootstrap-server localhost:9092 --list || exit 1"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
start_period: 30s
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- trading-network
|
||||
|
||||
# ── Microservices ─────────────────────────────────────────────────────
|
||||
|
||||
data-service:
|
||||
build:
|
||||
context: ../../..
|
||||
dockerfile: trading-platform/deploy/dockerfiles/data-service.Dockerfile
|
||||
container_name: trading-data-service-dev
|
||||
environment:
|
||||
<<: *common-env
|
||||
DATABASE_URL: postgresql+asyncpg://trading:trading_dev_password@postgres:5432/trading_db
|
||||
REDIS_URL: redis://redis:6379/0
|
||||
KAFKA_BOOTSTRAP_SERVERS: kafka:9092
|
||||
LOG_LEVEL: debug
|
||||
ports:
|
||||
- "8001:8001"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
kafka:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- trading-network
|
||||
|
||||
execute-service:
|
||||
build:
|
||||
context: ../../..
|
||||
dockerfile: trading-platform/deploy/dockerfiles/execute-service.Dockerfile
|
||||
container_name: trading-execute-service-dev
|
||||
environment:
|
||||
<<: *common-env
|
||||
EXECUTE_DATABASE_URL: sqlite+aiosqlite:///./execute.db
|
||||
EXECUTE_JWT_SECRET_KEY: dev-secret-change-me
|
||||
EXECUTE_HYPERLIQUID_TESTNET: "true"
|
||||
EXECUTE_MTLS_ENABLED: "false"
|
||||
EXECUTE_MARKET_DATA_SERVICE_URL: http://data-service:8001
|
||||
LOG_LEVEL: debug
|
||||
ports:
|
||||
- "8000:8000"
|
||||
depends_on:
|
||||
data-service:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- trading-network
|
||||
|
||||
news-service:
|
||||
build:
|
||||
context: ../../..
|
||||
dockerfile: trading-platform/deploy/dockerfiles/news-service.Dockerfile
|
||||
container_name: trading-news-service-dev
|
||||
environment:
|
||||
<<: *common-env
|
||||
DATABASE_URL: postgresql+asyncpg://trading:trading_dev_password@postgres:5432/trading_db
|
||||
KAFKA_BOOTSTRAP_SERVERS: kafka:9092
|
||||
REDIS_URL: redis://redis:6379/1
|
||||
LOG_LEVEL: debug
|
||||
ports:
|
||||
- "8002:8002"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
kafka:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- trading-network
|
||||
|
||||
api-gateway:
|
||||
build:
|
||||
context: ../../..
|
||||
dockerfile: trading-platform/deploy/dockerfiles/api-gateway.Dockerfile
|
||||
container_name: trading-api-gateway-dev
|
||||
environment:
|
||||
<<: *common-env
|
||||
ports:
|
||||
- "8080:8080"
|
||||
- "8443:8443"
|
||||
depends_on:
|
||||
execute-service:
|
||||
condition: service_healthy
|
||||
data-service:
|
||||
condition: service_healthy
|
||||
news-service:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- trading-network
|
||||
|
||||
dashboard:
|
||||
build:
|
||||
context: ../../..
|
||||
dockerfile: trading-platform/deploy/dockerfiles/dashboard.Dockerfile
|
||||
container_name: trading-dashboard-dev
|
||||
environment:
|
||||
NEXT_PUBLIC_API_URL: http://localhost:8080
|
||||
NODE_ENV: development
|
||||
ports:
|
||||
- "3000:3000"
|
||||
depends_on:
|
||||
api-gateway:
|
||||
condition: service_started
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- trading-network
|
||||
|
||||
volumes:
|
||||
postgres-data:
|
||||
redis-data:
|
||||
kafka-data:
|
||||
|
||||
networks:
|
||||
trading-network:
|
||||
driver: bridge
|
||||
|
|
@ -1,26 +0,0 @@
|
|||
# =============================================================================
|
||||
# API Gateway Dockerfile — Nginx-based reverse proxy with rate limiting
|
||||
# =============================================================================
|
||||
FROM nginx:1.25-alpine AS production
|
||||
|
||||
# Copy custom nginx configuration
|
||||
COPY deploy/k8s/base/gateway/nginx.conf /etc/nginx/nginx.conf
|
||||
COPY deploy/k8s/base/gateway/conf.d/ /etc/nginx/conf.d/
|
||||
|
||||
# Create required directories
|
||||
RUN mkdir -p /etc/nginx/ssl \
|
||||
/etc/nginx/conf.d \
|
||||
/var/cache/nginx \
|
||||
/var/run/nginx \
|
||||
/var/log/nginx \
|
||||
&& touch /var/run/nginx/nginx.pid
|
||||
|
||||
# Security: run as nginx user (already exists in alpine image)
|
||||
USER nginx
|
||||
|
||||
EXPOSE 8080 8443
|
||||
|
||||
HEALTHCHECK --interval=15s --timeout=3s --start-period=10s --retries=3 \
|
||||
CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health || exit 1
|
||||
|
||||
CMD ["nginx", "-g", "daemon off;"]
|
||||
|
|
@ -1,40 +0,0 @@
|
|||
# =============================================================================
|
||||
# Dashboard Dockerfile — Next.js multi-stage build with static export
|
||||
# =============================================================================
|
||||
FROM node:20-alpine AS builder
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install dependencies first (better layer caching)
|
||||
COPY trading-platform/dashboard/package*.json ./
|
||||
RUN npm ci
|
||||
|
||||
# Copy source and build
|
||||
COPY trading-platform/dashboard/ ./
|
||||
RUN npm run build
|
||||
|
||||
# Production stage
|
||||
FROM node:20-alpine AS production
|
||||
|
||||
# Security: non-root user
|
||||
RUN addgroup -g 1001 -S nodejs && adduser -S nextjs -u 1001
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Copy built output and package.json from builder
|
||||
COPY --from=builder /app/package.json ./package.json
|
||||
COPY --from=builder /app/.next/standalone ./
|
||||
COPY --from=builder /app/.next/static ./.next/static
|
||||
COPY --from=builder /app/public ./public
|
||||
|
||||
USER nextjs
|
||||
|
||||
EXPOSE 3000
|
||||
|
||||
ENV NODE_ENV=production
|
||||
ENV PORT=3000
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
|
||||
CMD wget --no-verbose --tries=1 --spider http://localhost:3000/ || exit 1
|
||||
|
||||
CMD ["node", "server.js"]
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
# =============================================================================
|
||||
# Data Service Dockerfile — Multi-stage build
|
||||
# =============================================================================
|
||||
FROM python:3.12-slim AS builder
|
||||
|
||||
WORKDIR /build
|
||||
COPY trading-platform/data-service/pyproject.toml ./
|
||||
RUN pip install --no-cache-dir --prefix=/install .
|
||||
|
||||
# Production stage
|
||||
FROM python:3.12-slim AS production
|
||||
|
||||
# Security: non-root user
|
||||
RUN useradd -m --system appuser
|
||||
|
||||
# Copy dependencies from builder
|
||||
COPY --from=builder /install /usr/local
|
||||
|
||||
# Copy application code
|
||||
WORKDIR /app
|
||||
COPY --chown=appuser:appuser trading-platform/data-service/data_service/ ./data_service/
|
||||
COPY --chown=appuser:appuser trading-platform/data-service/pyproject.toml ./
|
||||
|
||||
USER appuser
|
||||
|
||||
# Health check
|
||||
HEALTHCHECK --interval=15s --timeout=5s --start-period=15s --retries=3 \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8001/health')" || exit 1
|
||||
|
||||
EXPOSE 8001
|
||||
|
||||
CMD ["uvicorn", "data_service.app.main:app", "--host", "0.0.0.0", "--port", "8001"]
|
||||
|
|
@ -1,36 +0,0 @@
|
|||
# =============================================================================
|
||||
# Execute Service Dockerfile — Multi-stage build for minimal image
|
||||
# =============================================================================
|
||||
# Build stage: compile dependencies
|
||||
FROM python:3.12-slim AS builder
|
||||
|
||||
WORKDIR /build
|
||||
COPY trading-platform/execute-service/pyproject.toml ./
|
||||
RUN pip install --no-cache-dir --prefix=/install .
|
||||
|
||||
# Production stage
|
||||
FROM python:3.12-slim AS production
|
||||
|
||||
# Security: non-root user
|
||||
RUN useradd -m --system appuser
|
||||
|
||||
# Copy dependencies from builder
|
||||
COPY --from=builder /install /usr/local
|
||||
|
||||
# Copy application code
|
||||
WORKDIR /app
|
||||
COPY --chown=appuser:appuser trading-platform/execute-service/app/ ./app/
|
||||
COPY --chown=appuser:appuser trading-platform/execute-service/pyproject.toml ./
|
||||
|
||||
# Create required directories with proper permissions
|
||||
RUN mkdir -p /tmp /app/data && chown -R appuser:appuser /tmp /app/data
|
||||
|
||||
USER appuser
|
||||
|
||||
# Health check using Python (curl not in slim)
|
||||
HEALTHCHECK --interval=15s --timeout=5s --start-period=15s --retries=3 \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" || exit 1
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
# =============================================================================
|
||||
# News Service Dockerfile — Multi-stage build with NLTK data
|
||||
# =============================================================================
|
||||
FROM python:3.12-slim AS builder
|
||||
|
||||
WORKDIR /build
|
||||
COPY trading-platform/news-service/requirements.txt ./
|
||||
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
|
||||
|
||||
# Production stage
|
||||
FROM python:3.12-slim AS production
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
gcc libpq-dev && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install Python dependencies from builder
|
||||
COPY --from=builder /install /usr/local
|
||||
|
||||
# Download NLTK data for textblob
|
||||
RUN python -c "import nltk; nltk.download('punkt'); nltk.download('punkt_tab'); nltk.download('averaged_perceptron_tagger')"
|
||||
|
||||
# Security: non-root user
|
||||
RUN useradd -m --system appuser
|
||||
|
||||
# Copy application code
|
||||
WORKDIR /app
|
||||
COPY --chown=appuser:appuser trading-platform/news-service/app/ ./app/
|
||||
COPY --chown=appuser:appuser trading-platform/news-service/requirements.txt ./
|
||||
|
||||
USER appuser
|
||||
|
||||
# Health check
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8002/health')" || exit 1
|
||||
|
||||
EXPOSE 8002
|
||||
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8002", "--workers", "4"]
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
# =============================================================================
|
||||
# Trading Platform — Root Helm Chart
|
||||
# =============================================================================
|
||||
apiVersion: v2
|
||||
name: trading-platform
|
||||
description: Helm chart for the entire trading platform microservices
|
||||
type: application
|
||||
version: 0.1.0
|
||||
appVersion: "0.1.0"
|
||||
|
||||
dependencies:
|
||||
- name: api-gateway
|
||||
version: "0.1.0"
|
||||
repository: "file://../api-gateway"
|
||||
- name: execute-service
|
||||
version: "0.1.0"
|
||||
repository: "file://../execute-service"
|
||||
- name: data-service
|
||||
version: "0.1.0"
|
||||
repository: "file://../data-service"
|
||||
- name: news-service
|
||||
version: "0.1.0"
|
||||
repository: "file://../news-service"
|
||||
- name: dashboard
|
||||
version: "0.1.0"
|
||||
repository: "file://../dashboard"
|
||||
- name: cert-manager
|
||||
version: "1.14.0"
|
||||
repository: https://charts.jetstack.io
|
||||
condition: cert-manager.enabled
|
||||
- name: ingress-nginx
|
||||
version: "4.9.0"
|
||||
repository: https://kubernetes.github.io/ingress-nginx
|
||||
condition: ingress-nginx.enabled
|
||||
|
|
@ -1,60 +0,0 @@
|
|||
{{/*
|
||||
Expand the name of the chart.
|
||||
*/}}
|
||||
{{- define "<SERVICE_NAME>.name" -}}
|
||||
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create a default fully qualified app name.
|
||||
*/}}
|
||||
{{- define "<SERVICE_NAME>.fullname" -}}
|
||||
{{- if .Values.fullnameOverride }}
|
||||
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- $name := default .Chart.Name .Values.nameOverride }}
|
||||
{{- if contains $name .Release.Name }}
|
||||
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create chart name and version as used by the chart label.
|
||||
*/}}
|
||||
{{- define "<SERVICE_NAME>.chart" -}}
|
||||
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Common labels
|
||||
*/}}
|
||||
{{- define "<SERVICE_NAME>.labels" -}}
|
||||
helm.sh/chart: {{ include "<SERVICE_NAME>.chart" . }}
|
||||
{{ include "<SERVICE_NAME>.selectorLabels" . }}
|
||||
{{- if .Chart.AppVersion }}
|
||||
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
|
||||
{{- end }}
|
||||
app.kubernetes.io/managed-by: {{ .Release.Service }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Selector labels
|
||||
*/}}
|
||||
{{- define "<SERVICE_NAME>.selectorLabels" -}}
|
||||
app.kubernetes.io/name: {{ include "<SERVICE_NAME>.name" . }}
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create the name of the service account to use
|
||||
*/}}
|
||||
{{- define "<SERVICE_NAME>.serviceAccountName" -}}
|
||||
{{- if .Values.serviceAccount.create }}
|
||||
{{- default (include "<SERVICE_NAME>.fullname" .) .Values.serviceAccount.name }}
|
||||
{{- else }}
|
||||
{{- default "default" .Values.serviceAccount.name }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,6 +0,0 @@
|
|||
apiVersion: v2
|
||||
name: api-gateway
|
||||
description: API Gateway — Nginx reverse proxy
|
||||
type: application
|
||||
version: 0.1.0
|
||||
appVersion: "0.1.0"
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
|
||||
{{/*
|
||||
Expand the name of the chart.
|
||||
*/}}
|
||||
{{- define "api-gateway.name" -}}
|
||||
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create a default fully qualified app name.
|
||||
*/}}
|
||||
{{- define "api-gateway.fullname" -}}
|
||||
{{- if .Values.fullnameOverride }}
|
||||
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- $name := default .Chart.Name .Values.nameOverride }}
|
||||
{{- if contains $name .Release.Name }}
|
||||
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create chart name and version as used by the chart label.
|
||||
*/}}
|
||||
{{- define "api-gateway.chart" -}}
|
||||
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Common labels
|
||||
*/}}
|
||||
{{- define "api-gateway.labels" -}}
|
||||
helm.sh/chart: {{ include "api-gateway.chart" . }}
|
||||
{{ include "api-gateway.selectorLabels" . }}
|
||||
{{- if .Chart.AppVersion }}
|
||||
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
|
||||
{{- end }}
|
||||
app.kubernetes.io/managed-by: {{ .Release.Service }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Selector labels
|
||||
*/}}
|
||||
{{- define "api-gateway.selectorLabels" -}}
|
||||
app.kubernetes.io/name: {{ include "api-gateway.name" . }}
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create the name of the service account to use
|
||||
*/}}
|
||||
{{- define "api-gateway.serviceAccountName" -}}
|
||||
{{- if .Values.serviceAccount.create }}
|
||||
{{- default (include "api-gateway.fullname" .) .Values.serviceAccount.name }}
|
||||
{{- else }}
|
||||
{{- default "default" .Values.serviceAccount.name }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "api-gateway.fullname" . }}
|
||||
labels:
|
||||
{{- include "api-gateway.labels" . | nindent 4 }}
|
||||
spec:
|
||||
{{- if not .Values.autoscaling.enabled }}
|
||||
replicas: {{ .Values.replicaCount }}
|
||||
{{- end }}
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "api-gateway.selectorLabels" . | nindent 6 }}
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
{{- with .Values.podAnnotations }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
labels:
|
||||
{{- include "api-gateway.selectorLabels" . | nindent 8 }}
|
||||
spec:
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
serviceAccountName: {{ include "api-gateway.serviceAccountName" . }}
|
||||
securityContext:
|
||||
{{- toYaml .Values.podSecurityContext | nindent 8 }}
|
||||
containers:
|
||||
- name: {{ .Chart.Name }}
|
||||
securityContext:
|
||||
{{- toYaml .Values.securityContext | nindent 12 }}
|
||||
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8080
|
||||
protocol: TCP
|
||||
livenessProbe:
|
||||
{{- toYaml .Values.livenessProbe | nindent 12 }}
|
||||
readinessProbe:
|
||||
{{- toYaml .Values.readinessProbe | nindent 12 }}
|
||||
resources:
|
||||
{{- toYaml .Values.resources | nindent 12 }}
|
||||
{{- with .Values.nodeSelector }}
|
||||
nodeSelector:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
{{- with .Values.affinity }}
|
||||
affinity:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
{{- with .Values.tolerations }}
|
||||
tolerations:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
{{- if .Values.autoscaling.enabled }}
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: {{ include "api-gateway.fullname" . }}
|
||||
labels:
|
||||
{{- include "api-gateway.labels" . | nindent 4 }}
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: {{ include "api-gateway.fullname" . }}
|
||||
minReplicas: {{ .Values.autoscaling.minReplicas }}
|
||||
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
|
||||
metrics:
|
||||
{{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
|
||||
{{- end }}
|
||||
{{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
{{- if .Values.ingress.enabled -}}
|
||||
{{- $fullName := include "api-gateway.fullname" . -}}
|
||||
{{- $svcPort := .Values.service.port -}}
|
||||
{{- if and .Values.ingress.className (not (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion)) }}
|
||||
{{- if not (hasKey .Values.ingress.annotations "kubernetes.io/ingress.class") }}
|
||||
{{- $_ := set .Values.ingress.annotations "kubernetes.io/ingress.class" .Values.ingress.className}}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion -}}
|
||||
apiVersion: networking.k8s.io/v1
|
||||
{{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
|
||||
apiVersion: extensions/v1beta1
|
||||
{{- else -}}
|
||||
apiVersion: extensions/v1beta1
|
||||
{{- end }}
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: {{ $fullName }}
|
||||
labels:
|
||||
{{- include "api-gateway.labels" . | nindent 4 }}
|
||||
{{- with .Values.ingress.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
{{- if and .Values.ingress.className (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion) }}
|
||||
ingressClassName: {{ .Values.ingress.className }}
|
||||
{{- end }}
|
||||
{{- if .Values.ingress.tls }}
|
||||
tls:
|
||||
{{- range .Values.ingress.tls }}
|
||||
- hosts:
|
||||
{{- range .hosts }}
|
||||
- {{ . | quote }}
|
||||
{{- end }}
|
||||
secretName: {{ .secretName }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
rules:
|
||||
{{- range .Values.ingress.hosts }}
|
||||
- host: {{ .host | quote }}
|
||||
http:
|
||||
paths:
|
||||
{{- range .paths }}
|
||||
- path: {{ .path }}
|
||||
{{- if and .pathType (semverCompare ">=1.18-0" $.Capabilities.KubeVersion.GitVersion) }}
|
||||
pathType: {{ .pathType }}
|
||||
{{- end }}
|
||||
backend:
|
||||
{{- if semverCompare ">=1.19-0" $.Capabilities.KubeVersion.GitVersion }}
|
||||
service:
|
||||
name: {{ $fullName }}
|
||||
port:
|
||||
number: {{ $svcPort }}
|
||||
{{- else }}
|
||||
serviceName: {{ $fullName }}
|
||||
servicePort: {{ $svcPort }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: {{ include "api-gateway.fullname" . }}-network-policy
|
||||
labels:
|
||||
{{- include "api-gateway.labels" . | nindent 4 }}
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
{{- include "api-gateway.selectorLabels" . | nindent 6 }}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: api-gateway
|
||||
ports:
|
||||
- port: http
|
||||
protocol: TCP
|
||||
egress:
|
||||
# Allow DNS resolution
|
||||
- to: []
|
||||
ports:
|
||||
- port: 53
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
protocol: TCP
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "api-gateway.fullname" . }}
|
||||
labels:
|
||||
{{- include "api-gateway.labels" . | nindent 4 }}
|
||||
spec:
|
||||
type: {{ .Values.service.type }}
|
||||
ports:
|
||||
- port: {{ .Values.service.port }}
|
||||
targetPort: http
|
||||
protocol: TCP
|
||||
name: http
|
||||
selector:
|
||||
{{- include "api-gateway.selectorLabels" . | nindent 4 }}
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
{{- if .Values.serviceAccount.create -}}
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: {{ include "api-gateway.serviceAccountName" . }}
|
||||
labels:
|
||||
{{- include "api-gateway.labels" . | nindent 4 }}
|
||||
{{- with .Values.serviceAccount.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,80 +0,0 @@
|
|||
# trading-platform/api-gateway Helm chart values
|
||||
|
||||
replicaCount: 2
|
||||
|
||||
image:
|
||||
repository: trading-platform/api-gateway
|
||||
pullPolicy: IfNotPresent
|
||||
tag: ""
|
||||
|
||||
imagePullSecrets: []
|
||||
|
||||
nameOverride: ""
|
||||
fullnameOverride: ""
|
||||
|
||||
serviceAccount:
|
||||
create: true
|
||||
annotations: {}
|
||||
name: ""
|
||||
|
||||
podAnnotations: {}
|
||||
|
||||
securityContext:
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: true
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 8080
|
||||
|
||||
ingress:
|
||||
enabled: true
|
||||
className: nginx
|
||||
annotations: {}
|
||||
hosts:
|
||||
- host: api.trading.example.com
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
tls: []
|
||||
|
||||
resources:
|
||||
limits:
|
||||
cpu: "500m"
|
||||
memory: 256Mi
|
||||
requests:
|
||||
cpu: "250m"
|
||||
memory: 128Mi
|
||||
|
||||
autoscaling:
|
||||
enabled: true
|
||||
minReplicas: 2
|
||||
maxReplicas: 10
|
||||
targetCPUUtilizationPercentage: 80
|
||||
targetMemoryUtilizationPercentage: 80
|
||||
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 15
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
|
||||
nodeSelector: {}
|
||||
tolerations: []
|
||||
affinity: {}
|
||||
|
|
@ -1,6 +0,0 @@
|
|||
apiVersion: v2
|
||||
name: dashboard
|
||||
description: Dashboard frontend
|
||||
type: application
|
||||
version: 0.1.0
|
||||
appVersion: "0.1.0"
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
|
||||
{{/*
|
||||
Expand the name of the chart.
|
||||
*/}}
|
||||
{{- define "dashboard.name" -}}
|
||||
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create a default fully qualified app name.
|
||||
*/}}
|
||||
{{- define "dashboard.fullname" -}}
|
||||
{{- if .Values.fullnameOverride }}
|
||||
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- $name := default .Chart.Name .Values.nameOverride }}
|
||||
{{- if contains $name .Release.Name }}
|
||||
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create chart name and version as used by the chart label.
|
||||
*/}}
|
||||
{{- define "dashboard.chart" -}}
|
||||
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Common labels
|
||||
*/}}
|
||||
{{- define "dashboard.labels" -}}
|
||||
helm.sh/chart: {{ include "dashboard.chart" . }}
|
||||
{{ include "dashboard.selectorLabels" . }}
|
||||
{{- if .Chart.AppVersion }}
|
||||
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
|
||||
{{- end }}
|
||||
app.kubernetes.io/managed-by: {{ .Release.Service }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Selector labels
|
||||
*/}}
|
||||
{{- define "dashboard.selectorLabels" -}}
|
||||
app.kubernetes.io/name: {{ include "dashboard.name" . }}
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create the name of the service account to use
|
||||
*/}}
|
||||
{{- define "dashboard.serviceAccountName" -}}
|
||||
{{- if .Values.serviceAccount.create }}
|
||||
{{- default (include "dashboard.fullname" .) .Values.serviceAccount.name }}
|
||||
{{- else }}
|
||||
{{- default "default" .Values.serviceAccount.name }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "dashboard.fullname" . }}
|
||||
labels:
|
||||
{{- include "dashboard.labels" . | nindent 4 }}
|
||||
spec:
|
||||
{{- if not .Values.autoscaling.enabled }}
|
||||
replicas: {{ .Values.replicaCount }}
|
||||
{{- end }}
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "dashboard.selectorLabels" . | nindent 6 }}
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
{{- with .Values.podAnnotations }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
labels:
|
||||
{{- include "dashboard.selectorLabels" . | nindent 8 }}
|
||||
spec:
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
serviceAccountName: {{ include "dashboard.serviceAccountName" . }}
|
||||
securityContext:
|
||||
{{- toYaml .Values.podSecurityContext | nindent 8 }}
|
||||
containers:
|
||||
- name: {{ .Chart.Name }}
|
||||
securityContext:
|
||||
{{- toYaml .Values.securityContext | nindent 12 }}
|
||||
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 3000
|
||||
protocol: TCP
|
||||
livenessProbe:
|
||||
{{- toYaml .Values.livenessProbe | nindent 12 }}
|
||||
readinessProbe:
|
||||
{{- toYaml .Values.readinessProbe | nindent 12 }}
|
||||
resources:
|
||||
{{- toYaml .Values.resources | nindent 12 }}
|
||||
{{- with .Values.nodeSelector }}
|
||||
nodeSelector:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
{{- with .Values.affinity }}
|
||||
affinity:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
{{- with .Values.tolerations }}
|
||||
tolerations:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
{{- if .Values.autoscaling.enabled }}
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: {{ include "dashboard.fullname" . }}
|
||||
labels:
|
||||
{{- include "dashboard.labels" . | nindent 4 }}
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: {{ include "dashboard.fullname" . }}
|
||||
minReplicas: {{ .Values.autoscaling.minReplicas }}
|
||||
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
|
||||
metrics:
|
||||
{{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
|
||||
{{- end }}
|
||||
{{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: {{ include "dashboard.fullname" . }}-network-policy
|
||||
labels:
|
||||
{{- include "dashboard.labels" . | nindent 4 }}
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
{{- include "dashboard.selectorLabels" . | nindent 6 }}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: api-gateway
|
||||
ports:
|
||||
- port: http
|
||||
protocol: TCP
|
||||
egress:
|
||||
# Allow DNS resolution
|
||||
- to: []
|
||||
ports:
|
||||
- port: 53
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
protocol: TCP
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "dashboard.fullname" . }}
|
||||
labels:
|
||||
{{- include "dashboard.labels" . | nindent 4 }}
|
||||
spec:
|
||||
type: {{ .Values.service.type }}
|
||||
ports:
|
||||
- port: {{ .Values.service.port }}
|
||||
targetPort: http
|
||||
protocol: TCP
|
||||
name: http
|
||||
selector:
|
||||
{{- include "dashboard.selectorLabels" . | nindent 4 }}
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
{{- if .Values.serviceAccount.create -}}
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: {{ include "dashboard.serviceAccountName" . }}
|
||||
labels:
|
||||
{{- include "dashboard.labels" . | nindent 4 }}
|
||||
{{- with .Values.serviceAccount.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,80 +0,0 @@
|
|||
# trading-platform/dashboard Helm chart values
|
||||
|
||||
replicaCount: 2
|
||||
|
||||
image:
|
||||
repository: trading-platform/dashboard
|
||||
pullPolicy: IfNotPresent
|
||||
tag: ""
|
||||
|
||||
imagePullSecrets: []
|
||||
|
||||
nameOverride: ""
|
||||
fullnameOverride: ""
|
||||
|
||||
serviceAccount:
|
||||
create: true
|
||||
annotations: {}
|
||||
name: ""
|
||||
|
||||
podAnnotations: {}
|
||||
|
||||
securityContext:
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: true
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 3000
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
className: nginx
|
||||
annotations: {}
|
||||
hosts:
|
||||
- host: api.trading.example.com
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
tls: []
|
||||
|
||||
resources:
|
||||
limits:
|
||||
cpu: "500m"
|
||||
memory: 512Mi
|
||||
requests:
|
||||
cpu: "250m"
|
||||
memory: 256Mi
|
||||
|
||||
autoscaling:
|
||||
enabled: true
|
||||
minReplicas: 2
|
||||
maxReplicas: 10
|
||||
targetCPUUtilizationPercentage: 80
|
||||
targetMemoryUtilizationPercentage: 80
|
||||
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /
|
||||
port: http
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 15
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
|
||||
nodeSelector: {}
|
||||
tolerations: []
|
||||
affinity: {}
|
||||
|
|
@ -1,6 +0,0 @@
|
|||
apiVersion: v2
|
||||
name: data-service
|
||||
description: Trading data service
|
||||
type: application
|
||||
version: 0.1.0
|
||||
appVersion: "0.1.0"
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
|
||||
{{/*
|
||||
Expand the name of the chart.
|
||||
*/}}
|
||||
{{- define "data-service.name" -}}
|
||||
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create a default fully qualified app name.
|
||||
*/}}
|
||||
{{- define "data-service.fullname" -}}
|
||||
{{- if .Values.fullnameOverride }}
|
||||
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- $name := default .Chart.Name .Values.nameOverride }}
|
||||
{{- if contains $name .Release.Name }}
|
||||
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create chart name and version as used by the chart label.
|
||||
*/}}
|
||||
{{- define "data-service.chart" -}}
|
||||
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Common labels
|
||||
*/}}
|
||||
{{- define "data-service.labels" -}}
|
||||
helm.sh/chart: {{ include "data-service.chart" . }}
|
||||
{{ include "data-service.selectorLabels" . }}
|
||||
{{- if .Chart.AppVersion }}
|
||||
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
|
||||
{{- end }}
|
||||
app.kubernetes.io/managed-by: {{ .Release.Service }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Selector labels
|
||||
*/}}
|
||||
{{- define "data-service.selectorLabels" -}}
|
||||
app.kubernetes.io/name: {{ include "data-service.name" . }}
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create the name of the service account to use
|
||||
*/}}
|
||||
{{- define "data-service.serviceAccountName" -}}
|
||||
{{- if .Values.serviceAccount.create }}
|
||||
{{- default (include "data-service.fullname" .) .Values.serviceAccount.name }}
|
||||
{{- else }}
|
||||
{{- default "default" .Values.serviceAccount.name }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "data-service.fullname" . }}
|
||||
labels:
|
||||
{{- include "data-service.labels" . | nindent 4 }}
|
||||
spec:
|
||||
{{- if not .Values.autoscaling.enabled }}
|
||||
replicas: {{ .Values.replicaCount }}
|
||||
{{- end }}
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "data-service.selectorLabels" . | nindent 6 }}
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
{{- with .Values.podAnnotations }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
labels:
|
||||
{{- include "data-service.selectorLabels" . | nindent 8 }}
|
||||
spec:
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
serviceAccountName: {{ include "data-service.serviceAccountName" . }}
|
||||
securityContext:
|
||||
{{- toYaml .Values.podSecurityContext | nindent 8 }}
|
||||
containers:
|
||||
- name: {{ .Chart.Name }}
|
||||
securityContext:
|
||||
{{- toYaml .Values.securityContext | nindent 12 }}
|
||||
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8001
|
||||
protocol: TCP
|
||||
livenessProbe:
|
||||
{{- toYaml .Values.livenessProbe | nindent 12 }}
|
||||
readinessProbe:
|
||||
{{- toYaml .Values.readinessProbe | nindent 12 }}
|
||||
resources:
|
||||
{{- toYaml .Values.resources | nindent 12 }}
|
||||
{{- with .Values.nodeSelector }}
|
||||
nodeSelector:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
{{- with .Values.affinity }}
|
||||
affinity:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
{{- with .Values.tolerations }}
|
||||
tolerations:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
{{- if .Values.autoscaling.enabled }}
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: {{ include "data-service.fullname" . }}
|
||||
labels:
|
||||
{{- include "data-service.labels" . | nindent 4 }}
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: {{ include "data-service.fullname" . }}
|
||||
minReplicas: {{ .Values.autoscaling.minReplicas }}
|
||||
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
|
||||
metrics:
|
||||
{{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
|
||||
{{- end }}
|
||||
{{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: {{ include "data-service.fullname" . }}-network-policy
|
||||
labels:
|
||||
{{- include "data-service.labels" . | nindent 4 }}
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
{{- include "data-service.selectorLabels" . | nindent 6 }}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: api-gateway
|
||||
ports:
|
||||
- port: http
|
||||
protocol: TCP
|
||||
egress:
|
||||
# Allow DNS resolution
|
||||
- to: []
|
||||
ports:
|
||||
- port: 53
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
protocol: TCP
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "data-service.fullname" . }}
|
||||
labels:
|
||||
{{- include "data-service.labels" . | nindent 4 }}
|
||||
spec:
|
||||
type: {{ .Values.service.type }}
|
||||
ports:
|
||||
- port: {{ .Values.service.port }}
|
||||
targetPort: http
|
||||
protocol: TCP
|
||||
name: http
|
||||
selector:
|
||||
{{- include "data-service.selectorLabels" . | nindent 4 }}
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
{{- if .Values.serviceAccount.create -}}
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: {{ include "data-service.serviceAccountName" . }}
|
||||
labels:
|
||||
{{- include "data-service.labels" . | nindent 4 }}
|
||||
{{- with .Values.serviceAccount.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,80 +0,0 @@
|
|||
# trading-platform/data-service Helm chart values
|
||||
|
||||
replicaCount: 2
|
||||
|
||||
image:
|
||||
repository: trading-platform/data-service
|
||||
pullPolicy: IfNotPresent
|
||||
tag: ""
|
||||
|
||||
imagePullSecrets: []
|
||||
|
||||
nameOverride: ""
|
||||
fullnameOverride: ""
|
||||
|
||||
serviceAccount:
|
||||
create: true
|
||||
annotations: {}
|
||||
name: ""
|
||||
|
||||
podAnnotations: {}
|
||||
|
||||
securityContext:
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: true
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 8001
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
className: nginx
|
||||
annotations: {}
|
||||
hosts:
|
||||
- host: api.trading.example.com
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
tls: []
|
||||
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1000m"
|
||||
memory: 1Gi
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: 512Mi
|
||||
|
||||
autoscaling:
|
||||
enabled: true
|
||||
minReplicas: 2
|
||||
maxReplicas: 10
|
||||
targetCPUUtilizationPercentage: 80
|
||||
targetMemoryUtilizationPercentage: 80
|
||||
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 15
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
|
||||
nodeSelector: {}
|
||||
tolerations: []
|
||||
affinity: {}
|
||||
|
|
@ -1,6 +0,0 @@
|
|||
apiVersion: v2
|
||||
name: execute-service
|
||||
description: Trading execution service
|
||||
type: application
|
||||
version: 0.1.0
|
||||
appVersion: "0.1.0"
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
|
||||
{{/*
|
||||
Expand the name of the chart.
|
||||
*/}}
|
||||
{{- define "execute-service.name" -}}
|
||||
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create a default fully qualified app name.
|
||||
*/}}
|
||||
{{- define "execute-service.fullname" -}}
|
||||
{{- if .Values.fullnameOverride }}
|
||||
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- $name := default .Chart.Name .Values.nameOverride }}
|
||||
{{- if contains $name .Release.Name }}
|
||||
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create chart name and version as used by the chart label.
|
||||
*/}}
|
||||
{{- define "execute-service.chart" -}}
|
||||
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Common labels
|
||||
*/}}
|
||||
{{- define "execute-service.labels" -}}
|
||||
helm.sh/chart: {{ include "execute-service.chart" . }}
|
||||
{{ include "execute-service.selectorLabels" . }}
|
||||
{{- if .Chart.AppVersion }}
|
||||
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
|
||||
{{- end }}
|
||||
app.kubernetes.io/managed-by: {{ .Release.Service }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Selector labels
|
||||
*/}}
|
||||
{{- define "execute-service.selectorLabels" -}}
|
||||
app.kubernetes.io/name: {{ include "execute-service.name" . }}
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create the name of the service account to use
|
||||
*/}}
|
||||
{{- define "execute-service.serviceAccountName" -}}
|
||||
{{- if .Values.serviceAccount.create }}
|
||||
{{- default (include "execute-service.fullname" .) .Values.serviceAccount.name }}
|
||||
{{- else }}
|
||||
{{- default "default" .Values.serviceAccount.name }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "execute-service.fullname" . }}
|
||||
labels:
|
||||
{{- include "execute-service.labels" . | nindent 4 }}
|
||||
spec:
|
||||
{{- if not .Values.autoscaling.enabled }}
|
||||
replicas: {{ .Values.replicaCount }}
|
||||
{{- end }}
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "execute-service.selectorLabels" . | nindent 6 }}
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
{{- with .Values.podAnnotations }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
labels:
|
||||
{{- include "execute-service.selectorLabels" . | nindent 8 }}
|
||||
spec:
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
serviceAccountName: {{ include "execute-service.serviceAccountName" . }}
|
||||
securityContext:
|
||||
{{- toYaml .Values.podSecurityContext | nindent 8 }}
|
||||
containers:
|
||||
- name: {{ .Chart.Name }}
|
||||
securityContext:
|
||||
{{- toYaml .Values.securityContext | nindent 12 }}
|
||||
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8000
|
||||
protocol: TCP
|
||||
livenessProbe:
|
||||
{{- toYaml .Values.livenessProbe | nindent 12 }}
|
||||
readinessProbe:
|
||||
{{- toYaml .Values.readinessProbe | nindent 12 }}
|
||||
resources:
|
||||
{{- toYaml .Values.resources | nindent 12 }}
|
||||
{{- with .Values.nodeSelector }}
|
||||
nodeSelector:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
{{- with .Values.affinity }}
|
||||
affinity:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
{{- with .Values.tolerations }}
|
||||
tolerations:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
{{- if .Values.autoscaling.enabled }}
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: {{ include "execute-service.fullname" . }}
|
||||
labels:
|
||||
{{- include "execute-service.labels" . | nindent 4 }}
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: {{ include "execute-service.fullname" . }}
|
||||
minReplicas: {{ .Values.autoscaling.minReplicas }}
|
||||
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
|
||||
metrics:
|
||||
{{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
|
||||
{{- end }}
|
||||
{{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: {{ include "execute-service.fullname" . }}-network-policy
|
||||
labels:
|
||||
{{- include "execute-service.labels" . | nindent 4 }}
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
{{- include "execute-service.selectorLabels" . | nindent 6 }}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: api-gateway
|
||||
ports:
|
||||
- port: http
|
||||
protocol: TCP
|
||||
egress:
|
||||
# Allow DNS resolution
|
||||
- to: []
|
||||
ports:
|
||||
- port: 53
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
protocol: TCP
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "execute-service.fullname" . }}
|
||||
labels:
|
||||
{{- include "execute-service.labels" . | nindent 4 }}
|
||||
spec:
|
||||
type: {{ .Values.service.type }}
|
||||
ports:
|
||||
- port: {{ .Values.service.port }}
|
||||
targetPort: http
|
||||
protocol: TCP
|
||||
name: http
|
||||
selector:
|
||||
{{- include "execute-service.selectorLabels" . | nindent 4 }}
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
{{- if .Values.serviceAccount.create -}}
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: {{ include "execute-service.serviceAccountName" . }}
|
||||
labels:
|
||||
{{- include "execute-service.labels" . | nindent 4 }}
|
||||
{{- with .Values.serviceAccount.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,80 +0,0 @@
|
|||
# trading-platform/execute-service Helm chart values
|
||||
|
||||
replicaCount: 2
|
||||
|
||||
image:
|
||||
repository: trading-platform/execute-service
|
||||
pullPolicy: IfNotPresent
|
||||
tag: ""
|
||||
|
||||
imagePullSecrets: []
|
||||
|
||||
nameOverride: ""
|
||||
fullnameOverride: ""
|
||||
|
||||
serviceAccount:
|
||||
create: true
|
||||
annotations: {}
|
||||
name: ""
|
||||
|
||||
podAnnotations: {}
|
||||
|
||||
securityContext:
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: true
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 8000
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
className: nginx
|
||||
annotations: {}
|
||||
hosts:
|
||||
- host: api.trading.example.com
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
tls: []
|
||||
|
||||
resources:
|
||||
limits:
|
||||
cpu: "500m"
|
||||
memory: 512Mi
|
||||
requests:
|
||||
cpu: "250m"
|
||||
memory: 256Mi
|
||||
|
||||
autoscaling:
|
||||
enabled: true
|
||||
minReplicas: 2
|
||||
maxReplicas: 10
|
||||
targetCPUUtilizationPercentage: 80
|
||||
targetMemoryUtilizationPercentage: 80
|
||||
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 15
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
|
||||
nodeSelector: {}
|
||||
tolerations: []
|
||||
affinity: {}
|
||||
|
|
@ -1,6 +0,0 @@
|
|||
apiVersion: v2
|
||||
name: news-service
|
||||
description: News analysis service
|
||||
type: application
|
||||
version: 0.1.0
|
||||
appVersion: "0.1.0"
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
|
||||
{{/*
|
||||
Expand the name of the chart.
|
||||
*/}}
|
||||
{{- define "news-service.name" -}}
|
||||
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create a default fully qualified app name.
|
||||
*/}}
|
||||
{{- define "news-service.fullname" -}}
|
||||
{{- if .Values.fullnameOverride }}
|
||||
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- $name := default .Chart.Name .Values.nameOverride }}
|
||||
{{- if contains $name .Release.Name }}
|
||||
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create chart name and version as used by the chart label.
|
||||
*/}}
|
||||
{{- define "news-service.chart" -}}
|
||||
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Common labels
|
||||
*/}}
|
||||
{{- define "news-service.labels" -}}
|
||||
helm.sh/chart: {{ include "news-service.chart" . }}
|
||||
{{ include "news-service.selectorLabels" . }}
|
||||
{{- if .Chart.AppVersion }}
|
||||
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
|
||||
{{- end }}
|
||||
app.kubernetes.io/managed-by: {{ .Release.Service }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Selector labels
|
||||
*/}}
|
||||
{{- define "news-service.selectorLabels" -}}
|
||||
app.kubernetes.io/name: {{ include "news-service.name" . }}
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create the name of the service account to use
|
||||
*/}}
|
||||
{{- define "news-service.serviceAccountName" -}}
|
||||
{{- if .Values.serviceAccount.create }}
|
||||
{{- default (include "news-service.fullname" .) .Values.serviceAccount.name }}
|
||||
{{- else }}
|
||||
{{- default "default" .Values.serviceAccount.name }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "news-service.fullname" . }}
|
||||
labels:
|
||||
{{- include "news-service.labels" . | nindent 4 }}
|
||||
spec:
|
||||
{{- if not .Values.autoscaling.enabled }}
|
||||
replicas: {{ .Values.replicaCount }}
|
||||
{{- end }}
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "news-service.selectorLabels" . | nindent 6 }}
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
{{- with .Values.podAnnotations }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
labels:
|
||||
{{- include "news-service.selectorLabels" . | nindent 8 }}
|
||||
spec:
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
serviceAccountName: {{ include "news-service.serviceAccountName" . }}
|
||||
securityContext:
|
||||
{{- toYaml .Values.podSecurityContext | nindent 8 }}
|
||||
containers:
|
||||
- name: {{ .Chart.Name }}
|
||||
securityContext:
|
||||
{{- toYaml .Values.securityContext | nindent 12 }}
|
||||
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8002
|
||||
protocol: TCP
|
||||
livenessProbe:
|
||||
{{- toYaml .Values.livenessProbe | nindent 12 }}
|
||||
readinessProbe:
|
||||
{{- toYaml .Values.readinessProbe | nindent 12 }}
|
||||
resources:
|
||||
{{- toYaml .Values.resources | nindent 12 }}
|
||||
{{- with .Values.nodeSelector }}
|
||||
nodeSelector:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
{{- with .Values.affinity }}
|
||||
affinity:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
{{- with .Values.tolerations }}
|
||||
tolerations:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
{{- if .Values.autoscaling.enabled }}
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: {{ include "news-service.fullname" . }}
|
||||
labels:
|
||||
{{- include "news-service.labels" . | nindent 4 }}
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: {{ include "news-service.fullname" . }}
|
||||
minReplicas: {{ .Values.autoscaling.minReplicas }}
|
||||
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
|
||||
metrics:
|
||||
{{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
|
||||
{{- end }}
|
||||
{{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: {{ include "news-service.fullname" . }}-network-policy
|
||||
labels:
|
||||
{{- include "news-service.labels" . | nindent 4 }}
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
{{- include "news-service.selectorLabels" . | nindent 6 }}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: api-gateway
|
||||
ports:
|
||||
- port: http
|
||||
protocol: TCP
|
||||
egress:
|
||||
# Allow DNS resolution
|
||||
- to: []
|
||||
ports:
|
||||
- port: 53
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
protocol: TCP
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "news-service.fullname" . }}
|
||||
labels:
|
||||
{{- include "news-service.labels" . | nindent 4 }}
|
||||
spec:
|
||||
type: {{ .Values.service.type }}
|
||||
ports:
|
||||
- port: {{ .Values.service.port }}
|
||||
targetPort: http
|
||||
protocol: TCP
|
||||
name: http
|
||||
selector:
|
||||
{{- include "news-service.selectorLabels" . | nindent 4 }}
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Reference in a new issue