Commit graph

631 commits

Author SHA1 Message Date
sirius0xdev
e28d6ccf10 fix(trading): update network policy egress to siriusdevops-pgdb
Migrate trading platform DB egress rule from hermes-pgdb to
siriusdevops-pgdb to complete the trading DB migration. The
configmaps already point to siriusdevops-pgdb-rw but the
network policy still allowed egress to the old cluster.
2026-05-25 20:08:07 +00:00
sirius0xdev
c357b69743
Merge pull request #156 from sirius0xdev/fix/pgvector-image-and-cluster
chore(news-bot): switch DB to siriusdevops-pgdb (main apps cluster)
2026-05-25 14:56:56 -04:00
Hermes Agent
22498cd97b chore(news-bot): switch to siriusdevops-pgdb as main apps cluster
- news-app-db.yaml: point Database CR to siriusdevops-pgdb
- All ConfigMaps: DB_HOST=siriusdevops-pgdb-rw, LLM_BASE_URL for vLLM
- hermes-pgdb now dedicated to pgvector/RAG for agents only
- Aligns with CNPG extension pattern and active kanban (devops startupProbes, backend exec-service)

PR to master per repo rules.
2026-05-25 18:56:08 +00:00
sirius0xdev
67410a8ac0
Merge pull request #155 from sirius0xdev/fix/pgvector-image-and-cluster
feat: add pgvector image build and update cluster imageName
2026-05-25 14:03:31 -04:00
sirius0xdev
6ca9d9532f
Merge pull request #154 from sirius0xdev/feat/migrate-trading-to-siriusdevops-pgdb
feat: migrate all trading-platform services from hermes-pgdb to siriusdevops-pgdb
2026-05-25 14:03:20 -04:00
Hermes Agent
2f15684895 feat: add pgvector image build workflow and update cluster imageName
- Add build-pgvector-image.yml GH Actions workflow for GCR
- Set imageName to gcr.io/devops-lab-cluster/postgres-pgvector:15.2-0.8.0
- Image change only (CNPG rule: no config changes in same update)
2026-05-25 08:15:12 +00:00
Hermes Agent
3f96800222 feat: migrate all trading-platform services from hermes-pgdb to siriusdevops-pgdb
- Add 'trading' managed role to siriusdevops-pgdb cluster
- Move trading-db-credentials from hermes-db to siriusdevops-db
- Create trading-data-db.yaml (Database kind) pointing to siriusdevops-pgdb
- Update DB_HOST in all 4 trading-platform configmaps to siriusdevops-pgdb-rw
- Remove trading-data-db.yaml and trading-db-credentials.yaml from hermes-db
- Update siriusdevops-db kustomization.yaml to include new resources
2026-05-25 07:55:15 +00:00
Hermes DevOps
7e8fd296c4 fix(platform): allow tailscale namespace in NetworkPolicy for proxy access
Tailscale proxy pods run in the tailscale namespace, so the existing
customer1-only namespaceSelector blocked proxy → dashboard traffic.
2026-05-25 07:15:59 +00:00
sirius0xdev
10d9295eb2
Merge pull request #152 from sirius0xdev/fix/trading-dashboard-network-policy
fix(platform): add port 3000 to trading-platform NetworkPolicy
2026-05-25 03:07:18 -04:00
Hermes DevOps
113324abc1 fix(platform): add port 3000 to trading-platform NetworkPolicy ingress rules
Dashboard listens on port 3000 but NetworkPolicy only allowed 8000-8003.
This blocked all traffic including Tailscale proxy connections to trade:3000.
2026-05-25 07:05:45 +00:00
sirius0xdev
4f4f761fdd
Update kustomization.yaml 2026-05-25 02:44:21 -04:00
569a3d5de9 fix(dashboard): change tailscale hostname to "trade" for reliable resolution
Short hostname "trade" (matches /trade HTTPRoute) instead of "trading-dashboard".

This fixes the "Could not resolve host: trading-dashboard" error on tailnet devices while keeping the same annotation style as Grafana.
2026-05-25 06:32:24 +00:00
4be12802d0 fix(execute-service): add timeoutSeconds: 10 to startup/readiness probes
The /health/ready endpoint calls await initialize() on Hyperliquid and Solana executors. Without explicit timeout, the default 1s probe timeout caused repeated failures even though the app was running.

Added timeoutSeconds: 10 + reduced readiness failureThreshold.

This is the final piece for the "running but not ready" state.
2026-05-25 06:26:09 +00:00
605e15d55c fix(dashboard): add tmp volumeMount for readOnlyRootFilesystem
Next.js standalone mode needs a writable /tmp directory for cache, temp files, and logs.

With , the container was crashing after "Ready in 228ms" (EOF in log stream).

Added the standard emptyDir volumeMount used by other services.

This should make the dashboard stable and accessible on tailnet.
2026-05-25 06:17:28 +00:00
abecefe0e1 fix(execute-service): update startupProbe to /health/ready with 30 failureThreshold
The service was 'running but not ready' because:
- startupProbe used /health (always-ok) with only failureThreshold: 6 (~30s grace)
- Executor initialization (Hyperliquid SDK network calls + Solana client) takes longer

Now uses /health/ready (checks actual executor state) with failureThreshold: 30 (~5min grace), matching the pattern used for data-service and dashboard.

Also updated Helm template for consistency.
2026-05-25 06:09:20 +00:00
ceb832b169 fix(dashboard): add explicit tailscale.com/ports: "http:3000" annotation
Matches the exact annotation pattern used for other tailnet services (Grafana, trade-dashboard, hermes-webui, etc.).

This ensures the Tailscale operator correctly maps the exposed port after the 3000 change.
2026-05-25 06:05:51 +00:00
d002a7896b fix(dashboard): align port to 3000 and probes to root path
- containerPort: 3000 (matches Next.js default + Dockerfile ENV PORT=3000)
- All probes (startup/liveness/readiness): path  (Next.js serves root when ready)
- Service targetPort: 3000

Fixes 'not ready' and 'can't reach it' (previous mismatch with 8000 + /api/health which didn't exist).

Live manifests updated (source of truth).
2026-05-25 05:55:21 +00:00
a07259cd25 fix(trading-platform): increase startupProbe failureThreshold to 30 for data-service and dashboard
Gives ~5min grace period for heavy lifespan init (Redis, DB, executors, model loading).

Addresses 'up but not ready' state on data-service and dashboard (0/1 Ready).

Live manifests in apps/base/customer1/... updated (source of truth). Helm templates can be synced later.

See gke-hermes-infrastructure skill (trading-startup-probe-fix-2026-05-26.md).
2026-05-25 05:49:05 +00:00
9ca32408b1 Fix: readiness probe for execute-service uses /health/ready
Change readiness probe from /health (always ok) to /health/ready
(checks executor initialization state). Applied to both:
- apps/base K8s deployment
- Helm chart template

Pairs with app fix making executor init non-blocking.
2026-05-25 05:39:24 +00:00
Hermes Agent
605667df1d fix: wire JWT_SECRET_KEY in live kustomize path
Previous PR #139 added the secret to trading-platform/deploy/k8s/base/
but the live cluster uses apps/base/customer1/trading-platform/ kustomize.

- Create execute-service-jwt-secret.yaml (SOPS encrypted) in secrets/
- Add to secrets/ kustomization.yaml resources
- Wire JWT_SECRET_KEY env var in execute-service deployment
- Points to execute-service-jwt-secret via secretKeyRef
2026-05-25 01:53:19 +00:00
Hermes Agent
f17ab24ecc fix: update embedding-service image to GHCR
Point to ghcr.io/sirius0xdev/trading-embedding-service:latest
built automatically by hermes-projects CI on push to main
2026-05-25 01:35:28 +00:00
Hermes Agent
27b7334dfb fix: match pg-cluster manifest to live CNPG cluster state
- Remove imageName (was never applied, live cluster uses default CNPG image)
- Remove maintenance_work_mem (live cluster has no custom parameters)
- Set postgresql: {} to match live state
- Keep memory role superuser: true (the only needed change)

Fixes CNPG admission webhook: 'Can't change image name and configuration at the same time'
2026-05-25 01:11:27 +00:00
Hermes Agent
9f60ca894d fix: remove unsupported sql field from agent-memory-rag Database
CNPG version on cluster does not support .spec.sql on Database CR.
The rag-init-job.yaml already handles RAG schema initialization separately.

Unblocks Flux dry-run for agent-memory-rag.
2026-05-25 01:08:12 +00:00
Hermes Agent
9754c2aed2 fix: remove unsupported CNPG fields, keep memory superuser
- Remove sharedPreloadLibraries and allowedExtensions (not supported by installed CNPG version)
- Keep memory role superuser: true (from PR #135) — sufficient since custom pgvector image already bundles the extension
- Unblocks dry-run validation on hermes-pgdb cluster
2026-05-25 01:01:52 +00:00
sirius0xdev
ead9cff2d3
Merge pull request #134 from sirius0xdev/backend/t_6977ed0c-fix-execute-db-config
fix: align execute-service DB config with EXECUTE_ env prefix
2026-05-24 20:59:27 -04:00
sirius0xdev
9731719564
Merge pull request #135 from sirius0xdev/fix/t_3d94b392-cnpg-allowed-pgvector
fix: enable pgvector for mem0 plugin
2026-05-24 20:54:15 -04:00
sirius0xdev
8ccd21500e fix secrets 2026-05-25 00:49:20 +00:00
Hermes Agent
9ca7f7bd7a fix: enable pgvector for mem0 plugin
- Add allowedExtensions: ['*'] so roles can CREATE EXTENSION
- Grant superuser to 'memory' role (was createdb/createrole, insufficient)
- Needed for mem0 plugin pgvector backend on agent_memory DB

Unblocks kanban task t_3d94b392
2026-05-25 00:36:04 +00:00
Hermes Agent
a2644b39c7 fix: rename K8s env vars to EXECUTE_ prefix for execute-service
- ConfigMap keys: DB_HOST -> EXECUTE_DB_HOST, DB_PORT -> EXECUTE_DB_PORT, etc.
- Deployment secrets: DB_USER -> EXECUTE_DB_USER, DB_PASSWORD -> EXECUTE_DB_PASSWORD
- Helm configmap: add EXECUTE_DB_* keys alongside legacy POSTGRES_* keys
- Base deployment: add EXECUTE_DB_USER/PASSWORD from secrets
2026-05-24 23:57:04 +00:00
Hermes Agent
c04823d416 Grant CREATEDB and CREATEROLE to memory role on hermes-pgdb
The memory user needs CREATEDB and CREATEROLE privileges to install
the pgvector extension on the agent_memory database.
2026-05-24 22:14:49 +00:00
sirius0xdev
8bc3738a3f
Merge pull request #127 from sirius0xdev/feat/add-trading-dashboard-db-to-siriusdevops-pgdb
Feat/add trading dashboard db to siriusdevops pgdb
2026-05-24 16:56:27 -04:00
sirius0xdev
9f9bdba2b7
Merge pull request #128 from sirius0xdev/backend/t_438b663e-pgvector-rag-kb
feat(customer1): add pgvector RAG knowledge base with embedding service
2026-05-24 16:56:06 -04:00
Hermes Agent
9723674d9e feat(customer1): add embedding service kustomization, Service manifest, and staging reference 2026-05-24 20:33:50 +00:00
Hermes Agent
4df7450461 feat(customer1): add pgvector RAG knowledge base with embedding service
- Custom PostgreSQL 15.2 image with pgvector 0.8.0 extension
- Updated pg-cluster-hermes.yaml: custom image, sharedPreloadLibraries, maintenance_work_mem
- RAG schema: documents table with vector(768) embeddings + HNSW index
- RAG init job: ConfigMap + Job to apply schema to agent_memory db
- Embedding service: FastAPI with nomic-embed-text-v1.5
  - OpenAI-compatible /v1/embeddings endpoint
  - Deployment (1 replica, 2Gi-4Gi memory) + Service manifests
- Updated kustomization.yaml to include new resources
2026-05-24 20:31:59 +00:00
Hermes Agent
57e41178c3 feat(customer1): add trading_dashboard database to siriusdevops-pgdb CNPG cluster
- Add trading_dashboard role + Database CR on siriusdevops-pgdb
- Create SOPS-encrypted credentials secret (template)
- Update trade-dashboard to use the new dedicated DB
  - New host: siriusdevops-pgdb-rw
  - New database: trading_dashboard
  - New credentials secret
2026-05-24 20:02:20 +00:00
Hermes Agent
1d2d016eda fix(data-service): inject REDIS_URL from ConfigMap
- Add REDIS_URL env var using variable expansion from the existing
  trading-data-service-config ConfigMap (REDIS_HOST + REDIS_PORT).
- This resolves the 'connecting to localhost:6379' crash on startup.
- Matches the actual infrastructure configuration in customer1.
2026-05-23 18:34:51 +00:00
sirius0xdev
ac60dc5a51 Fix NATS URL service name mismatch and split combined ConfigMap+StatefulSet
- F8: Correct NATS_URLS, REDIS_URL, MINIO_ENDPOINT to use actual service names
  (nats, redis-master, minio) instead of fullname-prefixed names that don't exist
- F9: Split combined ConfigMap+StatefulSet in nats/statefulset.yaml into separate
  configmap.yaml and statefulset.yaml files for maintainability
2026-05-23 05:12:24 +00:00
sirius0xdev
54f2f42710 fix(osint-dashboard): fix critical manifest issues found during audit
- Fix PostgreSQL topologySpreadConstraints label selector (cnpg.io/cluster)
- Fix Ingress HTTPRoutes: route API and WS traffic to correct backends
- Fix API NetworkPolicy port mismatch (4000 -> 8000)
- Fix MinIO bucket index syntax (_0 -> 0) in ConfigMap
- Add postgresql.port and postgresql.database to values.yaml
- Add container-level securityContext to API deployment
- Restrict monitoring NetworkPolicy ingress to monitoring namespace
- Enable readOnlyRootFilesystem for MinIO container
- Add activeDeadlineSeconds (3600s) to all 4 CronJobs
2026-05-23 04:38:51 +00:00
sirius0xdev
cc2074a4cb change hermes image 2026-05-22 03:09:59 +00:00
sirius0xdev
f3c2e4991b change hermes image 2026-05-22 03:06:36 +00:00
sirius0xdev
c76675335d switch hermes image to main 2026-05-22 02:33:54 +00:00
sirius0xdev
75cea5abb7 fix secret issue 2026-05-22 02:11:22 +00:00
Hermes Agent
a433be1195 fix: resolve merge conflict in osint-dashboard values.yaml (keep real API config) 2026-05-21 14:17:52 +00:00
93754dcdf6 feat: OSINT Dashboard app + Helm templates
Add FastAPI backend for real-time geospatial OSINT dashboard:
- Full-text search via PostgreSQL tsvector (parameterized queries)
- Entity tracking, alert management, sentiment analytics
- Data ingestion: RSS feeds, GDELT, USGS earthquakes, social signals
- NATS JetStream consumer for event ingestion
- MinIO document storage integration
- Redis caching layer
- Alembic migrations with PostGIS + TimescaleDB extensions
- Single-page dashboard UI with live polling
- OpenTelemetry distributed tracing

Helm chart with infrastructure:
- CNPG PostgreSQL cluster (PostGIS + TimescaleDB)
- NATS JetStream with persistent streams
- MinIO distributed object storage (3 buckets)
- Redis Sentinel (1 primary + 2 replicas)
- NGINX Ingress with TLS and WebSocket support
- Prometheus + Grafana + Alertmanager monitoring stack
- Network policies with default deny
- ConfigMap, CronJob, Deployment, Service templates

Fixes applied during review:
- SQL injection in search endpoint (parameterized :q binding)
- Dockerfile PYTHONPATH mismatch (/app/app -> /app)
- Hardcoded DB credentials in alembic.ini
- RSS timestamp parsing (feedparser published_parsed -> parsedate_to_datetime)
- Removed dead PGVECTOR import
2026-05-21 13:39:52 +00:00
sirius0xdev
4a01c457ab
Update kustomization.yaml 2026-05-21 09:36:11 -04:00
8255467313 feat: add OSINT Dashboard Kubernetes infrastructure
- Helm chart scaffold (Chart.yaml, values.yaml, _helpers.tpl)
- Namespace + RBAC manifests
- PostgreSQL (CNPG, 3 replicas, PostGIS + TimescaleDB)
- NATS JetStream (3 replicas, persistent, custom subjects)
- Redis Sentinel (1 primary + 2 replicas, HA)
- MinIO distributed (4 replicas, bucket init job)
- Gateway API HTTPRoute + cert-manager TLS certificates
- Monitoring stack (Prometheus, Grafana, Alertmanager, exporters)
- NetworkPolicies (default deny + per-component policies)
- GitHub Actions CI/CD pipeline (lint, template, security scan)
- Flux CD staging overlay
2026-05-21 13:25:15 +00:00
b0748538c0 feat: add trading-platform K8s manifests for customer1
- Scaffold trading-platform under apps/base/customer1/trading-platform/
- Add 4 microservice deployments: data-service, execute-service, news-service, dashboard
- Add ConfigMaps with DB/Kafka/Redis connection configs
- Add HTTPRoutes via Gateway API (external-http-gateway)
- Add NetworkPolicies for inter-service and DB/Kafka/Redis egress
- Add SOPS-encrypted secrets (age key)
- Add Kafka (KRaft) and Redis infrastructure to hermes-db/
- Update root and staging kustomizations

All containers: non-root, readOnlyRootFilesystem, resource limits, health probes
Images: ghcr.io/sirius0xdev/trading-{service}:latest
2026-05-21 04:17:05 +00:00
5e477492e6 feat: add OSINT Dashboard Kubernetes infrastructure
- Helm chart scaffold (Chart.yaml, values.yaml, _helpers.tpl)
- Namespace + RBAC manifests
- PostgreSQL (CNPG, 3 replicas, PostGIS + TimescaleDB)
- NATS JetStream (3 replicas, persistent, custom subjects)
- Redis Sentinel (1 primary + 2 replicas, HA)
- MinIO distributed (4 replicas, bucket init job)
- Gateway API HTTPRoute + cert-manager TLS certificates
- Monitoring stack (Prometheus, Grafana, Alertmanager, exporters)
- NetworkPolicies (default deny + per-component policies)
- GitHub Actions CI/CD pipeline (lint, template, security scan)
- Flux CD staging overlay
2026-05-21 04:15:55 +00:00
sirius0xdev
a06a105f31
Update kustomization.yaml 2026-05-20 22:18:37 -04:00
sirius0xdev
1a3804294d
Update kustomization.yaml 2026-05-20 22:12:07 -04:00