refactor: remove trade-dashboard app code and rays-new-deployment.yaml
- Removed trade-dashboard/ (moved to hermes-projects/trade-dashboard/) - Removed .github/workflows/trade-dashboard.yml (CI now handled by hermes-projects root workflow) - Deleted rays-new-deployment.yaml (orphan deployment YAML) K8s manifests remain in apps/base/customer1/trade-dashboard/. See hermes-projects PR #8 for the addition side.
This commit is contained in:
Hermes Agent
parent
3a9a0476d2
commit
d89c1944e6
15 changed files with 648 additions and 955 deletions
52
.github/workflows/trade-dashboard.yml
vendored
52
.github/workflows/trade-dashboard.yml
vendored
|
|
@ -1,52 +0,0 @@
|
|||
name: Build and Push Trade Dashboard
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
paths:
|
||||
- 'trade-dashboard/**'
|
||||
workflow_dispatch:
|
||||
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_NAME: ${{ github.repository_owner }}/trade-dashboard
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Login to GHCR
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Extract metadata
|
||||
id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
tags: |
|
||||
type=sha,prefix=
|
||||
type=raw,value=latest,enable={{is_default_branch}}
|
||||
|
||||
- name: Build and push
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: trade-dashboard/
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
303
analyses/telegram-webhook-container-analysis.md
Normal file
303
analyses/telegram-webhook-container-analysis.md
Normal file
|
|
@ -0,0 +1,303 @@
|
|||
# Telegram Webhook Failure Analysis - Deep Dive (Container-Level)
|
||||
|
||||
**Date:** 2026-05-10
|
||||
**Repo:** https://github.com/sirius0xdev/gcloud-lab
|
||||
**Cluster:** devops-lab (GKE), Namespace: customer1
|
||||
**Webhook URL:** https://ws.siriusdevops.com/telegram/webhook/default
|
||||
|
||||
---
|
||||
|
||||
## Status Update
|
||||
|
||||
**TLS certificates ARE provisioned** in GCP admin console. The Gateway certmap annotation is working.
|
||||
|
||||
The real issues are in the **container configuration** and **missing probes**.
|
||||
|
||||
---
|
||||
|
||||
## Root Cause #1: NO Liveness/Readiness Probes (CONFIRMED CRITICAL)
|
||||
|
||||
**The hermes-agent deployment has ZERO probes defined.**
|
||||
|
||||
This means:
|
||||
- Kubernetes marks pods "Ready" immediately after container start
|
||||
- The Gateway routes webhook traffic before the process has bound to port 9118
|
||||
- During restarts, there is no graceful drain
|
||||
- **A crashed or hung pod stays in the endpoint list forever**
|
||||
|
||||
### The /health endpoint problem
|
||||
|
||||
The Telegram webhook server runs on **port 9118** via python-telegram-bot's `start_webhook()`. This internally starts an aiohttp server that **ONLY registers the webhook path** (`/telegram/webhook/default`). It does NOT expose a `/health` endpoint.
|
||||
|
||||
So a probe like this would FAIL:
|
||||
```yaml
|
||||
# THIS WON'T WORK - port 9118 has no /health
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 9118
|
||||
```
|
||||
|
||||
### What DOES have a health endpoint?
|
||||
|
||||
The **generic webhook adapter** on **port 8644** IS a Hermes-managed server. From the source code, it exposes `/health`. This IS enabled via `WEBHOOK_ENABLED: "true"`.
|
||||
|
||||
### The Fix
|
||||
|
||||
Add probes targeting port 8644 (the generic webhook server that IS healthy when the gateway is running):
|
||||
|
||||
```yaml
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8644
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8644
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 30
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
```
|
||||
|
||||
Alternatively, enable the API server on port 8642 and probe there (it also has `/health`).
|
||||
|
||||
---
|
||||
|
||||
## Root Cause #2: NO Resource Limits (HIGH)
|
||||
|
||||
The **new deployment dropped all resource limits** that existed in the old deployment.
|
||||
|
||||
**Old deployment (`deployment.yaml`):**
|
||||
```yaml
|
||||
resources:
|
||||
requests:
|
||||
memory: 2Gi
|
||||
cpu: "1"
|
||||
limits:
|
||||
memory: 3Gi
|
||||
cpu: "2"
|
||||
```
|
||||
|
||||
**New deployment (`new-deployment.yaml`):**
|
||||
```yaml
|
||||
# NO resources block for the hermes-agent container
|
||||
```
|
||||
|
||||
Only the `hermes-webui` sidecar has limits (500Mi-1Gi memory, 100m-500m CPU).
|
||||
|
||||
### Impact
|
||||
- The hermes-agent container can consume unbounded memory
|
||||
- With `agent.max_turns: 90` and `gateway_timeout: 1800` (30 min), a single agent run can eat massive memory
|
||||
- Pod may be OOMKilled by the node, but K8s won't restart it gracefully without probes
|
||||
- **This is a likely cause of intermittent failures**
|
||||
|
||||
### Fix
|
||||
Add resource limits back:
|
||||
```yaml
|
||||
resources:
|
||||
requests:
|
||||
memory: 2Gi
|
||||
cpu: "1"
|
||||
limits:
|
||||
memory: 4Gi
|
||||
cpu: "2"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Root Cause #3: Path Handling Concern (MEDIUM)
|
||||
|
||||
**Source code** (`/opt/hermes/gateway/platforms/telegram.py:1213`):
|
||||
```python
|
||||
webhook_path = urlparse(webhook_url).path or "/telegram"
|
||||
```
|
||||
|
||||
For `TELEGRAM_WEBHOOK_URL=https://ws.siriusdevops.com/telegram/webhook/default`:
|
||||
- `webhook_path` = `/telegram/webhook/default`
|
||||
- PTB's aiohttp server registers a handler at exactly `/telegram/webhook/default` on port 9118
|
||||
|
||||
**HTTPRoute** (`hermes-webhook.yaml`):
|
||||
```yaml
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
type: PathPrefix
|
||||
value: /telegram/webhook
|
||||
backendRefs:
|
||||
- name: http-tele-webhook
|
||||
port: 9118
|
||||
```
|
||||
|
||||
**The question:** Does the GKE Gateway strip the `/telegram/webhook` prefix before forwarding?
|
||||
|
||||
- If it does NOT strip: Pod receives `/telegram/webhook/default` -> **OK**
|
||||
- If it DOES strip to `/default`: Pod receives `/default` -> **404**
|
||||
- If it strips to `/`: Pod receives `/` -> **404**
|
||||
|
||||
Most GatewayAPI implementations pass the full original path by default, but some ingress controllers strip the matched prefix. **Verify this on the running cluster.**
|
||||
|
||||
---
|
||||
|
||||
## Root Cause #4: API Server Not Enabled (MEDIUM)
|
||||
|
||||
The old deployment had:
|
||||
```yaml
|
||||
- name: API_SERVER_ENABLED
|
||||
value: "true"
|
||||
- name: API_SERVER_HOST
|
||||
value: "0.0.0.0"
|
||||
- name: API_SERVER_PORT
|
||||
value: "8642"
|
||||
- name: API_SERVER_KEY
|
||||
valueFrom: {secretKeyRef: ...}
|
||||
- name: API_SERVER_MODEL_NAME
|
||||
value: "hermes-agent"
|
||||
```
|
||||
|
||||
**These are ALL absent from `new-deployment.yaml`.** Port 8642 is declared as a containerPort but nothing listens on it because `API_SERVER_ENABLED` is not set.
|
||||
|
||||
The API server exposes `/health` and `/health/detailed` endpoints. Without it, you lose a convenient health check and the API server interface.
|
||||
|
||||
---
|
||||
|
||||
## Other Findings
|
||||
|
||||
### 5. TELEGRAM_WEBHOOK_ENABLED is Ignored (INFO)
|
||||
|
||||
The code only checks if `TELEGRAM_WEBHOOK_URL` is set (non-empty). It does NOT read `TELEGRAM_WEBHOOK_ENABLED`. The env var in the deployment is a dead config — set but ignored.
|
||||
|
||||
### 6. vLLM Service Name Mismatch (INFO)
|
||||
|
||||
```yaml
|
||||
OPENAI_BASE_URL: "http://openclaw-brain-service.customer1.svc.cluster.local:8000/v1"
|
||||
```
|
||||
|
||||
But the active vLLM deployment (`rtx6000-vllm.yaml`) creates a service named `rtx6000-brain-service`. If Hermes ever switches from xAI to the openai provider, local vLLM is unreachable.
|
||||
|
||||
### 7. SOPS Secrets (VERIFY)
|
||||
|
||||
Both `hermes-secret.yaml` and `tele-webhook.yaml` are SOPS-encrypted. Verify they are decrypted in the cluster:
|
||||
```bash
|
||||
kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d
|
||||
kubectl get secret telegram-webhook -n customer1 -o jsonpath='{.data.TELEGRAM_WEBHOOK_SECRET}' | base64 -d
|
||||
```
|
||||
|
||||
### 8. Container Env Vars Summary
|
||||
|
||||
| Variable | Value | Notes |
|
||||
|----------|-------|-------|
|
||||
| TELEGRAM_WEBHOOK_URL | https://ws.siriusdevops.com/telegram/webhook/default | OK |
|
||||
| TELEGRAM_WEBHOOK_PORT | 9118 | OK |
|
||||
| TELEGRAM_WEBHOOK_SECRET | From SOPS secret | Verify decrypted |
|
||||
| TELEGRAM_WEBHOOK_ENABLED | "true" | **Ignored by code** |
|
||||
| TELEGRAM_BOT_TOKEN | From SOPS secret | Verify decrypted |
|
||||
| TELEGRAM_ALLOWED_USERS | 7528130947 | OK |
|
||||
| WEBHOOK_ENABLED | "true" | OK (enables port 8644 /health) |
|
||||
| WEBHOOK_PORT | 8644 | OK |
|
||||
| API_SERVER_ENABLED | **NOT SET** | Port 8642 has no listener |
|
||||
| HERMES_MODEL_PROVIDER | xai | OK (uses Grok) |
|
||||
| HERMES_MODEL | grok-4.20-0309-reasoning | OK |
|
||||
|
||||
---
|
||||
|
||||
## Recommended Fix (Priority Order)
|
||||
|
||||
### P0 - Add Probes (will detect and restart hung/crashed pods)
|
||||
|
||||
Add to `new-deployment.yaml` under the hermes-agent container spec:
|
||||
|
||||
```yaml
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8644
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8644
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 30
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
```
|
||||
|
||||
### P1 - Add Resource Limits (prevents OOM kills)
|
||||
|
||||
```yaml
|
||||
resources:
|
||||
requests:
|
||||
memory: 2Gi
|
||||
cpu: "1"
|
||||
limits:
|
||||
memory: 4Gi
|
||||
cpu: "2"
|
||||
```
|
||||
|
||||
### P2 - Verify Path Handling
|
||||
|
||||
Test if the Gateway passes the full path:
|
||||
```bash
|
||||
# From inside the pod, check what the webhook server receives
|
||||
kubectl exec -n customer1 deploy/hermes-agent -- curl -s http://localhost:9118/telegram/webhook/default -X POST -H "Content-Type: application/json" -d '{}'
|
||||
```
|
||||
|
||||
### P3 - Enable API Server (optional, gives /health on 8642)
|
||||
|
||||
Add back the API_SERVER_ENABLED env var if you want the API server health endpoint.
|
||||
|
||||
---
|
||||
|
||||
## Diagnostic Commands
|
||||
|
||||
Run these on the cluster RIGHT NOW to confirm the current state:
|
||||
|
||||
```bash
|
||||
# 1. Check pod status and restart count
|
||||
kubectl get pods -n customer1 -l app=hermes-agent -o wide
|
||||
|
||||
# 2. Check events for OOMKilled or probe failures
|
||||
kubectl describe pod -n customer1 -l app=hermes-agent | grep -A5 -i 'oom\|probe\|restart'
|
||||
|
||||
# 3. Check if the webhook port is actually listening
|
||||
kubectl exec -n customer1 deploy/hermes-agent -- ss -tlnp | grep 9118
|
||||
|
||||
# 4. Check if port 8644 health endpoint works
|
||||
kubectl exec -n customer1 deploy/hermes-agent -- curl -s http://localhost:8644/health
|
||||
|
||||
# 5. Check logs for webhook startup messages
|
||||
kubectl logs -n customer1 deploy/hermes-agent --tail=50 | grep -i 'webhook\|listening\|9118'
|
||||
|
||||
# 6. Verify SOPS secrets are decrypted
|
||||
kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d && echo
|
||||
kubectl get secret telegram-webhook -n customer1 -o jsonpath='{.data.TELEGRAM_WEBHOOK_SECRET}' | base64 -d && echo
|
||||
|
||||
# 7. Check if the webhook is registered with Telegram (using the pod's bot token)
|
||||
BOT_TOKEN=$(kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d)
|
||||
curl -s "https://api.telegram.org/bot${BOT_TOKEN}/getWebhookInfo" | python3 -m json.tool
|
||||
|
||||
# 8. Test the full path through the Gateway
|
||||
curl -v https://ws.siriusdevops.com/telegram/webhook/default -X POST -H "Content-Type: application/json" -d '{}' 2>&1
|
||||
|
||||
# 9. Check container memory usage
|
||||
kubectl top pod -n customer1 -l app=hermes-agent 2>/dev/null || echo "metrics-server not available"
|
||||
|
||||
# 10. Check resource limits on the container
|
||||
kubectl get pod -n customer1 -l app=hermes-agent -o jsonpath='{.items[0].spec.containers[0].resources}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Files to Modify
|
||||
|
||||
1. `apps/base/customer1/hermes-agent/new-deployment.yaml` - Add probes + resource limits
|
||||
2. Optionally: Re-enable API server env vars in `new-deployment.yaml`
|
||||
345
analyses/telegram-webhook-failure-analysis.md
Normal file
345
analyses/telegram-webhook-failure-analysis.md
Normal file
|
|
@ -0,0 +1,345 @@
|
|||
# Telegram Webhook Failure Analysis - Hermes Agent on GKE
|
||||
|
||||
**Date:** 2026-05-10
|
||||
**Repo:** https://github.com/sirius0xdev/gcloud-lab
|
||||
**Cluster:** devops-lab (GKE)
|
||||
**Namespace:** customer1
|
||||
**Webhook URL:** https://ws.siriusdevops.com/telegram/webhook/default
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The Telegram webhook is failing because **there is no valid TLS certificate for `ws.siriusdevops.com`** at the Gateway layer. Telegram strictly requires HTTPS with a publicly-trusted certificate for webhook delivery. The Gateway listener declares `protocol: HTTPS` but has no `tls.certificateRefs` and relies on a GKE `CertMap` annotation referencing `gateway-cert-map` — a resource that **does not exist** in the repository. Cert-manager is configured but disconnected from the GatewayAPI setup (HTTP-01 solver points to a non-existent Traefik ingress class).
|
||||
|
||||
**TL;DR:** Telegram tries to POST to `https://ws.siriusdevops.com/...`, but the Gateway has no certificate to present during the TLS handshake. Connection fails before it ever reaches the hermes-agent pod.
|
||||
|
||||
---
|
||||
|
||||
## Root Cause #1: Missing TLS Certificate (CRITICAL)
|
||||
|
||||
### What's happening
|
||||
|
||||
**File:** `infrastructure/gatewayapi/apigateway.yaml`
|
||||
|
||||
```yaml
|
||||
listeners:
|
||||
- name: https
|
||||
protocol: HTTPS
|
||||
port: 443
|
||||
allowedRoutes:
|
||||
namespaces:
|
||||
from: All
|
||||
```
|
||||
|
||||
The listener says HTTPS but has **no `tls:` block**. It relies entirely on this annotation:
|
||||
|
||||
```yaml
|
||||
annotations:
|
||||
networking.gke.io/certmap: gateway-cert-map
|
||||
```
|
||||
|
||||
### The problem
|
||||
|
||||
- **No `CertMap` or `CertMapEntry` resource** exists anywhere in the repository for `gateway-cert-map`
|
||||
- Without it, GKE has no managed certificate to attach to the Gateway
|
||||
- Telegram's webhook delivery gets a TLS handshake failure or no certificate
|
||||
|
||||
### Cert-manager is also broken
|
||||
|
||||
**File:** `infrastructure/controllers/base/certmanager/clusterissuer.yaml`
|
||||
|
||||
```yaml
|
||||
solvers:
|
||||
- http01:
|
||||
ingress:
|
||||
class: traefik
|
||||
```
|
||||
|
||||
- HTTP-01 solver references `class: traefik`, but **no Traefik ingress controller exists** in the cluster
|
||||
- **No `Certificate` CRs** exist for `ws.siriusdevops.com` or any other domain
|
||||
- Cert-manager is completely disconnected from the GatewayAPI setup
|
||||
|
||||
### How to fix (choose ONE approach)
|
||||
|
||||
**Option A: GKE Managed Certificates (recommended for GatewayAPI)**
|
||||
|
||||
Create a `ManagedCertificate` + `BackendConfig` or `CertMap`/`CertMapEntry`:
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.gke.io/v1
|
||||
kind: ManagedCertificate
|
||||
metadata:
|
||||
name: hermes-webhook-cert
|
||||
namespace: customer1
|
||||
spec:
|
||||
domains:
|
||||
- ws.siriusdevops.com
|
||||
- brain.siriusdevops.com
|
||||
- paaas.siriusdevops.com
|
||||
```
|
||||
|
||||
Then add `tls.certificateRefs` to the Gateway listener:
|
||||
|
||||
```yaml
|
||||
listeners:
|
||||
- name: https
|
||||
protocol: HTTPS
|
||||
port: 443
|
||||
tls:
|
||||
certificateRefs:
|
||||
- name: hermes-webhook-cert
|
||||
group: networking.gke.io
|
||||
```
|
||||
|
||||
**Option B: Fix Cert-manager with DNS-01**
|
||||
|
||||
Switch ClusterIssuer from HTTP-01/Traefik to DNS-01 (e.g., Cloudflare, GCP DNS, or Route53), then create `Certificate` resources for each domain.
|
||||
|
||||
---
|
||||
|
||||
## Root Cause #2: No Readiness/Liveness Probes (HIGH)
|
||||
|
||||
**File:** `apps/base/customer1/hermes-agent/new-deployment.yaml`
|
||||
|
||||
No readiness or liveness probes are defined on any hermes-agent deployment.
|
||||
|
||||
### Impact
|
||||
- Pods are marked "Ready" immediately after container start
|
||||
- Gateway routes traffic to the webhook port (9118) before the process has bound to it
|
||||
- During restarts, traffic hits pods that haven't initialized
|
||||
|
||||
### Fix
|
||||
Add probes to the deployment:
|
||||
|
||||
```yaml
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 9118
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 9118
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 30
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Root Cause #3: SOPS Encrypted Secrets (HIGH)
|
||||
|
||||
**Files:**
|
||||
- `apps/base/customer1/hermes-agent/hermes-secret.yaml` (SOPS encrypted)
|
||||
- `apps/base/customer1/hermes-agent/tele-webhook.yaml` (SOPS encrypted)
|
||||
|
||||
These contain `TELEGRAM_BOT_TOKEN` and `TELEGRAM_WEBHOOK_SECRET`.
|
||||
|
||||
### Risk
|
||||
- If the deployment pipeline (Flux/Kustomize controller) is **not decrypting SOPS secrets**, pods receive literal `ENC[...]` strings
|
||||
- The bot token would be invalid, so even if TLS worked, Telegram auth would fail
|
||||
- The webhook secret would not match, causing Telegram to reject payloads
|
||||
|
||||
### Verify
|
||||
Run `kubectl get secret hermes-secrets -n customer1 -o yaml` and check if values are base64-encoded real tokens or `ENC[...]` strings.
|
||||
|
||||
---
|
||||
|
||||
## Secondary Issues
|
||||
|
||||
### 3a. Service Name Mismatch (vLLM Integration)
|
||||
|
||||
**File:** `apps/base/customer1/hermes-agent/new-deployment.yaml`
|
||||
|
||||
```yaml
|
||||
OPENAI_BASE_URL: "http://openclaw-brain-service.customer1.svc.cluster.local:8000/v1"
|
||||
```
|
||||
|
||||
But the active vLLM deployment (`rtx6000-vllm.yaml`) creates a service named **`rtx6000-brain-service`**.
|
||||
|
||||
- `OPENAI_BASE_URL` points to `openclaw-brain-service` which may not exist
|
||||
- If Hermes ever switches to the `openai` provider (instead of `xai`), local vLLM is unreachable
|
||||
- The model provider defaults to `HERMES_MODEL_PROVIDER: xai` (Grok external API)
|
||||
|
||||
### 3b. Duplicate HTTPRoute Deployment
|
||||
|
||||
The webhook HTTPRoute (`hermes-webhook.yaml`) is included in **two** kustomization trees:
|
||||
|
||||
1. `infrastructure/gatewayapi/gateway-routes/kustomization.yaml` -> deployed via Flux
|
||||
2. `apps/base/customer1/hermes-agent/kustomization.yaml` -> deployed via Flux
|
||||
|
||||
Same resource (`http-telegram-webhook` in `customer1`) from two sources. This may cause Flux reconciliation conflicts.
|
||||
|
||||
### 3c. Empty HF_TOKEN in vLLM Deployments
|
||||
|
||||
All vLLM deployments have:
|
||||
|
||||
```yaml
|
||||
- name: HF_TOKEN
|
||||
value: ""
|
||||
```
|
||||
|
||||
If the model `edp1096/Huihui-Qwen3.6-27B-abliterated-FP8` is a gated model on HuggingFace, it will fail to download.
|
||||
|
||||
### 3d. KEDA Scale-to-Zero
|
||||
|
||||
**File:** `infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml`
|
||||
|
||||
```yaml
|
||||
minReplicaCount: 0
|
||||
maxReplicaCount: 1
|
||||
```
|
||||
|
||||
- vLLM scales to **zero** when idle
|
||||
- First request after cold start incurs full model load time (30-60 seconds)
|
||||
- For real-time Telegram responses, this causes visible latency
|
||||
|
||||
### 3e. PVC Name Collision
|
||||
|
||||
Both `rtx6000-vllm.yaml` and `a100-vllm.yaml` define a PVC named `vllm-model-qwen3.6-27b-uncensored` in namespace `customer1`. If both are ever active simultaneously, they conflict.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
Internet
|
||||
|
|
||||
v
|
||||
[GKE External LB]
|
||||
|
|
||||
Gateway: external-http-gateway
|
||||
(port 443/HTTPS, NO TLS cert!)
|
||||
|
|
||||
+-----------+-----------+
|
||||
| | |
|
||||
ws.siriusdevops.com brain.siriusdevops.com paaas.siriusdevops.com
|
||||
| | |
|
||||
v v v
|
||||
/telegram/webhook / /
|
||||
| | |
|
||||
v v v
|
||||
http-tele-webhook rtx6000- paaas-landing
|
||||
:9118 brain- :8080
|
||||
service:8000
|
||||
|
|
||||
v
|
||||
hermes-agent pod
|
||||
(ports: 8642, 8644, 9118)
|
||||
|
|
||||
v
|
||||
Model Provider: xai (Grok via external API)
|
||||
Fallback: OPENAI_BASE_URL -> openclaw-brain-service (MISMATCHED)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## vLLM Server Status
|
||||
|
||||
| Server | GPU | Model | Quantization | Status |
|
||||
|--------|-----|-------|-------------|--------|
|
||||
| RTX 6000 | 1x Pro 6000 (96GB) | edp1096/Huihui-Qwen3.6-27B-abliterated-FP8 | FP8 | **ACTIVE** |
|
||||
| A100 | 1x A100 (80GB) | Youssofal/Qwen3.6-27B-Abliterated-Heretic-Uncensored-BF16 | BF16 | Commented out |
|
||||
| L4 | 1x L4 (24GB) | p-e-w/Qwen3-8B-heretic | auto | Commented out |
|
||||
| Gemma | 1x A100 (80GB) | coder3101/Qwen3.5-27B-heretic | BF16 | Not in kustomization |
|
||||
|
||||
**Note:** The architecture plan (`plans/AI_ARCHITECTURE.md`) describes a dual-tier L4 dispatcher + A100 deep thinker setup, but the active deployment only has RTX 6000.
|
||||
|
||||
---
|
||||
|
||||
## Action Plan (Priority Order)
|
||||
|
||||
### P0 - Fix TLS (will unblock Telegram webhooks)
|
||||
|
||||
1. **Create a `CertMap`/`CertMapEntry`** or **`ManagedCertificate`** resource for `ws.siriusdevops.com`
|
||||
2. **Add `tls.certificateRefs`** to the Gateway listener in `apigateway.yaml`
|
||||
3. Verify with: `curl -vI https://ws.siriusdevops.com/telegram/webhook/default`
|
||||
4. If cert is valid, Telegram should start delivering webhooks
|
||||
|
||||
### P1 - Verify Secrets
|
||||
|
||||
5. Check if SOPS secrets are actually decrypted in the cluster
|
||||
6. Run: `kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d`
|
||||
|
||||
### P2 - Add Probes
|
||||
|
||||
7. Add readiness/liveness probes to hermes-agent deployment
|
||||
8. Redeploy
|
||||
|
||||
### P3 - Clean Up GatewayAPI
|
||||
|
||||
9. Remove duplicate `hermes-webhook.yaml` reference from one kustomization
|
||||
10. Fix or remove Traefik-referencing ClusterIssuers
|
||||
|
||||
### P4 - Fix vLLM Integration
|
||||
|
||||
11. Fix `OPENAI_BASE_URL` service name or update kustomization to create `openclaw-brain-service`
|
||||
12. Set real `HF_TOKEN` in vLLM deployments
|
||||
13. Consider setting `minReplicaCount: 1` in KEDA for consistent response times
|
||||
|
||||
---
|
||||
|
||||
## Files Referenced
|
||||
|
||||
### GatewayAPI & TLS
|
||||
- `infrastructure/gatewayapi/apigateway.yaml` - Gateway definition (missing TLS)
|
||||
- `infrastructure/gatewayapi/gateway-routes/hermes-webhook.yaml` - Webhook HTTPRoute
|
||||
- `infrastructure/gatewayapi/gateway-routes/route.yaml` - General route
|
||||
- `infrastructure/controllers/base/certmanager/clusterissuer.yaml` - Cert-manager (Traefik mismatch)
|
||||
|
||||
### Hermes Agent Deployment
|
||||
- `apps/base/customer1/hermes-agent/new-deployment.yaml` - Active deployment + webhook service
|
||||
- `apps/base/customer1/hermes-agent/deployment.yaml` - Old deployment (no webhook config)
|
||||
- `apps/base/customer1/hermes-agent/configmap.yaml` - Hermes config
|
||||
- `apps/base/customer1/hermes-agent/hermes-secret.yaml` - SOPS encrypted secrets
|
||||
- `apps/base/customer1/hermes-agent/tele-webhook.yaml` - SOPS encrypted webhook secret
|
||||
- `apps/base/customer1/hermes-agent/kustomization.yaml` - Kustomize composition
|
||||
|
||||
### vLLM Infrastructure
|
||||
- `infrastructure/gpus/base/vllm-servers/rtx6000-vllm.yaml` - Active vLLM server
|
||||
- `infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml` - KEDA scaling
|
||||
- `infrastructure/gpus/base/keda-gpu-scaling/vllm-route.yaml` - vLLM external route
|
||||
|
||||
### Architecture Plans
|
||||
- `plans/AI_ARCHITECTURE.md` - AI architecture plan
|
||||
- `plans/models-to-try.md` - Models being considered
|
||||
|
||||
---
|
||||
|
||||
## Quick Diagnostic Commands
|
||||
|
||||
Run these on the cluster to confirm findings:
|
||||
|
||||
```bash
|
||||
# 1. Check if TLS cert exists for the webhook domain
|
||||
curl -vI https://ws.siriusdevops.com/telegram/webhook/default 2>&1 | grep -E 'SSL|certificate|subject|issuer'
|
||||
|
||||
# 2. Check Gateway status
|
||||
kubectl get gateway external-http-gateway -n customer1 -o yaml
|
||||
|
||||
# 3. Check HTTPRoute status (attached/programmed)
|
||||
kubectl get httproute http-telegram-webhook -n customer1 -o yaml
|
||||
|
||||
# 4. Verify secrets are decrypted
|
||||
kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d && echo
|
||||
|
||||
# 5. Check if pods are actually ready
|
||||
kubectl get pods -n customer1 -l app=hermes-agent -o wide
|
||||
|
||||
# 6. Check CertMap exists
|
||||
kubectl get certmap gateway-cert-map -A 2>/dev/null || echo "CertMap NOT FOUND"
|
||||
|
||||
# 7. Check managed certificates
|
||||
kubectl get managedcertificate -A 2>/dev/null || echo "No ManagedCertificates"
|
||||
|
||||
# 8. Check cert-manager certificates
|
||||
kubectl get certificate -A 2>/dev/null || echo "No Certificates"
|
||||
|
||||
# 9. Check vLLM pod status
|
||||
kubectl get pods -n customer1 -l app=rtx6000-brain-vllm -o wide
|
||||
|
||||
# 10. Test internal webhook endpoint
|
||||
kubectl exec -n customer1 deploy/hermes-agent -- curl -s http://localhost:9118/health || echo "Health check failed"
|
||||
```
|
||||
|
|
@ -1,217 +0,0 @@
|
|||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: rays-hermes-agent
|
||||
namespace: customer1
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: rays-hermes-agent
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: rays-hermes-agent
|
||||
spec:
|
||||
# 1. Pod-level security context to ensure volumes inherit the right group
|
||||
shareProcessNamespace: true
|
||||
securityContext:
|
||||
fsGroup: 1000
|
||||
|
||||
# 2. Define our shared bridge volumes
|
||||
volumes:
|
||||
- name: hermes-home
|
||||
persistentVolumeClaim:
|
||||
claimName: rays-hermes-agent-pvc
|
||||
- name: hermes-agent-src
|
||||
emptyDir: {}
|
||||
- name: hermes-workspace
|
||||
emptyDir: {}
|
||||
- name: hermes-webui-app
|
||||
emptyDir: {}
|
||||
- name: hermes-configmap
|
||||
configMap:
|
||||
name: hermes-config
|
||||
|
||||
initContainers:
|
||||
# 3. K8s workaround: Copy the agent source code into the shared emptyDir
|
||||
- name: copy-agent-source
|
||||
image: nousresearch/hermes-agent:latest
|
||||
command:
|
||||
- "sh"
|
||||
- "-c"
|
||||
- |
|
||||
cp -a /opt/hermes/. /shared-src/ && chown -R 1024:1000 /shared-src /shared-home
|
||||
|
||||
if [ -f /tmp/hermes/config.yaml ]; then
|
||||
cp -f /tmp/hermes/config.yaml /shared-home/config.yaml
|
||||
fi
|
||||
|
||||
mkdir -p /shared-home/.local/bin
|
||||
echo '#!/bin/sh' > /shared-home/.local/bin/gh
|
||||
echo 'exit 1' >> /shared-home/.local/bin/gh
|
||||
chmod +x /shared-home/.local/bin/gh
|
||||
|
||||
securityContext:
|
||||
runAsUser: 0 # Run as root briefly to copy and fix permissions
|
||||
runAsNonRoot: false
|
||||
volumeMounts:
|
||||
- name: hermes-agent-src
|
||||
mountPath: /shared-src
|
||||
- name: hermes-home
|
||||
mountPath: /shared-home
|
||||
- name: hermes-configmap
|
||||
mountPath: /tmp/hermes/config.yaml
|
||||
subPath: config.yaml
|
||||
containers:
|
||||
# ==========================================
|
||||
# CONTAINER 1: HERMES AGENT
|
||||
# ==========================================
|
||||
- name: rays-hermes-agent
|
||||
image: nousresearch/hermes-agent:latest
|
||||
args: ["gateway", "run"]
|
||||
ports:
|
||||
- containerPort: 8642
|
||||
|
||||
env:
|
||||
- name: PATH
|
||||
value: "/home/hermes/.hermes/.local/bin:/opt/hermes/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
||||
- name: HOME
|
||||
value: "/home/hermes/.hermes"
|
||||
- name: HERMES_HOME
|
||||
value: "/home/hermes/.hermes"
|
||||
- name: HERMES_UID
|
||||
value: "1024"
|
||||
- name: HERMES_GID
|
||||
value: "1000"
|
||||
|
||||
- name: TELEGRAM_BOT_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hermes-secrets
|
||||
key: TELEGRAM_BOT_TOKEN
|
||||
|
||||
- name: XAI_API_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: xai-apikey
|
||||
key: XAI_API_KEY
|
||||
|
||||
- name: TELEGRAM_ALLOWED_USERS
|
||||
value: "7105451284"
|
||||
|
||||
# === Local vLLM (OpenAI-compatible) ===
|
||||
- name: OPENAI_BASE_URL
|
||||
value: "http://openclaw-brain-service.customer1.svc.cluster.local:8000/v1" # ← adjust if your service name differs
|
||||
|
||||
- name: HERMES_MODEL_PROVIDER
|
||||
value: xai
|
||||
|
||||
- name: HERMES_MODEL
|
||||
value: grok-4.20-0309-reasoning
|
||||
|
||||
- name: OPENAI_API_KEY
|
||||
value: "dummy"
|
||||
|
||||
|
||||
|
||||
volumeMounts:
|
||||
- name: hermes-home
|
||||
mountPath: /home/hermes/.hermes
|
||||
- name: hermes-agent-src
|
||||
mountPath: /opt/hermes
|
||||
|
||||
securityContext:
|
||||
runAsUser: 1024
|
||||
runAsGroup: 1000
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: true
|
||||
|
||||
# ==========================================
|
||||
# CONTAINER 2: HERMES WEBUI
|
||||
# ==========================================
|
||||
- name: hermes-webui
|
||||
image: ghcr.io/nesquena/hermes-webui:latest
|
||||
ports:
|
||||
- containerPort: 8787
|
||||
|
||||
env:
|
||||
- name: HOME
|
||||
value: "/home/hermeswebui/.hermes"
|
||||
- name: HERMES_HOME
|
||||
value: "/home/hermeswebui/.hermes"
|
||||
- name: HERMES_WEBUI_HOST
|
||||
value: "0.0.0.0"
|
||||
- name: HERMES_WEBUI_PORT
|
||||
value: "8787"
|
||||
- name: HERMES_WEBUI_STATE_DIR
|
||||
value: "/home/hermeswebui/.hermes/webui"
|
||||
- name: WANTED_UID
|
||||
value: "1024"
|
||||
- name: WANTED_GID
|
||||
value: "1000"
|
||||
- name: HERMES_SKIP_CHMOD
|
||||
value: "1"
|
||||
|
||||
volumeMounts:
|
||||
- name: hermes-home
|
||||
mountPath: /home/hermeswebui/.hermes
|
||||
# This is where the WebUI looks for the agent source code to run `uv pip install`
|
||||
- name: hermes-agent-src
|
||||
mountPath: /home/hermeswebui/.hermes/hermes-agent
|
||||
- name: hermes-workspace
|
||||
mountPath: /workspace
|
||||
- name: hermes-webui-app
|
||||
mountPath: /app
|
||||
|
||||
resources:
|
||||
requests:
|
||||
memory: 500Mi
|
||||
cpu: "100m"
|
||||
limits:
|
||||
memory: 1Gi
|
||||
cpu: "500m"
|
||||
|
||||
securityContext:
|
||||
runAsUser: 1024
|
||||
runAsGroup: 1000
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: true
|
||||
readOnlyRootFilesystem: false
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
---
|
||||
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: rays-hermes-agent-pvc
|
||||
namespace: customer1
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 25Gi
|
||||
|
||||
---
|
||||
|
||||
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: rays-hermes-webui-service
|
||||
namespace: customer1
|
||||
annotations:
|
||||
tailscale.com/expose: "true"
|
||||
tailscale.com/hostname: "rays-hermes-webui"
|
||||
tailscale.com/tags: "tag:k8s-operator"
|
||||
tailscale.com/ports: "http:8787"
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app: hermes-agent
|
||||
ports:
|
||||
- port: 8787
|
||||
targetPort: 8787
|
||||
name: http
|
||||
|
|
@ -1,18 +0,0 @@
|
|||
FROM python:3.13-slim AS base
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
gcc libpq-dev \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY app/requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY app/ ./app/
|
||||
COPY alembic.ini ./alembic.ini
|
||||
COPY alembic/ ./alembic/
|
||||
|
||||
EXPOSE 8000
|
||||
ENV PYTHONPATH=/app/app
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
|
|
@ -1,36 +0,0 @@
|
|||
[alembic]
|
||||
script_location = alembic
|
||||
sqlalchemy.url = postgresql+asyncpg://trading:CHANGE_ME@hermes-pgdb-rw.customer1.svc.cluster.local:5432/trading_data
|
||||
|
||||
[loggers]
|
||||
keys = root,sqlalchemy,alembic
|
||||
|
||||
[handlers]
|
||||
keys = console
|
||||
|
||||
[formatters]
|
||||
keys = generic
|
||||
|
||||
[logger_root]
|
||||
level = WARN
|
||||
handlers = console
|
||||
|
||||
[logger_sqlalchemy]
|
||||
level = WARN
|
||||
handlers =
|
||||
qualname = sqlalchemy.engine
|
||||
|
||||
[logger_alembic]
|
||||
level = INFO
|
||||
handlers =
|
||||
qualname = alembic
|
||||
|
||||
[handler_console]
|
||||
class = StreamHandler
|
||||
args = (sys.stderr,)
|
||||
level = NOTSET
|
||||
formatter = generic
|
||||
|
||||
[formatter_generic]
|
||||
format = %(levelname)-5.5s [%(name)s] %(message)s
|
||||
datefmt = %H:%M:%S
|
||||
|
|
@ -1,62 +0,0 @@
|
|||
"""Alembic environment configuration."""
|
||||
|
||||
import sys
|
||||
from logging.config import fileConfig
|
||||
from pathlib import Path
|
||||
|
||||
from alembic import context
|
||||
from sqlalchemy import engine_from_config, pool
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "app"))
|
||||
|
||||
config = context.config
|
||||
|
||||
if config.config_file_name is not None:
|
||||
fileConfig(config.config_file_name)
|
||||
|
||||
from models import metadata # noqa: E402
|
||||
|
||||
target_metadata = metadata
|
||||
|
||||
|
||||
def run_migrations_offline() -> None:
|
||||
"""Run migrations in 'offline' mode."""
|
||||
url = config.get_main_option("sqlalchemy.url")
|
||||
context.configure(
|
||||
url=url,
|
||||
target_metadata=target_metadata,
|
||||
literal_binds=True,
|
||||
dialect_opts={"paramstyle": "named"},
|
||||
)
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
def do_run_migrations(connection):
|
||||
context.configure(connection=connection, target_metadata=target_metadata)
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
async def run_migrations_online() -> None:
|
||||
"""Run migrations in 'online' mode."""
|
||||
connectable = AsyncEngine(
|
||||
engine_from_config(
|
||||
config.get_section(config.config_ini_section) or {},
|
||||
prefix="sqlalchemy.",
|
||||
poolclass=pool.NullPool,
|
||||
future=True,
|
||||
)
|
||||
)
|
||||
async with connectable.connect() as connection:
|
||||
await connection.run_sync(do_run_migrations)
|
||||
await connectable.dispose()
|
||||
|
||||
|
||||
import asyncio
|
||||
|
||||
if context.is_offline_mode():
|
||||
run_migrations_offline()
|
||||
else:
|
||||
asyncio.run(run_migrations_online())
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
# Alembic migration script - DO NOT EDIT MANUALLY
|
||||
# Use: alembic revision --autogenerate -m "description"
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
"""initial schema — positions table
|
||||
|
||||
Revision ID: 001_initial
|
||||
Create Date: 2026-05-02
|
||||
"""
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
revision = "001_initial"
|
||||
down_revision = None
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute('CREATE TYPE position_direction AS ENUM (\'long\', \'short\')')
|
||||
|
||||
op.create_table(
|
||||
"positions",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True, server_default=sa.text("gen_random_uuid()")),
|
||||
sa.Column("symbol", sa.String(32), nullable=False),
|
||||
sa.Column("direction", postgresql.ENUM("long", "short", name="position_direction", create_type=False), nullable=False),
|
||||
sa.Column("entry_price", sa.Numeric(precision=16, scale=8), nullable=False),
|
||||
sa.Column("exit_price", sa.Numeric(precision=16, scale=8)),
|
||||
sa.Column("quantity", sa.Numeric(precision=16, scale=8), nullable=False),
|
||||
sa.Column("exchange", sa.String(32), nullable=False),
|
||||
sa.Column("opened_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
sa.Column("closed_at", sa.DateTime(timezone=True)),
|
||||
sa.Column("pnl", sa.Numeric(precision=16, scale=2)),
|
||||
sa.Column("metadata", sa.JSON),
|
||||
)
|
||||
|
||||
op.create_index(op.f("ix_positions_symbol"), "positions", ["symbol"])
|
||||
op.create_index(op.f("ix_positions_exchange"), "positions", ["exchange"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index(op.f("ix_positions_exchange"), table_name="positions")
|
||||
op.drop_index(op.f("ix_positions_symbol"), table_name="positions")
|
||||
op.drop_table("positions")
|
||||
op.execute("DROP TYPE IF EXISTS position_direction")
|
||||
|
|
@ -1,16 +0,0 @@
|
|||
import os
|
||||
|
||||
from sqlalchemy import MetaData
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
|
||||
|
||||
DB_USER = os.getenv("DB_USER", "trading")
|
||||
DB_PASS = os.getenv("DB_PASSWORD", "")
|
||||
DB_HOST = os.getenv("DB_HOST", "hermes-pgdb-rw.customer1.svc.cluster.local")
|
||||
DB_PORT = os.getenv("DB_PORT", "5432")
|
||||
DB_NAME = os.getenv("DB_NAME", "trading_data")
|
||||
|
||||
DATABASE_URL = f"postgresql+asyncpg://{DB_USER}:{DB_PASS}@{DB_HOST}:{DB_PORT}/{DB_NAME}"
|
||||
|
||||
engine = create_async_engine(DATABASE_URL, echo=False, pool_size=5, max_overflow=10)
|
||||
async_session = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
|
||||
metadata = MetaData()
|
||||
|
|
@ -1,253 +0,0 @@
|
|||
"""Trade Dashboard — FastAPI service for tracking PnL and open positions."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import FastAPI, HTTPException, Query
|
||||
from fastapi.responses import FileResponse, HTMLResponse
|
||||
from sqlalchemy import and_, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from database import async_session
|
||||
from models import positions
|
||||
from schemas import (
|
||||
Direction,
|
||||
PnLSnapshot,
|
||||
PositionCreate,
|
||||
PositionOut,
|
||||
PositionUpdate,
|
||||
WebhookTrade,
|
||||
)
|
||||
|
||||
app = FastAPI(title="Trade Dashboard", version="0.1.0")
|
||||
|
||||
STATIC_DIR = Path(__file__).parent / "static"
|
||||
|
||||
|
||||
# ── Helpers ─────────────────────────────────────────────────────────────
|
||||
|
||||
def position_to_out(row: dict) -> PositionOut:
|
||||
return PositionOut(
|
||||
id=row["id"],
|
||||
symbol=row["symbol"],
|
||||
direction=row["direction"],
|
||||
entry_price=row["entry_price"],
|
||||
exit_price=row["exit_price"],
|
||||
quantity=row["quantity"],
|
||||
exchange=row["exchange"],
|
||||
opened_at=row["opened_at"],
|
||||
closed_at=row["closed_at"],
|
||||
pnl=row["pnl"],
|
||||
metadata=row["metadata"],
|
||||
)
|
||||
|
||||
|
||||
# ── Health ─────────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/api/health")
|
||||
async def health():
|
||||
async with async_session() as session:
|
||||
result = await session.execute(select(func.now()))
|
||||
db_time = result.scalar()
|
||||
return {"status": "ok", "db_time": db_time.isoformat()}
|
||||
|
||||
|
||||
# ── Positions ────────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/api/positions", response_model=list[PositionOut])
|
||||
async def list_positions(
|
||||
open_only: bool = Query(True, description="Only show open positions"),
|
||||
):
|
||||
async with async_session() as session:
|
||||
if open_only:
|
||||
stmt = select(positions).where(positions.c.closed_at.is_(None)).order_by(positions.c.opened_at.desc())
|
||||
else:
|
||||
stmt = select(positions).order_by(positions.c.opened_at.desc())
|
||||
rows = (await session.execute(stmt)).mappings().all()
|
||||
return [position_to_out(r) for r in rows]
|
||||
|
||||
|
||||
@app.post("/api/positions", status_code=201)
|
||||
async def create_position(payload: PositionCreate):
|
||||
async with async_session() as session:
|
||||
values = payload.model_dump()
|
||||
result = await session.execute(positions.insert().values(**values))
|
||||
session.commit()
|
||||
pk = result.inserted_primary_key[0]
|
||||
return {"id": str(pk)}
|
||||
|
||||
|
||||
@app.patch("/api/positions/{position_id}")
|
||||
async def update_position(position_id: UUID, payload: PositionUpdate):
|
||||
async with async_session() as session:
|
||||
row = await session.execute(
|
||||
select(positions).where(positions.c.id == position_id)
|
||||
)
|
||||
row = row.mappings().one_or_none()
|
||||
if not row:
|
||||
raise HTTPException(404, "Position not found")
|
||||
|
||||
updates = payload.model_dump(exclude_unset=True)
|
||||
|
||||
# Auto-compute PnL if closing
|
||||
if "exit_price" in updates:
|
||||
entry = row["entry_price"]
|
||||
qty = row["quantity"]
|
||||
exit_p = updates["exit_price"]
|
||||
direction = row["direction"]
|
||||
if direction == "long":
|
||||
updates["pnl"] = float((exit_p - entry) * qty)
|
||||
else:
|
||||
updates["pnl"] = float((entry - exit_p) * qty)
|
||||
updates["closed_at"] = datetime.now(timezone.utc)
|
||||
|
||||
await session.execute(
|
||||
positions.update().where(positions.c.id == position_id).values(**updates)
|
||||
)
|
||||
session.commit()
|
||||
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
@app.delete("/api/positions/{position_id}")
|
||||
async def close_position(position_id: UUID, exit_price: Decimal = Query(None)):
|
||||
async with async_session() as session:
|
||||
row = await session.execute(
|
||||
select(positions).where(positions.c.id == position_id)
|
||||
)
|
||||
row = row.mappings().one_or_none()
|
||||
if not row:
|
||||
raise HTTPException(404, "Position not found")
|
||||
|
||||
if row["closed_at"]:
|
||||
raise HTTPException(400, "Position already closed")
|
||||
|
||||
exit_p = exit_price or row["entry_price"] # breakeven default
|
||||
entry = row["entry_price"]
|
||||
qty = row["quantity"]
|
||||
direction = row["direction"]
|
||||
|
||||
if direction == "long":
|
||||
pnl = float((exit_p - entry) * qty)
|
||||
else:
|
||||
pnl = float((entry - exit_p) * qty)
|
||||
|
||||
await session.execute(
|
||||
positions.update()
|
||||
.where(positions.c.id == position_id)
|
||||
.values(exit_price=exit_p, closed_at=datetime.now(timezone.utc), pnl=pnl)
|
||||
)
|
||||
session.commit()
|
||||
|
||||
return {"ok": True, "pnl": pnl, "exit_price": float(exit_p)}
|
||||
|
||||
|
||||
# ── PnL ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/api/pnl", response_model=PnLSnapshot)
|
||||
async def get_pnl():
|
||||
async with async_session() as session:
|
||||
now = datetime.now(timezone.utc)
|
||||
today = now.replace(hour=0, minute=0, second=0, microsecond=0)
|
||||
week_start = today - timedelta(days=now.weekday())
|
||||
month_start = today.replace(day=1)
|
||||
|
||||
# Summary for closed trades
|
||||
closed = select(
|
||||
func.coalesce(func.sum(positions.c.pnl), 0).label("total"),
|
||||
func.count(positions.c.id).label("count"),
|
||||
).where(positions.c.closed_at.isnot(None))
|
||||
|
||||
result = (await session.execute(closed)).mappings().one()
|
||||
all_time_pnl = float(result["total"])
|
||||
total_trades = result["count"]
|
||||
|
||||
# PnL by period
|
||||
def period_query(start):
|
||||
return select(
|
||||
func.coalesce(func.sum(positions.c.pnl), 0)
|
||||
).where(
|
||||
and_(
|
||||
positions.c.closed_at.isnot(None),
|
||||
positions.c.closed_at >= start,
|
||||
)
|
||||
)
|
||||
|
||||
today_pnl = float((await session.execute(period_query(today))).scalar())
|
||||
week_pnl = float((await session.execute(period_query(week_start))).scalar())
|
||||
month_pnl = float((await session.execute(period_query(month_start))).scalar())
|
||||
|
||||
# Open count
|
||||
open_count = (await session.execute(
|
||||
select(func.count()).where(positions.c.closed_at.is_(None))
|
||||
)).scalar()
|
||||
|
||||
return PnLSnapshot(
|
||||
today_pnl=Decimal(str(today_pnl)),
|
||||
week_pnl=Decimal(str(week_pnl)),
|
||||
month_pnl=Decimal(str(month_pnl)),
|
||||
all_time_pnl=Decimal(str(all_time_pnl)),
|
||||
total_trades=total_trades,
|
||||
open_positions=open_count,
|
||||
)
|
||||
|
||||
|
||||
@app.get("/api/pnl/history", response_model=list[PositionOut])
|
||||
async def pnl_history(
|
||||
limit: int = Query(50, ge=1, le=500),
|
||||
):
|
||||
async with async_session() as session:
|
||||
stmt = (
|
||||
select(positions)
|
||||
.where(positions.c.closed_at.isnot(None))
|
||||
.order_by(positions.c.closed_at.desc())
|
||||
.limit(limit)
|
||||
)
|
||||
rows = (await session.execute(stmt)).mappings().all()
|
||||
return [position_to_out(r) for r in rows]
|
||||
|
||||
|
||||
# ── Webhook (for scanner scripts) ───────────────────────────────────────
|
||||
|
||||
@app.post("/webhook/trade", status_code=201)
|
||||
async def webhook_trade(payload: WebhookTrade):
|
||||
meta = {"strategy": payload.strategy} if payload.strategy else {}
|
||||
async with async_session() as session:
|
||||
result = await session.execute(positions.insert().values(**{
|
||||
"symbol": payload.symbol,
|
||||
"direction": payload.direction,
|
||||
"entry_price": payload.entry_price,
|
||||
"quantity": payload.quantity,
|
||||
"exchange": payload.exchange,
|
||||
"metadata": meta,
|
||||
}))
|
||||
session.commit()
|
||||
pk = result.inserted_primary_key[0]
|
||||
return {"id": str(pk)}
|
||||
|
||||
|
||||
# ── Frontend ────────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/", response_class=HTMLResponse)
|
||||
async def index():
|
||||
return FileResponse(str(STATIC_DIR / "index.html"))
|
||||
|
||||
|
||||
# ── Startup: run Alembic migrations ──────────────────────────────────────
|
||||
|
||||
@app.on_event("startup")
|
||||
async def startup():
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
from pathlib import Path
|
||||
|
||||
alembic_cfg = Config(
|
||||
str(Path(__file__).parent.parent / "alembic.ini")
|
||||
)
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=8000)
|
||||
|
|
@ -1,21 +0,0 @@
|
|||
from sqlalchemy import Column, String, Numeric, Enum, DateTime, JSON, func, Table
|
||||
from sqlalchemy.dialects.postgresql import UUID
|
||||
import uuid
|
||||
|
||||
from database import metadata
|
||||
|
||||
positions = Table(
|
||||
"positions",
|
||||
metadata,
|
||||
Column("id", UUID(as_uuid=True), primary_key=True, default=uuid.uuid4),
|
||||
Column("symbol", String(32), nullable=False, index=True),
|
||||
Column("direction", Enum("long", "short", name="position_direction"), nullable=False),
|
||||
Column("entry_price", Numeric(precision=16, scale=8), nullable=False),
|
||||
Column("exit_price", Numeric(precision=16, scale=8)),
|
||||
Column("quantity", Numeric(precision=16, scale=8), nullable=False),
|
||||
Column("exchange", String(32), nullable=False, index=True),
|
||||
Column("opened_at", DateTime(timezone=True), server_default=func.now(), nullable=False),
|
||||
Column("closed_at", DateTime(timezone=True)),
|
||||
Column("pnl", Numeric(precision=16, scale=2)),
|
||||
Column("metadata", JSON),
|
||||
)
|
||||
|
|
@ -1,7 +0,0 @@
|
|||
fastapi==0.115.0
|
||||
uvicorn[standard]==0.32.0
|
||||
sqlalchemy[asyncio]==2.0.35
|
||||
asyncpg==0.30.0
|
||||
alembic==1.14.0
|
||||
pydantic==2.9.2
|
||||
python-dotenv==1.0.1
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from decimal import Decimal
|
||||
from enum import Enum
|
||||
from typing import Optional
|
||||
from uuid import UUID
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class Direction(str, Enum):
|
||||
long = "long"
|
||||
short = "short"
|
||||
|
||||
|
||||
# ─── Request schemas ──────────────────────────────────────────────
|
||||
|
||||
class PositionCreate(BaseModel):
|
||||
symbol: str
|
||||
direction: Direction
|
||||
entry_price: Decimal
|
||||
quantity: Decimal
|
||||
exchange: str
|
||||
metadata: Optional[dict] = None
|
||||
|
||||
|
||||
class PositionUpdate(BaseModel):
|
||||
entry_price: Optional[Decimal] = None
|
||||
exit_price: Optional[Decimal] = None
|
||||
quantity: Optional[Decimal] = None
|
||||
metadata: Optional[dict] = None
|
||||
|
||||
|
||||
class WebhookTrade(BaseModel):
|
||||
"""Payload from automated scanner scripts."""
|
||||
symbol: str
|
||||
direction: Direction
|
||||
entry_price: Decimal
|
||||
quantity: Decimal
|
||||
exchange: str
|
||||
strategy: Optional[str] = None
|
||||
|
||||
|
||||
# ─── Response schemas ─────────────────────────────────────────────
|
||||
|
||||
class PositionOut(BaseModel):
|
||||
id: UUID
|
||||
symbol: str
|
||||
direction: Direction
|
||||
entry_price: Decimal
|
||||
exit_price: Optional[Decimal]
|
||||
quantity: Decimal
|
||||
exchange: str
|
||||
opened_at: datetime
|
||||
closed_at: Optional[datetime]
|
||||
pnl: Optional[Decimal]
|
||||
metadata: Optional[dict]
|
||||
|
||||
model_config = {"from_attributes": True}
|
||||
|
||||
|
||||
class PnLSnapshot(BaseModel):
|
||||
today_pnl: Decimal
|
||||
week_pnl: Decimal
|
||||
month_pnl: Decimal
|
||||
all_time_pnl: Decimal
|
||||
total_trades: int
|
||||
open_positions: int
|
||||
|
||||
|
|
@ -1,158 +0,0 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Trade Dashboard</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #0d1117; --card: #161b22; --border: #30363d;
|
||||
--text: #c9d1d9; --muted: #8b949e; --green: #3fb950;
|
||||
--red: #f85149; --blue: #58a6ff; --accent: #1f6feb;
|
||||
}
|
||||
* { box-sizing: border-box; margin: 0; padding: 0; }
|
||||
body { background: var(--bg); color: var(--text); font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; padding: 1rem; }
|
||||
h1 { font-size: 1.5rem; margin-bottom: 1rem; color: var(--blue); }
|
||||
.grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(200px, 1fr)); gap: 1rem; margin-bottom: 1.5rem; }
|
||||
.card { background: var(--card); border: 1px solid var(--border); border-radius: 8px; padding: 1rem; }
|
||||
.card .label { font-size: 0.75rem; color: var(--muted); text-transform: uppercase; }
|
||||
.card .value { font-size: 1.5rem; font-weight: 600; margin-top: 0.25rem; }
|
||||
.card .value.positive { color: var(--green); }
|
||||
.card .value.negative { color: var(--red); }
|
||||
table { width: 100%; border-collapse: collapse; margin-top: 1rem; }
|
||||
th, td { text-align: left; padding: 0.5rem; border-bottom: 1px solid var(--border); font-size: 0.85rem; }
|
||||
th { color: var(--muted); font-weight: 500; }
|
||||
.badge { display: inline-block; padding: 2px 8px; border-radius: 12px; font-size: 0.75rem; font-weight: 600; }
|
||||
.badge.long { background: #12261e; color: var(--green); }
|
||||
.badge.short { background: #2a1215; color: var(--red); }
|
||||
.btn { background: var(--accent); color: #fff; border: none; padding: 0.5rem 1rem; border-radius: 6px; cursor: pointer; font-size: 0.85rem; }
|
||||
.btn:hover { opacity: 0.9; }
|
||||
.btn.close { background: var(--red); font-size: 0.75rem; padding: 0.25rem 0.5rem; }
|
||||
.section { margin-top: 2rem; }
|
||||
.section h2 { font-size: 1.1rem; margin-bottom: 0.75rem; color: var(--blue); }
|
||||
form { display: flex; gap: 0.5rem; flex-wrap: wrap; margin-bottom: 1rem; }
|
||||
input, select { background: var(--card); border: 1px solid var(--border); color: var(--text); padding: 0.5rem; border-radius: 6px; font-size: 0.85rem; }
|
||||
input:focus, select:focus { outline: 1px solid var(--accent); }
|
||||
.tab-bar { display: flex; gap: 0.5rem; margin-bottom: 1rem; }
|
||||
.tab { padding: 0.5rem 1rem; border-radius: 6px; cursor: pointer; font-size: 0.85rem; background: var(--card); border: 1px solid var(--border); }
|
||||
.tab.active { border-color: var(--accent); color: var(--blue); }
|
||||
#loading { color: var(--muted); font-size: 0.85rem; }
|
||||
@media (max-width: 600px) { .grid { grid-template-columns: 1fr 1fr; } }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>📊 Trade Dashboard</h1>
|
||||
|
||||
<!-- PnL Summary Cards -->
|
||||
<div class="grid" id="pnl-cards">
|
||||
<div class="card"><div class="label">Today PnL</div><div class="value" id="today-pnl">—</div></div>
|
||||
<div class="card"><div class="label">Week PnL</div><div class="value" id="week-pnl">—</div></div>
|
||||
<div class="card"><div class="label">Month PnL</div><div class="value" id="month-pnl">—</div></div>
|
||||
<div class="card"><div class="label">All Time</div><div class="value" id="all-pnl">—</div></div>
|
||||
</div>
|
||||
|
||||
<!-- Open Positions -->
|
||||
<div class="section">
|
||||
<h2>Open Positions <span id="open-count" class="badge long">0</span></h2>
|
||||
<table>
|
||||
<thead><tr><th>Symbol</th><th>Dir</th><th>Entry</th><th>Qty</th><th>Exchange</th><th>Opened</th><th>Action</th></tr></thead>
|
||||
<tbody id="positions-body"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- New Position Form -->
|
||||
<div class="section">
|
||||
<h2>Open New Position</h2>
|
||||
<form id="open-form">
|
||||
<input type="text" id="sym" placeholder="Symbol (e.g. SOL)" required>
|
||||
<select id="dir"><option value="long">Long</option><option value="short">Short</option></select>
|
||||
<input type="number" step="any" id="entry" placeholder="Entry Price" required>
|
||||
<input type="number" step="any" id="qty" placeholder="Quantity" required>
|
||||
<input type="text" id="exch" placeholder="Exchange (OKX, Bybit…)" required>
|
||||
<button type="submit" class="btn">Open Position</button>
|
||||
</form>
|
||||
</div>
|
||||
|
||||
<!-- Trade History -->
|
||||
<div class="section">
|
||||
<h2>Trade History</h2>
|
||||
<table>
|
||||
<thead><tr><th>Symbol</th><th>Dir</th><th>Entry</th><th>Exit</th><th>Qty</th><th>PnL</th><th>Exchange</th><th>Closed</th></tr></thead>
|
||||
<tbody id="history-body"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const fmt = (n) => {
|
||||
const v = parseFloat(n);
|
||||
const cls = v >= 0 ? 'positive' : 'negative';
|
||||
return `<span class="${cls}">${v >= 0 ? '+' : ''}${v.toFixed(2)}</span>`;
|
||||
};
|
||||
|
||||
async function loadPnL() {
|
||||
const r = await fetch('/api/pnl').then(x => x.json());
|
||||
document.getElementById('today-pnl').innerHTML = fmt(r.today_pnl);
|
||||
document.getElementById('week-pnl').innerHTML = fmt(r.week_pnl);
|
||||
document.getElementById('month-pnl').innerHTML = fmt(r.month_pnl);
|
||||
document.getElementById('all-pnl').innerHTML = fmt(r.all_time_pnl);
|
||||
}
|
||||
|
||||
async function loadPositions() {
|
||||
const positions = await fetch('/api/positions?open_only=true').then(x => x.json());
|
||||
document.getElementById('open-count').textContent = positions.length;
|
||||
const body = document.getElementById('positions-body');
|
||||
body.innerHTML = positions.map(p => `
|
||||
<tr>
|
||||
<td>${p.symbol}</td>
|
||||
<td><span class="badge ${p.direction}">${p.direction.toUpperCase()}</span></td>
|
||||
<td>${p.entry_price}</td>
|
||||
<td>${p.quantity}</td>
|
||||
<td>${p.exchange}</td>
|
||||
<td>${new Date(p.opened_at).toLocaleString()}</td>
|
||||
<td><button class="btn close" onclick="closePos('${p.id}')">Close</button></td>
|
||||
</tr>`).join('');
|
||||
}
|
||||
|
||||
async function loadHistory() {
|
||||
const history = await fetch('/api/pnl/history?limit=100').then(x => x.json());
|
||||
const body = document.getElementById('history-body');
|
||||
body.innerHTML = history.map(p => `
|
||||
<tr>
|
||||
<td>${p.symbol}</td>
|
||||
<td><span class="badge ${p.direction}">${p.direction.toUpperCase()}</span></td>
|
||||
<td>${p.entry_price}</td>
|
||||
<td>${p.exit_price ?? '—'}</td>
|
||||
<td>${p.quantity}</td>
|
||||
<td>${p.pnl != null ? (parseFloat(p.pnl) >= 0 ? '+' : '') + parseFloat(p.pnl).toFixed(2) : '—'}</td>
|
||||
<td>${p.exchange}</td>
|
||||
<td>${p.closed_at ? new Date(p.closed_at).toLocaleString() : '—'}</td>
|
||||
</tr>`).join('');
|
||||
}
|
||||
|
||||
async function closePos(id) {
|
||||
const price = prompt('Exit price:');
|
||||
if (!price) return;
|
||||
await fetch(`/api/positions/${id}?exit_price=${price}`, { method: 'DELETE' });
|
||||
refresh();
|
||||
}
|
||||
|
||||
document.getElementById('open-form').onsubmit = async (e) => {
|
||||
e.preventDefault();
|
||||
const body = {
|
||||
symbol: document.getElementById('sym').value,
|
||||
direction: document.getElementById('dir').value,
|
||||
entry_price: parseFloat(document.getElementById('entry').value),
|
||||
quantity: parseFloat(document.getElementById('qty').value),
|
||||
exchange: document.getElementById('exch').value,
|
||||
};
|
||||
await fetch('/api/positions', { method: 'POST', headers: {'Content-Type': 'application/json'}, body: JSON.stringify(body) });
|
||||
e.target.reset();
|
||||
refresh();
|
||||
};
|
||||
|
||||
async function refresh() { await loadPnL(); await loadPositions(); await loadHistory(); }
|
||||
refresh();
|
||||
setInterval(refresh, 30000); // auto-refresh every 30s
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
Loading…
Add table
Reference in a new issue