diff --git a/.github/workflows/trade-dashboard.yml b/.github/workflows/trade-dashboard.yml deleted file mode 100644 index a7f8371..0000000 --- a/.github/workflows/trade-dashboard.yml +++ /dev/null @@ -1,52 +0,0 @@ -name: Build and Push Trade Dashboard - -on: - push: - branches: [master] - paths: - - 'trade-dashboard/**' - workflow_dispatch: - -env: - REGISTRY: ghcr.io - IMAGE_NAME: ${{ github.repository_owner }}/trade-dashboard - -permissions: - contents: read - packages: write - -jobs: - build: - runs-on: ubuntu-latest - steps: - - name: Checkout - uses: actions/checkout@v4 - - - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v3 - - - name: Login to GHCR - uses: docker/login-action@v3 - with: - registry: ${{ env.REGISTRY }} - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - - - name: Extract metadata - id: meta - uses: docker/metadata-action@v5 - with: - images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} - tags: | - type=sha,prefix= - type=raw,value=latest,enable={{is_default_branch}} - - - name: Build and push - uses: docker/build-push-action@v5 - with: - context: trade-dashboard/ - push: true - tags: ${{ steps.meta.outputs.tags }} - labels: ${{ steps.meta.outputs.labels }} - cache-from: type=gha - cache-to: type=gha,mode=max diff --git a/analyses/telegram-webhook-container-analysis.md b/analyses/telegram-webhook-container-analysis.md new file mode 100644 index 0000000..d2ef6c7 --- /dev/null +++ b/analyses/telegram-webhook-container-analysis.md @@ -0,0 +1,303 @@ +# Telegram Webhook Failure Analysis - Deep Dive (Container-Level) + +**Date:** 2026-05-10 +**Repo:** https://github.com/sirius0xdev/gcloud-lab +**Cluster:** devops-lab (GKE), Namespace: customer1 +**Webhook URL:** https://ws.siriusdevops.com/telegram/webhook/default + +--- + +## Status Update + +**TLS certificates ARE provisioned** in GCP admin console. The Gateway certmap annotation is working. + +The real issues are in the **container configuration** and **missing probes**. + +--- + +## Root Cause #1: NO Liveness/Readiness Probes (CONFIRMED CRITICAL) + +**The hermes-agent deployment has ZERO probes defined.** + +This means: +- Kubernetes marks pods "Ready" immediately after container start +- The Gateway routes webhook traffic before the process has bound to port 9118 +- During restarts, there is no graceful drain +- **A crashed or hung pod stays in the endpoint list forever** + +### The /health endpoint problem + +The Telegram webhook server runs on **port 9118** via python-telegram-bot's `start_webhook()`. This internally starts an aiohttp server that **ONLY registers the webhook path** (`/telegram/webhook/default`). It does NOT expose a `/health` endpoint. + +So a probe like this would FAIL: +```yaml +# THIS WON'T WORK - port 9118 has no /health +readinessProbe: + httpGet: + path: /health + port: 9118 +``` + +### What DOES have a health endpoint? + +The **generic webhook adapter** on **port 8644** IS a Hermes-managed server. From the source code, it exposes `/health`. This IS enabled via `WEBHOOK_ENABLED: "true"`. + +### The Fix + +Add probes targeting port 8644 (the generic webhook server that IS healthy when the gateway is running): + +```yaml +readinessProbe: + httpGet: + path: /health + port: 8644 + initialDelaySeconds: 10 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 +livenessProbe: + httpGet: + path: /health + port: 8644 + initialDelaySeconds: 30 + periodSeconds: 30 + timeoutSeconds: 5 + failureThreshold: 3 +``` + +Alternatively, enable the API server on port 8642 and probe there (it also has `/health`). + +--- + +## Root Cause #2: NO Resource Limits (HIGH) + +The **new deployment dropped all resource limits** that existed in the old deployment. + +**Old deployment (`deployment.yaml`):** +```yaml +resources: + requests: + memory: 2Gi + cpu: "1" + limits: + memory: 3Gi + cpu: "2" +``` + +**New deployment (`new-deployment.yaml`):** +```yaml +# NO resources block for the hermes-agent container +``` + +Only the `hermes-webui` sidecar has limits (500Mi-1Gi memory, 100m-500m CPU). + +### Impact +- The hermes-agent container can consume unbounded memory +- With `agent.max_turns: 90` and `gateway_timeout: 1800` (30 min), a single agent run can eat massive memory +- Pod may be OOMKilled by the node, but K8s won't restart it gracefully without probes +- **This is a likely cause of intermittent failures** + +### Fix +Add resource limits back: +```yaml +resources: + requests: + memory: 2Gi + cpu: "1" + limits: + memory: 4Gi + cpu: "2" +``` + +--- + +## Root Cause #3: Path Handling Concern (MEDIUM) + +**Source code** (`/opt/hermes/gateway/platforms/telegram.py:1213`): +```python +webhook_path = urlparse(webhook_url).path or "/telegram" +``` + +For `TELEGRAM_WEBHOOK_URL=https://ws.siriusdevops.com/telegram/webhook/default`: +- `webhook_path` = `/telegram/webhook/default` +- PTB's aiohttp server registers a handler at exactly `/telegram/webhook/default` on port 9118 + +**HTTPRoute** (`hermes-webhook.yaml`): +```yaml +rules: +- matches: + - path: + type: PathPrefix + value: /telegram/webhook + backendRefs: + - name: http-tele-webhook + port: 9118 +``` + +**The question:** Does the GKE Gateway strip the `/telegram/webhook` prefix before forwarding? + +- If it does NOT strip: Pod receives `/telegram/webhook/default` -> **OK** +- If it DOES strip to `/default`: Pod receives `/default` -> **404** +- If it strips to `/`: Pod receives `/` -> **404** + +Most GatewayAPI implementations pass the full original path by default, but some ingress controllers strip the matched prefix. **Verify this on the running cluster.** + +--- + +## Root Cause #4: API Server Not Enabled (MEDIUM) + +The old deployment had: +```yaml +- name: API_SERVER_ENABLED + value: "true" +- name: API_SERVER_HOST + value: "0.0.0.0" +- name: API_SERVER_PORT + value: "8642" +- name: API_SERVER_KEY + valueFrom: {secretKeyRef: ...} +- name: API_SERVER_MODEL_NAME + value: "hermes-agent" +``` + +**These are ALL absent from `new-deployment.yaml`.** Port 8642 is declared as a containerPort but nothing listens on it because `API_SERVER_ENABLED` is not set. + +The API server exposes `/health` and `/health/detailed` endpoints. Without it, you lose a convenient health check and the API server interface. + +--- + +## Other Findings + +### 5. TELEGRAM_WEBHOOK_ENABLED is Ignored (INFO) + +The code only checks if `TELEGRAM_WEBHOOK_URL` is set (non-empty). It does NOT read `TELEGRAM_WEBHOOK_ENABLED`. The env var in the deployment is a dead config — set but ignored. + +### 6. vLLM Service Name Mismatch (INFO) + +```yaml +OPENAI_BASE_URL: "http://openclaw-brain-service.customer1.svc.cluster.local:8000/v1" +``` + +But the active vLLM deployment (`rtx6000-vllm.yaml`) creates a service named `rtx6000-brain-service`. If Hermes ever switches from xAI to the openai provider, local vLLM is unreachable. + +### 7. SOPS Secrets (VERIFY) + +Both `hermes-secret.yaml` and `tele-webhook.yaml` are SOPS-encrypted. Verify they are decrypted in the cluster: +```bash +kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d +kubectl get secret telegram-webhook -n customer1 -o jsonpath='{.data.TELEGRAM_WEBHOOK_SECRET}' | base64 -d +``` + +### 8. Container Env Vars Summary + +| Variable | Value | Notes | +|----------|-------|-------| +| TELEGRAM_WEBHOOK_URL | https://ws.siriusdevops.com/telegram/webhook/default | OK | +| TELEGRAM_WEBHOOK_PORT | 9118 | OK | +| TELEGRAM_WEBHOOK_SECRET | From SOPS secret | Verify decrypted | +| TELEGRAM_WEBHOOK_ENABLED | "true" | **Ignored by code** | +| TELEGRAM_BOT_TOKEN | From SOPS secret | Verify decrypted | +| TELEGRAM_ALLOWED_USERS | 7528130947 | OK | +| WEBHOOK_ENABLED | "true" | OK (enables port 8644 /health) | +| WEBHOOK_PORT | 8644 | OK | +| API_SERVER_ENABLED | **NOT SET** | Port 8642 has no listener | +| HERMES_MODEL_PROVIDER | xai | OK (uses Grok) | +| HERMES_MODEL | grok-4.20-0309-reasoning | OK | + +--- + +## Recommended Fix (Priority Order) + +### P0 - Add Probes (will detect and restart hung/crashed pods) + +Add to `new-deployment.yaml` under the hermes-agent container spec: + +```yaml +readinessProbe: + httpGet: + path: /health + port: 8644 + initialDelaySeconds: 10 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 +livenessProbe: + httpGet: + path: /health + port: 8644 + initialDelaySeconds: 30 + periodSeconds: 30 + timeoutSeconds: 5 + failureThreshold: 3 +``` + +### P1 - Add Resource Limits (prevents OOM kills) + +```yaml +resources: + requests: + memory: 2Gi + cpu: "1" + limits: + memory: 4Gi + cpu: "2" +``` + +### P2 - Verify Path Handling + +Test if the Gateway passes the full path: +```bash +# From inside the pod, check what the webhook server receives +kubectl exec -n customer1 deploy/hermes-agent -- curl -s http://localhost:9118/telegram/webhook/default -X POST -H "Content-Type: application/json" -d '{}' +``` + +### P3 - Enable API Server (optional, gives /health on 8642) + +Add back the API_SERVER_ENABLED env var if you want the API server health endpoint. + +--- + +## Diagnostic Commands + +Run these on the cluster RIGHT NOW to confirm the current state: + +```bash +# 1. Check pod status and restart count +kubectl get pods -n customer1 -l app=hermes-agent -o wide + +# 2. Check events for OOMKilled or probe failures +kubectl describe pod -n customer1 -l app=hermes-agent | grep -A5 -i 'oom\|probe\|restart' + +# 3. Check if the webhook port is actually listening +kubectl exec -n customer1 deploy/hermes-agent -- ss -tlnp | grep 9118 + +# 4. Check if port 8644 health endpoint works +kubectl exec -n customer1 deploy/hermes-agent -- curl -s http://localhost:8644/health + +# 5. Check logs for webhook startup messages +kubectl logs -n customer1 deploy/hermes-agent --tail=50 | grep -i 'webhook\|listening\|9118' + +# 6. Verify SOPS secrets are decrypted +kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d && echo +kubectl get secret telegram-webhook -n customer1 -o jsonpath='{.data.TELEGRAM_WEBHOOK_SECRET}' | base64 -d && echo + +# 7. Check if the webhook is registered with Telegram (using the pod's bot token) +BOT_TOKEN=$(kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d) +curl -s "https://api.telegram.org/bot${BOT_TOKEN}/getWebhookInfo" | python3 -m json.tool + +# 8. Test the full path through the Gateway +curl -v https://ws.siriusdevops.com/telegram/webhook/default -X POST -H "Content-Type: application/json" -d '{}' 2>&1 + +# 9. Check container memory usage +kubectl top pod -n customer1 -l app=hermes-agent 2>/dev/null || echo "metrics-server not available" + +# 10. Check resource limits on the container +kubectl get pod -n customer1 -l app=hermes-agent -o jsonpath='{.items[0].spec.containers[0].resources}' +``` + +--- + +## Files to Modify + +1. `apps/base/customer1/hermes-agent/new-deployment.yaml` - Add probes + resource limits +2. Optionally: Re-enable API server env vars in `new-deployment.yaml` diff --git a/analyses/telegram-webhook-failure-analysis.md b/analyses/telegram-webhook-failure-analysis.md new file mode 100644 index 0000000..cdcac55 --- /dev/null +++ b/analyses/telegram-webhook-failure-analysis.md @@ -0,0 +1,345 @@ +# Telegram Webhook Failure Analysis - Hermes Agent on GKE + +**Date:** 2026-05-10 +**Repo:** https://github.com/sirius0xdev/gcloud-lab +**Cluster:** devops-lab (GKE) +**Namespace:** customer1 +**Webhook URL:** https://ws.siriusdevops.com/telegram/webhook/default + +--- + +## Executive Summary + +The Telegram webhook is failing because **there is no valid TLS certificate for `ws.siriusdevops.com`** at the Gateway layer. Telegram strictly requires HTTPS with a publicly-trusted certificate for webhook delivery. The Gateway listener declares `protocol: HTTPS` but has no `tls.certificateRefs` and relies on a GKE `CertMap` annotation referencing `gateway-cert-map` — a resource that **does not exist** in the repository. Cert-manager is configured but disconnected from the GatewayAPI setup (HTTP-01 solver points to a non-existent Traefik ingress class). + +**TL;DR:** Telegram tries to POST to `https://ws.siriusdevops.com/...`, but the Gateway has no certificate to present during the TLS handshake. Connection fails before it ever reaches the hermes-agent pod. + +--- + +## Root Cause #1: Missing TLS Certificate (CRITICAL) + +### What's happening + +**File:** `infrastructure/gatewayapi/apigateway.yaml` + +```yaml +listeners: +- name: https + protocol: HTTPS + port: 443 + allowedRoutes: + namespaces: + from: All +``` + +The listener says HTTPS but has **no `tls:` block**. It relies entirely on this annotation: + +```yaml +annotations: + networking.gke.io/certmap: gateway-cert-map +``` + +### The problem + +- **No `CertMap` or `CertMapEntry` resource** exists anywhere in the repository for `gateway-cert-map` +- Without it, GKE has no managed certificate to attach to the Gateway +- Telegram's webhook delivery gets a TLS handshake failure or no certificate + +### Cert-manager is also broken + +**File:** `infrastructure/controllers/base/certmanager/clusterissuer.yaml` + +```yaml +solvers: + - http01: + ingress: + class: traefik +``` + +- HTTP-01 solver references `class: traefik`, but **no Traefik ingress controller exists** in the cluster +- **No `Certificate` CRs** exist for `ws.siriusdevops.com` or any other domain +- Cert-manager is completely disconnected from the GatewayAPI setup + +### How to fix (choose ONE approach) + +**Option A: GKE Managed Certificates (recommended for GatewayAPI)** + +Create a `ManagedCertificate` + `BackendConfig` or `CertMap`/`CertMapEntry`: + +```yaml +apiVersion: networking.gke.io/v1 +kind: ManagedCertificate +metadata: + name: hermes-webhook-cert + namespace: customer1 +spec: + domains: + - ws.siriusdevops.com + - brain.siriusdevops.com + - paaas.siriusdevops.com +``` + +Then add `tls.certificateRefs` to the Gateway listener: + +```yaml +listeners: +- name: https + protocol: HTTPS + port: 443 + tls: + certificateRefs: + - name: hermes-webhook-cert + group: networking.gke.io +``` + +**Option B: Fix Cert-manager with DNS-01** + +Switch ClusterIssuer from HTTP-01/Traefik to DNS-01 (e.g., Cloudflare, GCP DNS, or Route53), then create `Certificate` resources for each domain. + +--- + +## Root Cause #2: No Readiness/Liveness Probes (HIGH) + +**File:** `apps/base/customer1/hermes-agent/new-deployment.yaml` + +No readiness or liveness probes are defined on any hermes-agent deployment. + +### Impact +- Pods are marked "Ready" immediately after container start +- Gateway routes traffic to the webhook port (9118) before the process has bound to it +- During restarts, traffic hits pods that haven't initialized + +### Fix +Add probes to the deployment: + +```yaml +readinessProbe: + httpGet: + path: /health + port: 9118 + initialDelaySeconds: 5 + periodSeconds: 10 +livenessProbe: + httpGet: + path: /health + port: 9118 + initialDelaySeconds: 15 + periodSeconds: 30 +``` + +--- + +## Root Cause #3: SOPS Encrypted Secrets (HIGH) + +**Files:** +- `apps/base/customer1/hermes-agent/hermes-secret.yaml` (SOPS encrypted) +- `apps/base/customer1/hermes-agent/tele-webhook.yaml` (SOPS encrypted) + +These contain `TELEGRAM_BOT_TOKEN` and `TELEGRAM_WEBHOOK_SECRET`. + +### Risk +- If the deployment pipeline (Flux/Kustomize controller) is **not decrypting SOPS secrets**, pods receive literal `ENC[...]` strings +- The bot token would be invalid, so even if TLS worked, Telegram auth would fail +- The webhook secret would not match, causing Telegram to reject payloads + +### Verify +Run `kubectl get secret hermes-secrets -n customer1 -o yaml` and check if values are base64-encoded real tokens or `ENC[...]` strings. + +--- + +## Secondary Issues + +### 3a. Service Name Mismatch (vLLM Integration) + +**File:** `apps/base/customer1/hermes-agent/new-deployment.yaml` + +```yaml +OPENAI_BASE_URL: "http://openclaw-brain-service.customer1.svc.cluster.local:8000/v1" +``` + +But the active vLLM deployment (`rtx6000-vllm.yaml`) creates a service named **`rtx6000-brain-service`**. + +- `OPENAI_BASE_URL` points to `openclaw-brain-service` which may not exist +- If Hermes ever switches to the `openai` provider (instead of `xai`), local vLLM is unreachable +- The model provider defaults to `HERMES_MODEL_PROVIDER: xai` (Grok external API) + +### 3b. Duplicate HTTPRoute Deployment + +The webhook HTTPRoute (`hermes-webhook.yaml`) is included in **two** kustomization trees: + +1. `infrastructure/gatewayapi/gateway-routes/kustomization.yaml` -> deployed via Flux +2. `apps/base/customer1/hermes-agent/kustomization.yaml` -> deployed via Flux + +Same resource (`http-telegram-webhook` in `customer1`) from two sources. This may cause Flux reconciliation conflicts. + +### 3c. Empty HF_TOKEN in vLLM Deployments + +All vLLM deployments have: + +```yaml +- name: HF_TOKEN + value: "" +``` + +If the model `edp1096/Huihui-Qwen3.6-27B-abliterated-FP8` is a gated model on HuggingFace, it will fail to download. + +### 3d. KEDA Scale-to-Zero + +**File:** `infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml` + +```yaml +minReplicaCount: 0 +maxReplicaCount: 1 +``` + +- vLLM scales to **zero** when idle +- First request after cold start incurs full model load time (30-60 seconds) +- For real-time Telegram responses, this causes visible latency + +### 3e. PVC Name Collision + +Both `rtx6000-vllm.yaml` and `a100-vllm.yaml` define a PVC named `vllm-model-qwen3.6-27b-uncensored` in namespace `customer1`. If both are ever active simultaneously, they conflict. + +--- + +## Architecture Overview + +``` + Internet + | + v + [GKE External LB] + | + Gateway: external-http-gateway + (port 443/HTTPS, NO TLS cert!) + | + +-----------+-----------+ + | | | + ws.siriusdevops.com brain.siriusdevops.com paaas.siriusdevops.com + | | | + v v v + /telegram/webhook / / + | | | + v v v + http-tele-webhook rtx6000- paaas-landing + :9118 brain- :8080 + service:8000 + | + v + hermes-agent pod + (ports: 8642, 8644, 9118) + | + v + Model Provider: xai (Grok via external API) + Fallback: OPENAI_BASE_URL -> openclaw-brain-service (MISMATCHED) +``` + +--- + +## vLLM Server Status + +| Server | GPU | Model | Quantization | Status | +|--------|-----|-------|-------------|--------| +| RTX 6000 | 1x Pro 6000 (96GB) | edp1096/Huihui-Qwen3.6-27B-abliterated-FP8 | FP8 | **ACTIVE** | +| A100 | 1x A100 (80GB) | Youssofal/Qwen3.6-27B-Abliterated-Heretic-Uncensored-BF16 | BF16 | Commented out | +| L4 | 1x L4 (24GB) | p-e-w/Qwen3-8B-heretic | auto | Commented out | +| Gemma | 1x A100 (80GB) | coder3101/Qwen3.5-27B-heretic | BF16 | Not in kustomization | + +**Note:** The architecture plan (`plans/AI_ARCHITECTURE.md`) describes a dual-tier L4 dispatcher + A100 deep thinker setup, but the active deployment only has RTX 6000. + +--- + +## Action Plan (Priority Order) + +### P0 - Fix TLS (will unblock Telegram webhooks) + +1. **Create a `CertMap`/`CertMapEntry`** or **`ManagedCertificate`** resource for `ws.siriusdevops.com` +2. **Add `tls.certificateRefs`** to the Gateway listener in `apigateway.yaml` +3. Verify with: `curl -vI https://ws.siriusdevops.com/telegram/webhook/default` +4. If cert is valid, Telegram should start delivering webhooks + +### P1 - Verify Secrets + +5. Check if SOPS secrets are actually decrypted in the cluster +6. Run: `kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d` + +### P2 - Add Probes + +7. Add readiness/liveness probes to hermes-agent deployment +8. Redeploy + +### P3 - Clean Up GatewayAPI + +9. Remove duplicate `hermes-webhook.yaml` reference from one kustomization +10. Fix or remove Traefik-referencing ClusterIssuers + +### P4 - Fix vLLM Integration + +11. Fix `OPENAI_BASE_URL` service name or update kustomization to create `openclaw-brain-service` +12. Set real `HF_TOKEN` in vLLM deployments +13. Consider setting `minReplicaCount: 1` in KEDA for consistent response times + +--- + +## Files Referenced + +### GatewayAPI & TLS +- `infrastructure/gatewayapi/apigateway.yaml` - Gateway definition (missing TLS) +- `infrastructure/gatewayapi/gateway-routes/hermes-webhook.yaml` - Webhook HTTPRoute +- `infrastructure/gatewayapi/gateway-routes/route.yaml` - General route +- `infrastructure/controllers/base/certmanager/clusterissuer.yaml` - Cert-manager (Traefik mismatch) + +### Hermes Agent Deployment +- `apps/base/customer1/hermes-agent/new-deployment.yaml` - Active deployment + webhook service +- `apps/base/customer1/hermes-agent/deployment.yaml` - Old deployment (no webhook config) +- `apps/base/customer1/hermes-agent/configmap.yaml` - Hermes config +- `apps/base/customer1/hermes-agent/hermes-secret.yaml` - SOPS encrypted secrets +- `apps/base/customer1/hermes-agent/tele-webhook.yaml` - SOPS encrypted webhook secret +- `apps/base/customer1/hermes-agent/kustomization.yaml` - Kustomize composition + +### vLLM Infrastructure +- `infrastructure/gpus/base/vllm-servers/rtx6000-vllm.yaml` - Active vLLM server +- `infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml` - KEDA scaling +- `infrastructure/gpus/base/keda-gpu-scaling/vllm-route.yaml` - vLLM external route + +### Architecture Plans +- `plans/AI_ARCHITECTURE.md` - AI architecture plan +- `plans/models-to-try.md` - Models being considered + +--- + +## Quick Diagnostic Commands + +Run these on the cluster to confirm findings: + +```bash +# 1. Check if TLS cert exists for the webhook domain +curl -vI https://ws.siriusdevops.com/telegram/webhook/default 2>&1 | grep -E 'SSL|certificate|subject|issuer' + +# 2. Check Gateway status +kubectl get gateway external-http-gateway -n customer1 -o yaml + +# 3. Check HTTPRoute status (attached/programmed) +kubectl get httproute http-telegram-webhook -n customer1 -o yaml + +# 4. Verify secrets are decrypted +kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d && echo + +# 5. Check if pods are actually ready +kubectl get pods -n customer1 -l app=hermes-agent -o wide + +# 6. Check CertMap exists +kubectl get certmap gateway-cert-map -A 2>/dev/null || echo "CertMap NOT FOUND" + +# 7. Check managed certificates +kubectl get managedcertificate -A 2>/dev/null || echo "No ManagedCertificates" + +# 8. Check cert-manager certificates +kubectl get certificate -A 2>/dev/null || echo "No Certificates" + +# 9. Check vLLM pod status +kubectl get pods -n customer1 -l app=rtx6000-brain-vllm -o wide + +# 10. Test internal webhook endpoint +kubectl exec -n customer1 deploy/hermes-agent -- curl -s http://localhost:9118/health || echo "Health check failed" +``` diff --git a/rays-new-deployment.yaml b/rays-new-deployment.yaml deleted file mode 100644 index da38535..0000000 --- a/rays-new-deployment.yaml +++ /dev/null @@ -1,217 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: rays-hermes-agent - namespace: customer1 -spec: - replicas: 1 - selector: - matchLabels: - app: rays-hermes-agent - template: - metadata: - labels: - app: rays-hermes-agent - spec: - # 1. Pod-level security context to ensure volumes inherit the right group - shareProcessNamespace: true - securityContext: - fsGroup: 1000 - - # 2. Define our shared bridge volumes - volumes: - - name: hermes-home - persistentVolumeClaim: - claimName: rays-hermes-agent-pvc - - name: hermes-agent-src - emptyDir: {} - - name: hermes-workspace - emptyDir: {} - - name: hermes-webui-app - emptyDir: {} - - name: hermes-configmap - configMap: - name: hermes-config - - initContainers: - # 3. K8s workaround: Copy the agent source code into the shared emptyDir - - name: copy-agent-source - image: nousresearch/hermes-agent:latest - command: - - "sh" - - "-c" - - | - cp -a /opt/hermes/. /shared-src/ && chown -R 1024:1000 /shared-src /shared-home - - if [ -f /tmp/hermes/config.yaml ]; then - cp -f /tmp/hermes/config.yaml /shared-home/config.yaml - fi - - mkdir -p /shared-home/.local/bin - echo '#!/bin/sh' > /shared-home/.local/bin/gh - echo 'exit 1' >> /shared-home/.local/bin/gh - chmod +x /shared-home/.local/bin/gh - - securityContext: - runAsUser: 0 # Run as root briefly to copy and fix permissions - runAsNonRoot: false - volumeMounts: - - name: hermes-agent-src - mountPath: /shared-src - - name: hermes-home - mountPath: /shared-home - - name: hermes-configmap - mountPath: /tmp/hermes/config.yaml - subPath: config.yaml - containers: - # ========================================== - # CONTAINER 1: HERMES AGENT - # ========================================== - - name: rays-hermes-agent - image: nousresearch/hermes-agent:latest - args: ["gateway", "run"] - ports: - - containerPort: 8642 - - env: - - name: PATH - value: "/home/hermes/.hermes/.local/bin:/opt/hermes/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" - - name: HOME - value: "/home/hermes/.hermes" - - name: HERMES_HOME - value: "/home/hermes/.hermes" - - name: HERMES_UID - value: "1024" - - name: HERMES_GID - value: "1000" - - - name: TELEGRAM_BOT_TOKEN - valueFrom: - secretKeyRef: - name: hermes-secrets - key: TELEGRAM_BOT_TOKEN - - - name: XAI_API_KEY - valueFrom: - secretKeyRef: - name: xai-apikey - key: XAI_API_KEY - - - name: TELEGRAM_ALLOWED_USERS - value: "7105451284" - - # === Local vLLM (OpenAI-compatible) === - - name: OPENAI_BASE_URL - value: "http://openclaw-brain-service.customer1.svc.cluster.local:8000/v1" # ← adjust if your service name differs - - - name: HERMES_MODEL_PROVIDER - value: xai - - - name: HERMES_MODEL - value: grok-4.20-0309-reasoning - - - name: OPENAI_API_KEY - value: "dummy" - - - - volumeMounts: - - name: hermes-home - mountPath: /home/hermes/.hermes - - name: hermes-agent-src - mountPath: /opt/hermes - - securityContext: - runAsUser: 1024 - runAsGroup: 1000 - runAsNonRoot: true - allowPrivilegeEscalation: true - - # ========================================== - # CONTAINER 2: HERMES WEBUI - # ========================================== - - name: hermes-webui - image: ghcr.io/nesquena/hermes-webui:latest - ports: - - containerPort: 8787 - - env: - - name: HOME - value: "/home/hermeswebui/.hermes" - - name: HERMES_HOME - value: "/home/hermeswebui/.hermes" - - name: HERMES_WEBUI_HOST - value: "0.0.0.0" - - name: HERMES_WEBUI_PORT - value: "8787" - - name: HERMES_WEBUI_STATE_DIR - value: "/home/hermeswebui/.hermes/webui" - - name: WANTED_UID - value: "1024" - - name: WANTED_GID - value: "1000" - - name: HERMES_SKIP_CHMOD - value: "1" - - volumeMounts: - - name: hermes-home - mountPath: /home/hermeswebui/.hermes - # This is where the WebUI looks for the agent source code to run `uv pip install` - - name: hermes-agent-src - mountPath: /home/hermeswebui/.hermes/hermes-agent - - name: hermes-workspace - mountPath: /workspace - - name: hermes-webui-app - mountPath: /app - - resources: - requests: - memory: 500Mi - cpu: "100m" - limits: - memory: 1Gi - cpu: "500m" - - securityContext: - runAsUser: 1024 - runAsGroup: 1000 - runAsNonRoot: true - allowPrivilegeEscalation: true - readOnlyRootFilesystem: false - seccompProfile: - type: RuntimeDefault ---- - -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: rays-hermes-agent-pvc - namespace: customer1 -spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 25Gi - ---- - - -apiVersion: v1 -kind: Service -metadata: - name: rays-hermes-webui-service - namespace: customer1 - annotations: - tailscale.com/expose: "true" - tailscale.com/hostname: "rays-hermes-webui" - tailscale.com/tags: "tag:k8s-operator" - tailscale.com/ports: "http:8787" -spec: - type: ClusterIP - selector: - app: hermes-agent - ports: - - port: 8787 - targetPort: 8787 - name: http diff --git a/trade-dashboard/Dockerfile b/trade-dashboard/Dockerfile deleted file mode 100644 index 9ca7f24..0000000 --- a/trade-dashboard/Dockerfile +++ /dev/null @@ -1,18 +0,0 @@ -FROM python:3.13-slim AS base - -WORKDIR /app - -RUN apt-get update && apt-get install -y --no-install-recommends \ - gcc libpq-dev \ - && rm -rf /var/lib/apt/lists/* - -COPY app/requirements.txt . -RUN pip install --no-cache-dir -r requirements.txt - -COPY app/ ./app/ -COPY alembic.ini ./alembic.ini -COPY alembic/ ./alembic/ - -EXPOSE 8000 -ENV PYTHONPATH=/app/app -CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"] diff --git a/trade-dashboard/alembic.ini b/trade-dashboard/alembic.ini deleted file mode 100644 index 60ba74e..0000000 --- a/trade-dashboard/alembic.ini +++ /dev/null @@ -1,36 +0,0 @@ -[alembic] -script_location = alembic -sqlalchemy.url = postgresql+asyncpg://trading:CHANGE_ME@hermes-pgdb-rw.customer1.svc.cluster.local:5432/trading_data - -[loggers] -keys = root,sqlalchemy,alembic - -[handlers] -keys = console - -[formatters] -keys = generic - -[logger_root] -level = WARN -handlers = console - -[logger_sqlalchemy] -level = WARN -handlers = -qualname = sqlalchemy.engine - -[logger_alembic] -level = INFO -handlers = -qualname = alembic - -[handler_console] -class = StreamHandler -args = (sys.stderr,) -level = NOTSET -formatter = generic - -[formatter_generic] -format = %(levelname)-5.5s [%(name)s] %(message)s -datefmt = %H:%M:%S diff --git a/trade-dashboard/alembic/env.py b/trade-dashboard/alembic/env.py deleted file mode 100644 index 5fcf711..0000000 --- a/trade-dashboard/alembic/env.py +++ /dev/null @@ -1,62 +0,0 @@ -"""Alembic environment configuration.""" - -import sys -from logging.config import fileConfig -from pathlib import Path - -from alembic import context -from sqlalchemy import engine_from_config, pool -from sqlalchemy.ext.asyncio import AsyncEngine - -sys.path.insert(0, str(Path(__file__).parent.parent / "app")) - -config = context.config - -if config.config_file_name is not None: - fileConfig(config.config_file_name) - -from models import metadata # noqa: E402 - -target_metadata = metadata - - -def run_migrations_offline() -> None: - """Run migrations in 'offline' mode.""" - url = config.get_main_option("sqlalchemy.url") - context.configure( - url=url, - target_metadata=target_metadata, - literal_binds=True, - dialect_opts={"paramstyle": "named"}, - ) - with context.begin_transaction(): - context.run_migrations() - - -def do_run_migrations(connection): - context.configure(connection=connection, target_metadata=target_metadata) - with context.begin_transaction(): - context.run_migrations() - - -async def run_migrations_online() -> None: - """Run migrations in 'online' mode.""" - connectable = AsyncEngine( - engine_from_config( - config.get_section(config.config_ini_section) or {}, - prefix="sqlalchemy.", - poolclass=pool.NullPool, - future=True, - ) - ) - async with connectable.connect() as connection: - await connection.run_sync(do_run_migrations) - await connectable.dispose() - - -import asyncio - -if context.is_offline_mode(): - run_migrations_offline() -else: - asyncio.run(run_migrations_online()) diff --git a/trade-dashboard/alembic/script.py.mako b/trade-dashboard/alembic/script.py.mako deleted file mode 100644 index d458e51..0000000 --- a/trade-dashboard/alembic/script.py.mako +++ /dev/null @@ -1,2 +0,0 @@ -# Alembic migration script - DO NOT EDIT MANUALLY -# Use: alembic revision --autogenerate -m "description" diff --git a/trade-dashboard/alembic/versions/001_initial.py b/trade-dashboard/alembic/versions/001_initial.py deleted file mode 100644 index 702c57d..0000000 --- a/trade-dashboard/alembic/versions/001_initial.py +++ /dev/null @@ -1,43 +0,0 @@ -"""initial schema — positions table - -Revision ID: 001_initial -Create Date: 2026-05-02 -""" - -from alembic import op -import sqlalchemy as sa -from sqlalchemy.dialects import postgresql - -revision = "001_initial" -down_revision = None -branch_labels = None -depends_on = None - - -def upgrade() -> None: - op.execute('CREATE TYPE position_direction AS ENUM (\'long\', \'short\')') - - op.create_table( - "positions", - sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True, server_default=sa.text("gen_random_uuid()")), - sa.Column("symbol", sa.String(32), nullable=False), - sa.Column("direction", postgresql.ENUM("long", "short", name="position_direction", create_type=False), nullable=False), - sa.Column("entry_price", sa.Numeric(precision=16, scale=8), nullable=False), - sa.Column("exit_price", sa.Numeric(precision=16, scale=8)), - sa.Column("quantity", sa.Numeric(precision=16, scale=8), nullable=False), - sa.Column("exchange", sa.String(32), nullable=False), - sa.Column("opened_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), - sa.Column("closed_at", sa.DateTime(timezone=True)), - sa.Column("pnl", sa.Numeric(precision=16, scale=2)), - sa.Column("metadata", sa.JSON), - ) - - op.create_index(op.f("ix_positions_symbol"), "positions", ["symbol"]) - op.create_index(op.f("ix_positions_exchange"), "positions", ["exchange"]) - - -def downgrade() -> None: - op.drop_index(op.f("ix_positions_exchange"), table_name="positions") - op.drop_index(op.f("ix_positions_symbol"), table_name="positions") - op.drop_table("positions") - op.execute("DROP TYPE IF EXISTS position_direction") diff --git a/trade-dashboard/app/database.py b/trade-dashboard/app/database.py deleted file mode 100644 index f6c74d9..0000000 --- a/trade-dashboard/app/database.py +++ /dev/null @@ -1,16 +0,0 @@ -import os - -from sqlalchemy import MetaData -from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine - -DB_USER = os.getenv("DB_USER", "trading") -DB_PASS = os.getenv("DB_PASSWORD", "") -DB_HOST = os.getenv("DB_HOST", "hermes-pgdb-rw.customer1.svc.cluster.local") -DB_PORT = os.getenv("DB_PORT", "5432") -DB_NAME = os.getenv("DB_NAME", "trading_data") - -DATABASE_URL = f"postgresql+asyncpg://{DB_USER}:{DB_PASS}@{DB_HOST}:{DB_PORT}/{DB_NAME}" - -engine = create_async_engine(DATABASE_URL, echo=False, pool_size=5, max_overflow=10) -async_session = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False) -metadata = MetaData() diff --git a/trade-dashboard/app/main.py b/trade-dashboard/app/main.py deleted file mode 100644 index c144033..0000000 --- a/trade-dashboard/app/main.py +++ /dev/null @@ -1,253 +0,0 @@ -"""Trade Dashboard — FastAPI service for tracking PnL and open positions.""" - -from __future__ import annotations - -from datetime import datetime, timedelta, timezone -from decimal import Decimal -from pathlib import Path -from uuid import UUID - -from fastapi import FastAPI, HTTPException, Query -from fastapi.responses import FileResponse, HTMLResponse -from sqlalchemy import and_, func, select -from sqlalchemy.ext.asyncio import AsyncSession -from database import async_session -from models import positions -from schemas import ( - Direction, - PnLSnapshot, - PositionCreate, - PositionOut, - PositionUpdate, - WebhookTrade, -) - -app = FastAPI(title="Trade Dashboard", version="0.1.0") - -STATIC_DIR = Path(__file__).parent / "static" - - -# ── Helpers ───────────────────────────────────────────────────────────── - -def position_to_out(row: dict) -> PositionOut: - return PositionOut( - id=row["id"], - symbol=row["symbol"], - direction=row["direction"], - entry_price=row["entry_price"], - exit_price=row["exit_price"], - quantity=row["quantity"], - exchange=row["exchange"], - opened_at=row["opened_at"], - closed_at=row["closed_at"], - pnl=row["pnl"], - metadata=row["metadata"], - ) - - -# ── Health ───────────────────────────────────────────────────────────── - -@app.get("/api/health") -async def health(): - async with async_session() as session: - result = await session.execute(select(func.now())) - db_time = result.scalar() - return {"status": "ok", "db_time": db_time.isoformat()} - - -# ── Positions ──────────────────────────────────────────────────────────── - -@app.get("/api/positions", response_model=list[PositionOut]) -async def list_positions( - open_only: bool = Query(True, description="Only show open positions"), -): - async with async_session() as session: - if open_only: - stmt = select(positions).where(positions.c.closed_at.is_(None)).order_by(positions.c.opened_at.desc()) - else: - stmt = select(positions).order_by(positions.c.opened_at.desc()) - rows = (await session.execute(stmt)).mappings().all() - return [position_to_out(r) for r in rows] - - -@app.post("/api/positions", status_code=201) -async def create_position(payload: PositionCreate): - async with async_session() as session: - values = payload.model_dump() - result = await session.execute(positions.insert().values(**values)) - session.commit() - pk = result.inserted_primary_key[0] - return {"id": str(pk)} - - -@app.patch("/api/positions/{position_id}") -async def update_position(position_id: UUID, payload: PositionUpdate): - async with async_session() as session: - row = await session.execute( - select(positions).where(positions.c.id == position_id) - ) - row = row.mappings().one_or_none() - if not row: - raise HTTPException(404, "Position not found") - - updates = payload.model_dump(exclude_unset=True) - - # Auto-compute PnL if closing - if "exit_price" in updates: - entry = row["entry_price"] - qty = row["quantity"] - exit_p = updates["exit_price"] - direction = row["direction"] - if direction == "long": - updates["pnl"] = float((exit_p - entry) * qty) - else: - updates["pnl"] = float((entry - exit_p) * qty) - updates["closed_at"] = datetime.now(timezone.utc) - - await session.execute( - positions.update().where(positions.c.id == position_id).values(**updates) - ) - session.commit() - - return {"ok": True} - - -@app.delete("/api/positions/{position_id}") -async def close_position(position_id: UUID, exit_price: Decimal = Query(None)): - async with async_session() as session: - row = await session.execute( - select(positions).where(positions.c.id == position_id) - ) - row = row.mappings().one_or_none() - if not row: - raise HTTPException(404, "Position not found") - - if row["closed_at"]: - raise HTTPException(400, "Position already closed") - - exit_p = exit_price or row["entry_price"] # breakeven default - entry = row["entry_price"] - qty = row["quantity"] - direction = row["direction"] - - if direction == "long": - pnl = float((exit_p - entry) * qty) - else: - pnl = float((entry - exit_p) * qty) - - await session.execute( - positions.update() - .where(positions.c.id == position_id) - .values(exit_price=exit_p, closed_at=datetime.now(timezone.utc), pnl=pnl) - ) - session.commit() - - return {"ok": True, "pnl": pnl, "exit_price": float(exit_p)} - - -# ── PnL ───────────────────────────────────────────────────────────────── - -@app.get("/api/pnl", response_model=PnLSnapshot) -async def get_pnl(): - async with async_session() as session: - now = datetime.now(timezone.utc) - today = now.replace(hour=0, minute=0, second=0, microsecond=0) - week_start = today - timedelta(days=now.weekday()) - month_start = today.replace(day=1) - - # Summary for closed trades - closed = select( - func.coalesce(func.sum(positions.c.pnl), 0).label("total"), - func.count(positions.c.id).label("count"), - ).where(positions.c.closed_at.isnot(None)) - - result = (await session.execute(closed)).mappings().one() - all_time_pnl = float(result["total"]) - total_trades = result["count"] - - # PnL by period - def period_query(start): - return select( - func.coalesce(func.sum(positions.c.pnl), 0) - ).where( - and_( - positions.c.closed_at.isnot(None), - positions.c.closed_at >= start, - ) - ) - - today_pnl = float((await session.execute(period_query(today))).scalar()) - week_pnl = float((await session.execute(period_query(week_start))).scalar()) - month_pnl = float((await session.execute(period_query(month_start))).scalar()) - - # Open count - open_count = (await session.execute( - select(func.count()).where(positions.c.closed_at.is_(None)) - )).scalar() - - return PnLSnapshot( - today_pnl=Decimal(str(today_pnl)), - week_pnl=Decimal(str(week_pnl)), - month_pnl=Decimal(str(month_pnl)), - all_time_pnl=Decimal(str(all_time_pnl)), - total_trades=total_trades, - open_positions=open_count, - ) - - -@app.get("/api/pnl/history", response_model=list[PositionOut]) -async def pnl_history( - limit: int = Query(50, ge=1, le=500), -): - async with async_session() as session: - stmt = ( - select(positions) - .where(positions.c.closed_at.isnot(None)) - .order_by(positions.c.closed_at.desc()) - .limit(limit) - ) - rows = (await session.execute(stmt)).mappings().all() - return [position_to_out(r) for r in rows] - - -# ── Webhook (for scanner scripts) ─────────────────────────────────────── - -@app.post("/webhook/trade", status_code=201) -async def webhook_trade(payload: WebhookTrade): - meta = {"strategy": payload.strategy} if payload.strategy else {} - async with async_session() as session: - result = await session.execute(positions.insert().values(**{ - "symbol": payload.symbol, - "direction": payload.direction, - "entry_price": payload.entry_price, - "quantity": payload.quantity, - "exchange": payload.exchange, - "metadata": meta, - })) - session.commit() - pk = result.inserted_primary_key[0] - return {"id": str(pk)} - - -# ── Frontend ──────────────────────────────────────────────────────────── - -@app.get("/", response_class=HTMLResponse) -async def index(): - return FileResponse(str(STATIC_DIR / "index.html")) - - -# ── Startup: run Alembic migrations ────────────────────────────────────── - -@app.on_event("startup") -async def startup(): - from alembic import command - from alembic.config import Config - from pathlib import Path - - alembic_cfg = Config( - str(Path(__file__).parent.parent / "alembic.ini") - ) - -if __name__ == "__main__": - import uvicorn - uvicorn.run(app, host="0.0.0.0", port=8000) diff --git a/trade-dashboard/app/models.py b/trade-dashboard/app/models.py deleted file mode 100644 index 9e0e92d..0000000 --- a/trade-dashboard/app/models.py +++ /dev/null @@ -1,21 +0,0 @@ -from sqlalchemy import Column, String, Numeric, Enum, DateTime, JSON, func, Table -from sqlalchemy.dialects.postgresql import UUID -import uuid - -from database import metadata - -positions = Table( - "positions", - metadata, - Column("id", UUID(as_uuid=True), primary_key=True, default=uuid.uuid4), - Column("symbol", String(32), nullable=False, index=True), - Column("direction", Enum("long", "short", name="position_direction"), nullable=False), - Column("entry_price", Numeric(precision=16, scale=8), nullable=False), - Column("exit_price", Numeric(precision=16, scale=8)), - Column("quantity", Numeric(precision=16, scale=8), nullable=False), - Column("exchange", String(32), nullable=False, index=True), - Column("opened_at", DateTime(timezone=True), server_default=func.now(), nullable=False), - Column("closed_at", DateTime(timezone=True)), - Column("pnl", Numeric(precision=16, scale=2)), - Column("metadata", JSON), -) diff --git a/trade-dashboard/app/requirements.txt b/trade-dashboard/app/requirements.txt deleted file mode 100644 index 51462ea..0000000 --- a/trade-dashboard/app/requirements.txt +++ /dev/null @@ -1,7 +0,0 @@ -fastapi==0.115.0 -uvicorn[standard]==0.32.0 -sqlalchemy[asyncio]==2.0.35 -asyncpg==0.30.0 -alembic==1.14.0 -pydantic==2.9.2 -python-dotenv==1.0.1 diff --git a/trade-dashboard/app/schemas.py b/trade-dashboard/app/schemas.py deleted file mode 100644 index f911657..0000000 --- a/trade-dashboard/app/schemas.py +++ /dev/null @@ -1,70 +0,0 @@ -from __future__ import annotations - -from datetime import datetime -from decimal import Decimal -from enum import Enum -from typing import Optional -from uuid import UUID - -from pydantic import BaseModel - - -class Direction(str, Enum): - long = "long" - short = "short" - - -# ─── Request schemas ────────────────────────────────────────────── - -class PositionCreate(BaseModel): - symbol: str - direction: Direction - entry_price: Decimal - quantity: Decimal - exchange: str - metadata: Optional[dict] = None - - -class PositionUpdate(BaseModel): - entry_price: Optional[Decimal] = None - exit_price: Optional[Decimal] = None - quantity: Optional[Decimal] = None - metadata: Optional[dict] = None - - -class WebhookTrade(BaseModel): - """Payload from automated scanner scripts.""" - symbol: str - direction: Direction - entry_price: Decimal - quantity: Decimal - exchange: str - strategy: Optional[str] = None - - -# ─── Response schemas ───────────────────────────────────────────── - -class PositionOut(BaseModel): - id: UUID - symbol: str - direction: Direction - entry_price: Decimal - exit_price: Optional[Decimal] - quantity: Decimal - exchange: str - opened_at: datetime - closed_at: Optional[datetime] - pnl: Optional[Decimal] - metadata: Optional[dict] - - model_config = {"from_attributes": True} - - -class PnLSnapshot(BaseModel): - today_pnl: Decimal - week_pnl: Decimal - month_pnl: Decimal - all_time_pnl: Decimal - total_trades: int - open_positions: int - diff --git a/trade-dashboard/app/static/index.html b/trade-dashboard/app/static/index.html deleted file mode 100644 index 8795aba..0000000 --- a/trade-dashboard/app/static/index.html +++ /dev/null @@ -1,158 +0,0 @@ - - - - - -Trade Dashboard - - - -

📊 Trade Dashboard

- - -
-
Today PnL
—
-
Week PnL
—
-
Month PnL
—
-
All Time
—
-
- - -
-

Open Positions 0

- - - -
SymbolDirEntryQtyExchangeOpenedAction
-
- - -
-

Open New Position

-
- - - - - - -
-
- - -
-

Trade History

- - - -
SymbolDirEntryExitQtyPnLExchangeClosed
-
- - - -