fix secret issue

2026-05-22 02:11:22 +00:00 · 2026-05-22 02:11:22 +00:00 · 75cea5abb7
commit 75cea5abb7
parent 84e85d1432
3 changed files with 19 additions and 672 deletions
--- a/analyses/telegram-webhook-container-analysis.md
+++ b/analyses/telegram-webhook-container-analysis.md
@ -1,303 +0,0 @@
 # Telegram Webhook Failure Analysis - Deep Dive (Container-Level)
 **Date:** 2026-05-10
 **Repo:** https://github.com/sirius0xdev/gcloud-lab
 **Cluster:** devops-lab (GKE), Namespace: customer1
 **Webhook URL:** https://ws.siriusdevops.com/telegram/webhook/default
 ---
 ## Status Update
 **TLS certificates ARE provisioned** in GCP admin console. The Gateway certmap annotation is working.
 The real issues are in the **container configuration** and **missing probes**.
 ---
 ## Root Cause #1: NO Liveness/Readiness Probes (CONFIRMED CRITICAL)
 **The hermes-agent deployment has ZERO probes defined.**
 This means:
 - Kubernetes marks pods "Ready" immediately after container start
 - The Gateway routes webhook traffic before the process has bound to port 9118
 - During restarts, there is no graceful drain
 - **A crashed or hung pod stays in the endpoint list forever**
 ### The /health endpoint problem
 The Telegram webhook server runs on **port 9118** via python-telegram-bot's `start_webhook()`. This internally starts an aiohttp server that **ONLY registers the webhook path** (`/telegram/webhook/default`). It does NOT expose a `/health` endpoint.
 So a probe like this would FAIL:
 ```yaml
 # THIS WON'T WORK - port 9118 has no /health
 readinessProbe:
  httpGet:
    path: /health
    port: 9118
 ```
 ### What DOES have a health endpoint?
 The **generic webhook adapter** on **port 8644** IS a Hermes-managed server. From the source code, it exposes `/health`. This IS enabled via `WEBHOOK_ENABLED: "true"`.
 ### The Fix
 Add probes targeting port 8644 (the generic webhook server that IS healthy when the gateway is running):
 ```yaml
 readinessProbe:
  httpGet:
    path: /health
    port: 8644
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3
 livenessProbe:
  httpGet:
    path: /health
    port: 8644
  initialDelaySeconds: 30
  periodSeconds: 30
  timeoutSeconds: 5
  failureThreshold: 3
 ```
 Alternatively, enable the API server on port 8642 and probe there (it also has `/health`).
 ---
 ## Root Cause #2: NO Resource Limits (HIGH)
 The **new deployment dropped all resource limits** that existed in the old deployment.
 **Old deployment (`deployment.yaml`):**
 ```yaml
 resources:
  requests:
    memory: 2Gi
    cpu: "1"
  limits:
    memory: 3Gi
    cpu: "2"
 ```
 **New deployment (`new-deployment.yaml`):**
 ```yaml
 # NO resources block for the hermes-agent container
 ```
 Only the `hermes-webui` sidecar has limits (500Mi-1Gi memory, 100m-500m CPU).
 ### Impact
 - The hermes-agent container can consume unbounded memory
 - With `agent.max_turns: 90` and `gateway_timeout: 1800` (30 min), a single agent run can eat massive memory
 - Pod may be OOMKilled by the node, but K8s won't restart it gracefully without probes
 - **This is a likely cause of intermittent failures**
 ### Fix
 Add resource limits back:
 ```yaml
 resources:
  requests:
    memory: 2Gi
    cpu: "1"
  limits:
    memory: 4Gi
    cpu: "2"
 ```
 ---
 ## Root Cause #3: Path Handling Concern (MEDIUM)
 **Source code** (`/opt/hermes/gateway/platforms/telegram.py:1213`):
 ```python
 webhook_path = urlparse(webhook_url).path or "/telegram"
 ```
 For `TELEGRAM_WEBHOOK_URL=https://ws.siriusdevops.com/telegram/webhook/default`:
 - `webhook_path` = `/telegram/webhook/default`
 - PTB's aiohttp server registers a handler at exactly `/telegram/webhook/default` on port 9118
 **HTTPRoute** (`hermes-webhook.yaml`):
 ```yaml
 rules:
 - matches:
  - path:
      type: PathPrefix
      value: /telegram/webhook
  backendRefs:
  - name: http-tele-webhook
    port: 9118
 ```
 **The question:** Does the GKE Gateway strip the `/telegram/webhook` prefix before forwarding?
 - If it does NOT strip: Pod receives `/telegram/webhook/default` -> **OK**
 - If it DOES strip to `/default`: Pod receives `/default` -> **404**
 - If it strips to `/`: Pod receives `/` -> **404**
 Most GatewayAPI implementations pass the full original path by default, but some ingress controllers strip the matched prefix. **Verify this on the running cluster.**
 ---
 ## Root Cause #4: API Server Not Enabled (MEDIUM)
 The old deployment had:
 ```yaml
 - name: API_SERVER_ENABLED
  value: "true"
 - name: API_SERVER_HOST
  value: "0.0.0.0"
 - name: API_SERVER_PORT
  value: "8642"
 - name: API_SERVER_KEY
  valueFrom: {secretKeyRef: ...}
 - name: API_SERVER_MODEL_NAME
  value: "hermes-agent"
 ```
 **These are ALL absent from `new-deployment.yaml`.** Port 8642 is declared as a containerPort but nothing listens on it because `API_SERVER_ENABLED` is not set.
 The API server exposes `/health` and `/health/detailed` endpoints. Without it, you lose a convenient health check and the API server interface.
 ---
 ## Other Findings
 ### 5. TELEGRAM_WEBHOOK_ENABLED is Ignored (INFO)
 The code only checks if `TELEGRAM_WEBHOOK_URL` is set (non-empty). It does NOT read `TELEGRAM_WEBHOOK_ENABLED`. The env var in the deployment is a dead config — set but ignored.
 ### 6. vLLM Service Name Mismatch (INFO)
 ```yaml
 OPENAI_BASE_URL: "http://openclaw-brain-service.customer1.svc.cluster.local:8000/v1"
 ```
 But the active vLLM deployment (`rtx6000-vllm.yaml`) creates a service named `rtx6000-brain-service`. If Hermes ever switches from xAI to the openai provider, local vLLM is unreachable.
 ### 7. SOPS Secrets (VERIFY)
 Both `hermes-secret.yaml` and `tele-webhook.yaml` are SOPS-encrypted. Verify they are decrypted in the cluster:
 ```bash
 kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d
 kubectl get secret telegram-webhook -n customer1 -o jsonpath='{.data.TELEGRAM_WEBHOOK_SECRET}' | base64 -d
 ```
 ### 8. Container Env Vars Summary
 | Variable | Value | Notes |
 |----------|-------|-------|
 | TELEGRAM_WEBHOOK_URL | https://ws.siriusdevops.com/telegram/webhook/default | OK |
 | TELEGRAM_WEBHOOK_PORT | 9118 | OK |
 | TELEGRAM_WEBHOOK_SECRET | From SOPS secret | Verify decrypted |
 | TELEGRAM_WEBHOOK_ENABLED | "true" | **Ignored by code** |
 | TELEGRAM_BOT_TOKEN | From SOPS secret | Verify decrypted |
 | TELEGRAM_ALLOWED_USERS | 7528130947 | OK |
 | WEBHOOK_ENABLED | "true" | OK (enables port 8644 /health) |
 | WEBHOOK_PORT | 8644 | OK |
 | API_SERVER_ENABLED | **NOT SET** | Port 8642 has no listener |
 | HERMES_MODEL_PROVIDER | xai | OK (uses Grok) |
 | HERMES_MODEL | grok-4.20-0309-reasoning | OK |
 ---
 ## Recommended Fix (Priority Order)
 ### P0 - Add Probes (will detect and restart hung/crashed pods)
 Add to `new-deployment.yaml` under the hermes-agent container spec:
 ```yaml
 readinessProbe:
  httpGet:
    path: /health
    port: 8644
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3
 livenessProbe:
  httpGet:
    path: /health
    port: 8644
  initialDelaySeconds: 30
  periodSeconds: 30
  timeoutSeconds: 5
  failureThreshold: 3
 ```
 ### P1 - Add Resource Limits (prevents OOM kills)
 ```yaml
 resources:
  requests:
    memory: 2Gi
    cpu: "1"
  limits:
    memory: 4Gi
    cpu: "2"
 ```
 ### P2 - Verify Path Handling
 Test if the Gateway passes the full path:
 ```bash
 # From inside the pod, check what the webhook server receives
 kubectl exec -n customer1 deploy/hermes-agent -- curl -s http://localhost:9118/telegram/webhook/default -X POST -H "Content-Type: application/json" -d '{}'
 ```
 ### P3 - Enable API Server (optional, gives /health on 8642)
 Add back the API_SERVER_ENABLED env var if you want the API server health endpoint.
 ---
 ## Diagnostic Commands
 Run these on the cluster RIGHT NOW to confirm the current state:
 ```bash
 # 1. Check pod status and restart count
 kubectl get pods -n customer1 -l app=hermes-agent -o wide
 # 2. Check events for OOMKilled or probe failures
 kubectl describe pod -n customer1 -l app=hermes-agent | grep -A5 -i 'oom\|probe\|restart'
 # 3. Check if the webhook port is actually listening
 kubectl exec -n customer1 deploy/hermes-agent -- ss -tlnp | grep 9118
 # 4. Check if port 8644 health endpoint works
 kubectl exec -n customer1 deploy/hermes-agent -- curl -s http://localhost:8644/health
 # 5. Check logs for webhook startup messages
 kubectl logs -n customer1 deploy/hermes-agent --tail=50 | grep -i 'webhook\|listening\|9118'
 # 6. Verify SOPS secrets are decrypted
 kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d && echo
 kubectl get secret telegram-webhook -n customer1 -o jsonpath='{.data.TELEGRAM_WEBHOOK_SECRET}' | base64 -d && echo
 # 7. Check if the webhook is registered with Telegram (using the pod's bot token)
 BOT_TOKEN=$(kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d)
 curl -s "https://api.telegram.org/bot${BOT_TOKEN}/getWebhookInfo" | python3 -m json.tool
 # 8. Test the full path through the Gateway
 curl -v https://ws.siriusdevops.com/telegram/webhook/default -X POST -H "Content-Type: application/json" -d '{}' 2>&1
 # 9. Check container memory usage
 kubectl top pod -n customer1 -l app=hermes-agent 2>/dev/null || echo "metrics-server not available"
 # 10. Check resource limits on the container
 kubectl get pod -n customer1 -l app=hermes-agent -o jsonpath='{.items[0].spec.containers[0].resources}'
 ```
 ---
 ## Files to Modify
 1. `apps/base/customer1/hermes-agent/new-deployment.yaml` - Add probes + resource limits
 2. Optionally: Re-enable API server env vars in `new-deployment.yaml`
--- a/analyses/telegram-webhook-failure-analysis.md
+++ b/analyses/telegram-webhook-failure-analysis.md
@ -1,345 +0,0 @@
 # Telegram Webhook Failure Analysis - Hermes Agent on GKE
 **Date:** 2026-05-10
 **Repo:** https://github.com/sirius0xdev/gcloud-lab
 **Cluster:** devops-lab (GKE)
 **Namespace:** customer1
 **Webhook URL:** https://ws.siriusdevops.com/telegram/webhook/default
 ---
 ## Executive Summary
 The Telegram webhook is failing because **there is no valid TLS certificate for `ws.siriusdevops.com`** at the Gateway layer. Telegram strictly requires HTTPS with a publicly-trusted certificate for webhook delivery. The Gateway listener declares `protocol: HTTPS` but has no `tls.certificateRefs` and relies on a GKE `CertMap` annotation referencing `gateway-cert-map` — a resource that **does not exist** in the repository. Cert-manager is configured but disconnected from the GatewayAPI setup (HTTP-01 solver points to a non-existent Traefik ingress class).
 **TL;DR:** Telegram tries to POST to `https://ws.siriusdevops.com/...`, but the Gateway has no certificate to present during the TLS handshake. Connection fails before it ever reaches the hermes-agent pod.
 ---
 ## Root Cause #1: Missing TLS Certificate (CRITICAL)
 ### What's happening
 **File:** `infrastructure/gatewayapi/apigateway.yaml`
 ```yaml
 listeners:
 - name: https
  protocol: HTTPS
  port: 443
  allowedRoutes:
    namespaces:
      from: All
 ```
 The listener says HTTPS but has **no `tls:` block**. It relies entirely on this annotation:
 ```yaml
 annotations:
  networking.gke.io/certmap: gateway-cert-map
 ```
 ### The problem
 - **No `CertMap` or `CertMapEntry` resource** exists anywhere in the repository for `gateway-cert-map`
 - Without it, GKE has no managed certificate to attach to the Gateway
 - Telegram's webhook delivery gets a TLS handshake failure or no certificate
 ### Cert-manager is also broken
 **File:** `infrastructure/controllers/base/certmanager/clusterissuer.yaml`
 ```yaml
 solvers:
  - http01:
      ingress:
        class: traefik
 ```
 - HTTP-01 solver references `class: traefik`, but **no Traefik ingress controller exists** in the cluster
 - **No `Certificate` CRs** exist for `ws.siriusdevops.com` or any other domain
 - Cert-manager is completely disconnected from the GatewayAPI setup
 ### How to fix (choose ONE approach)
 **Option A: GKE Managed Certificates (recommended for GatewayAPI)**
 Create a `ManagedCertificate` + `BackendConfig` or `CertMap`/`CertMapEntry`:
 ```yaml
 apiVersion: networking.gke.io/v1
 kind: ManagedCertificate
 metadata:
  name: hermes-webhook-cert
  namespace: customer1
 spec:
  domains:
    - ws.siriusdevops.com
    - brain.siriusdevops.com
    - paaas.siriusdevops.com
 ```
 Then add `tls.certificateRefs` to the Gateway listener:
 ```yaml
 listeners:
 - name: https
  protocol: HTTPS
  port: 443
  tls:
    certificateRefs:
    - name: hermes-webhook-cert
      group: networking.gke.io
 ```
 **Option B: Fix Cert-manager with DNS-01**
 Switch ClusterIssuer from HTTP-01/Traefik to DNS-01 (e.g., Cloudflare, GCP DNS, or Route53), then create `Certificate` resources for each domain.
 ---
 ## Root Cause #2: No Readiness/Liveness Probes (HIGH)
 **File:** `apps/base/customer1/hermes-agent/new-deployment.yaml`
 No readiness or liveness probes are defined on any hermes-agent deployment.
 ### Impact
 - Pods are marked "Ready" immediately after container start
 - Gateway routes traffic to the webhook port (9118) before the process has bound to it
 - During restarts, traffic hits pods that haven't initialized
 ### Fix
 Add probes to the deployment:
 ```yaml
 readinessProbe:
  httpGet:
    path: /health
    port: 9118
  initialDelaySeconds: 5
  periodSeconds: 10
 livenessProbe:
  httpGet:
    path: /health
    port: 9118
  initialDelaySeconds: 15
  periodSeconds: 30
 ```
 ---
 ## Root Cause #3: SOPS Encrypted Secrets (HIGH)
 **Files:**
 - `apps/base/customer1/hermes-agent/hermes-secret.yaml` (SOPS encrypted)
 - `apps/base/customer1/hermes-agent/tele-webhook.yaml` (SOPS encrypted)
 These contain `TELEGRAM_BOT_TOKEN` and `TELEGRAM_WEBHOOK_SECRET`.
 ### Risk
 - If the deployment pipeline (Flux/Kustomize controller) is **not decrypting SOPS secrets**, pods receive literal `ENC[...]` strings
 - The bot token would be invalid, so even if TLS worked, Telegram auth would fail
 - The webhook secret would not match, causing Telegram to reject payloads
 ### Verify
 Run `kubectl get secret hermes-secrets -n customer1 -o yaml` and check if values are base64-encoded real tokens or `ENC[...]` strings.
 ---
 ## Secondary Issues
 ### 3a. Service Name Mismatch (vLLM Integration)
 **File:** `apps/base/customer1/hermes-agent/new-deployment.yaml`
 ```yaml
 OPENAI_BASE_URL: "http://openclaw-brain-service.customer1.svc.cluster.local:8000/v1"
 ```
 But the active vLLM deployment (`rtx6000-vllm.yaml`) creates a service named **`rtx6000-brain-service`**.
 - `OPENAI_BASE_URL` points to `openclaw-brain-service` which may not exist
 - If Hermes ever switches to the `openai` provider (instead of `xai`), local vLLM is unreachable
 - The model provider defaults to `HERMES_MODEL_PROVIDER: xai` (Grok external API)
 ### 3b. Duplicate HTTPRoute Deployment
 The webhook HTTPRoute (`hermes-webhook.yaml`) is included in **two** kustomization trees:
 1. `infrastructure/gatewayapi/gateway-routes/kustomization.yaml` -> deployed via Flux
 2. `apps/base/customer1/hermes-agent/kustomization.yaml` -> deployed via Flux
 Same resource (`http-telegram-webhook` in `customer1`) from two sources. This may cause Flux reconciliation conflicts.
 ### 3c. Empty HF_TOKEN in vLLM Deployments
 All vLLM deployments have:
 ```yaml
 - name: HF_TOKEN
  value: ""
 ```
 If the model `edp1096/Huihui-Qwen3.6-27B-abliterated-FP8` is a gated model on HuggingFace, it will fail to download.
 ### 3d. KEDA Scale-to-Zero
 **File:** `infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml`
 ```yaml
 minReplicaCount: 0
 maxReplicaCount: 1
 ```
 - vLLM scales to **zero** when idle
 - First request after cold start incurs full model load time (30-60 seconds)
 - For real-time Telegram responses, this causes visible latency
 ### 3e. PVC Name Collision
 Both `rtx6000-vllm.yaml` and `a100-vllm.yaml` define a PVC named `vllm-model-qwen3.6-27b-uncensored` in namespace `customer1`. If both are ever active simultaneously, they conflict.
 ---
 ## Architecture Overview
 ```
                     Internet
                        |
                        v
              [GKE External LB]
                        |
          Gateway: external-http-gateway
         (port 443/HTTPS, NO TLS cert!)
                        |
            +-----------+-----------+
            |           |           |
     ws.siriusdevops.com   brain.siriusdevops.com   paaas.siriusdevops.com
            |           |           |
            v           v           v
    /telegram/webhook    /           /
            |           |           |
            v           v           v
   http-tele-webhook   rtx6000-   paaas-landing
       :9118            brain-       :8080
                       service:8000
            |
            v
   hermes-agent pod
   (ports: 8642, 8644, 9118)
            |
            v
   Model Provider: xai (Grok via external API)
   Fallback: OPENAI_BASE_URL -> openclaw-brain-service (MISMATCHED)
 ```
 ---
 ## vLLM Server Status
 | Server | GPU | Model | Quantization | Status |
 |--------|-----|-------|-------------|--------|
 | RTX 6000 | 1x Pro 6000 (96GB) | edp1096/Huihui-Qwen3.6-27B-abliterated-FP8 | FP8 | **ACTIVE** |
 | A100 | 1x A100 (80GB) | Youssofal/Qwen3.6-27B-Abliterated-Heretic-Uncensored-BF16 | BF16 | Commented out |
 | L4 | 1x L4 (24GB) | p-e-w/Qwen3-8B-heretic | auto | Commented out |
 | Gemma | 1x A100 (80GB) | coder3101/Qwen3.5-27B-heretic | BF16 | Not in kustomization |
 **Note:** The architecture plan (`plans/AI_ARCHITECTURE.md`) describes a dual-tier L4 dispatcher + A100 deep thinker setup, but the active deployment only has RTX 6000.
 ---
 ## Action Plan (Priority Order)
 ### P0 - Fix TLS (will unblock Telegram webhooks)
 1. **Create a `CertMap`/`CertMapEntry`** or **`ManagedCertificate`** resource for `ws.siriusdevops.com`
 2. **Add `tls.certificateRefs`** to the Gateway listener in `apigateway.yaml`
 3. Verify with: `curl -vI https://ws.siriusdevops.com/telegram/webhook/default`
 4. If cert is valid, Telegram should start delivering webhooks
 ### P1 - Verify Secrets
 5. Check if SOPS secrets are actually decrypted in the cluster
 6. Run: `kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d`
 ### P2 - Add Probes
 7. Add readiness/liveness probes to hermes-agent deployment
 8. Redeploy
 ### P3 - Clean Up GatewayAPI
 9. Remove duplicate `hermes-webhook.yaml` reference from one kustomization
 10. Fix or remove Traefik-referencing ClusterIssuers
 ### P4 - Fix vLLM Integration
 11. Fix `OPENAI_BASE_URL` service name or update kustomization to create `openclaw-brain-service`
 12. Set real `HF_TOKEN` in vLLM deployments
 13. Consider setting `minReplicaCount: 1` in KEDA for consistent response times
 ---
 ## Files Referenced
 ### GatewayAPI & TLS
 - `infrastructure/gatewayapi/apigateway.yaml` - Gateway definition (missing TLS)
 - `infrastructure/gatewayapi/gateway-routes/hermes-webhook.yaml` - Webhook HTTPRoute
 - `infrastructure/gatewayapi/gateway-routes/route.yaml` - General route
 - `infrastructure/controllers/base/certmanager/clusterissuer.yaml` - Cert-manager (Traefik mismatch)
 ### Hermes Agent Deployment
 - `apps/base/customer1/hermes-agent/new-deployment.yaml` - Active deployment + webhook service
 - `apps/base/customer1/hermes-agent/deployment.yaml` - Old deployment (no webhook config)
 - `apps/base/customer1/hermes-agent/configmap.yaml` - Hermes config
 - `apps/base/customer1/hermes-agent/hermes-secret.yaml` - SOPS encrypted secrets
 - `apps/base/customer1/hermes-agent/tele-webhook.yaml` - SOPS encrypted webhook secret
 - `apps/base/customer1/hermes-agent/kustomization.yaml` - Kustomize composition
 ### vLLM Infrastructure
 - `infrastructure/gpus/base/vllm-servers/rtx6000-vllm.yaml` - Active vLLM server
 - `infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml` - KEDA scaling
 - `infrastructure/gpus/base/keda-gpu-scaling/vllm-route.yaml` - vLLM external route
 ### Architecture Plans
 - `plans/AI_ARCHITECTURE.md` - AI architecture plan
 - `plans/models-to-try.md` - Models being considered
 ---
 ## Quick Diagnostic Commands
 Run these on the cluster to confirm findings:
 ```bash
 # 1. Check if TLS cert exists for the webhook domain
 curl -vI https://ws.siriusdevops.com/telegram/webhook/default 2>&1 | grep -E 'SSL|certificate|subject|issuer'
 # 2. Check Gateway status
 kubectl get gateway external-http-gateway -n customer1 -o yaml
 # 3. Check HTTPRoute status (attached/programmed)
 kubectl get httproute http-telegram-webhook -n customer1 -o yaml
 # 4. Verify secrets are decrypted
 kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d && echo
 # 5. Check if pods are actually ready
 kubectl get pods -n customer1 -l app=hermes-agent -o wide
 # 6. Check CertMap exists
 kubectl get certmap gateway-cert-map -A 2>/dev/null || echo "CertMap NOT FOUND"
 # 7. Check managed certificates
 kubectl get managedcertificate -A 2>/dev/null || echo "No ManagedCertificates"
 # 8. Check cert-manager certificates
 kubectl get certificate -A 2>/dev/null || echo "No Certificates"
 # 9. Check vLLM pod status
 kubectl get pods -n customer1 -l app=rtx6000-brain-vllm -o wide
 # 10. Test internal webhook endpoint
 kubectl exec -n customer1 deploy/hermes-agent -- curl -s http://localhost:9118/health || echo "Health check failed"
 ```
--- a/apps/base/customer1/trading-platform/secrets/trading-secrets.yaml
+++ b/apps/base/customer1/trading-platform/secrets/trading-secrets.yaml
@ -1,32 +1,27 @@
-apiVersion: ENC[AES256_GCM,data:j4g=,iv:Bb3E3dbyD1MUIsthCltT2rRNNorduAL7QmPG8oRy31g=,tag:htxjh/00z6RGQNNyKCUviw==,type:str]
+apiVersion: v1
-kind: ENC[AES256_GCM,data:k3FRKBzN,iv:Pvtu7bnGWQNl9pkjtU8GpHjpeg00oFXatf3jxxtQEpw=,tag:zj/yAy/DpCfLO7MdH3hiMA==,type:str]
+kind: Secret
 metadata:
-    name: ENC[AES256_GCM,data:ZpbqQd2YyEfnugJ27JVAmqBf8Q7m75B0,iv:4njbS+2AYJuxzNERSOjOH8wmoI1KM0ocUT7OgeoGEgo=,tag:spwlHcpeQM0G8O3VPHoDOg==,type:str]
+    name: trading-platform-secrets
-    namespace: ENC[AES256_GCM,data:4ErEXcBaI8Yp,iv:IKHgZ6Gm5X21Atnnm2xOFU11IgSfw5X5Wdnl25EDyOI=,tag:znkdsNDkIb4mxBY4yJbX9g==,type:str]
+    namespace: customer1
-type: ENC[AES256_GCM,data:dd7uKLw+,iv:qRkV8K+ytp55rLGNIP1lG2yZ+LENt/FkdDiWzi/1tik=,tag:aUWQ+ZOQS7/hXcnceCyrTQ==,type:str]
+type: Opaque
 stringData:
-    #ENC[AES256_GCM,data:Xol7d8ednDll9VKfZ62jZRdARcmz8UJ/6ovDaw8OHrPb72aUdA==,iv:TvGk+LiK0+maCioF5daeWDTNKOSRA5pBFf9PgKL/9Z4=,tag:34kee3TQNsCdIpyEtE12mA==,type:comment]
+    #ENC[AES256_GCM,data:X6HnX4CXbHydreDklg2LsCfBN1i1dp7HFifHR4rN2A581dEbYw==,iv:5MocUK+2zYn1qxhayaVnwVR+5FLLdJRKo9uBvW6RiTE=,tag:v13dWH9t/K8Whbs9EXcgnQ==,type:comment]
-    news-api-key: ENC[AES256_GCM,data:eNMLhs55u3bwVqD4l2tQhgq1+wp/9rCa,iv:7BjlJqgJbg6BqdXxNphnIWKA/LYFZJc3qqJ360/EteY=,tag:hs5vfj+zP9OCq2tDucItIg==,type:str]
+    news-api-key: ENC[AES256_GCM,data:Ren1WNxCGBcaV49M+W5Gl3FjfZJh3o/X,iv:Ly1v+YdGodfeQHLkYIJra+2+qqa06wAcu2RGv6cueRE=,tag:IuWIJB1mzipgmnP5rD2c1A==,type:str]
-    market-data-api-key: ENC[AES256_GCM,data:ol3aAC9ijFcFUo7jEVQv7mKjCLpUXBuaVThwRzG/fQ==,iv:YouuIdmj6aK0tuYJtDuiOt91gnUtK2xdshVZxy+gCH4=,tag:MAs8mS/+ndC8AYdI+WdFcA==,type:str]
+    market-data-api-key: ENC[AES256_GCM,data:2x2vPbOu/T+4albtkeRrktBffX1i06vXQmnn31685g==,iv:LqfpNmK6d9tt2MeaaozI+PVim8KUK8TflEalSxptfjM=,tag:Tan91JSeh5aUhGiExns4PQ==,type:str]
-    #ENC[AES256_GCM,data:Si0AKl5/sWrYDNSYiC35iD7vtvP9CzAb,iv:cyBrgR5cFSta+bPdyyCUXrKH68Hi84UbGFqiWskQb7s=,tag:w9+sotMjBTuyfcGYIHfN+w==,type:comment]
+    #ENC[AES256_GCM,data:jVRQi+JQNlDUQ+bCe189YrOcsXjYYTGF,iv:SnCbB2w1jnPnXqWGPEivK/b9bWnYA6qL1Xkx6n8XzQ8=,tag:SYJbW0Aea/evPgvGNCqw9Q==,type:comment]
-    service-auth-token: ENC[AES256_GCM,data:NCu/mBKdGIFNXTA5n7M53UXjMaZy9UdzImC90GPU,iv:JPLeNfTDWV3VLbVIUidL6w1IrJN09LTGIyPn0tJlj90=,tag:RIzC8FfGXBmE2PRZzgaAww==,type:str]
+    service-auth-token: ENC[AES256_GCM,data:Xrqq82eI5obC86s+FJpi4DbX9+hDdTlYdmfL6DlW,iv:lDhfIuT+On7W6Xk36FKw7fE7zBheRPaMZwQBn7DF5qc=,tag:CdE0OWzQ9OLkSsDL8gq9gA==,type:str]
 sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age1uuxf066xuuqgvjppxfcmqkwfcufnwp3wcwnl9h20g9k4l8nkw9jsaungf7
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFTzBrN3V5elJmNjNGZ28y
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBOWHFUcm9aNFlzRXk3bmtY
-            Q1JOQUt5eURqSFZNZkJUNWMrV1NTZEFiYkgwCnlDSTFYTW8zcU02M0NJdXZjYSti
+            b3hMRC85a2x3R1lIVmZoTk4xT1JtMDF0MEZFCmhjRGdUYWJpVnFkMTZpTVg4U01Z
-            Ylh0bWpJd0k2MWt6VjNNTUlNU0pTUVUKLS0tIENHUEU4VElXbC96bXBGRmo3QXpQ
+            c0trekthejd6QUk3REZ2Rms0Y2hyOFUKLS0tIExseng1M0RNU3E2d1NXNlJQVGdh
-            ZkxxUDRubGt0dnRoQXVtS2xFSnhTRkUKs+rcKiZvgA7mffGo7GkkFL4vWnTIGAIn
+            andDeDdxM2pjU05GL0pKdXIwUzc2M00KBfnTHcj5QMYGlQLRVO34sbtPsjRz37ZJ
-            RXwlbDNPEhiK+6lh/TgkV2CYXDBt1Hwfk4fzhZknYjY3Psp0ufvY2Q==
+            LO1q7fngycQGm0oNgZAZnxQNMArlNIUduH3Y8M365XushwPc9BHuQw==
            -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-05-21T04:08:04Z"
+    lastmodified: "2026-05-22T02:10:53Z"
-    mac: ENC[AES256_GCM,data:EEuPQ1n7qAab7xkQYNt4rxzNy+u6PSYn+hFUTv3ZwmKbMHW7zQY3BgkwalDVrTC4ZNOLy3tGjVDB6v0KXmwXiXwvYL0Y17h8zRiU4id+zQl+oeZTMCFoUZ5Piz69DxO06cMaZF7+6K+9uQ0JLkZsnY3hb82xKAKbl9E/MFdvr6s=,iv:ldIAz5IKNFnbvcNpzo9qX6n0evix7tsLcTPiouB8lfk=,tag:KXDYrz4vfXkWH0cUHsOUdw==,type:str]
+    mac: ENC[AES256_GCM,data:xpMKA/p5pEtYSjOZFwznuKO2eUE+xXNzV8Yggxt0IgzLiNSiGuHvFRaLihrgdT6/HVVEYxAbYjtnDHBY0+mVulk+p9TRUO7Z3CtlzEKfW23lpkExeN0CblWwEIh+xL86ip++BAGsI7n2LSgjVUgqInmYgi3oYAyv9wwc0+ajZzo=,iv:noqbjzz0nEUPQ/3p0rK3dHyKthz4Hm3w0GLRlya3zag=,tag:rFvLcn7QBApQs+qPrjRvKw==,type:str]
-    pgp: []
+    encrypted_regex: ^(data|stringData)$
-    unencrypted_suffix: _unencrypted
+    version: 3.11.0
    version: 3.9.4