sirius/gcloud-lab
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix secret issue

Browse source
This commit is contained in:
sirius0xdev 2026-05-22 02:11:22 +00:00
parent 84e85d1432
commit 75cea5abb7
3 changed files with 19 additions and 672 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

303
analyses/telegram-webhook-container-analysis.md
View file

@ -1,303 +0,0 @@
# Telegram Webhook Failure Analysis - Deep Dive (Container-Level)
**Date:** 2026-05-10
**Repo:** https://github.com/sirius0xdev/gcloud-lab
**Cluster:** devops-lab (GKE), Namespace: customer1
**Webhook URL:** https://ws.siriusdevops.com/telegram/webhook/default
---
## Status Update
**TLS certificates ARE provisioned** in GCP admin console. The Gateway certmap annotation is working.
The real issues are in the **container configuration** and **missing probes**.
---
## Root Cause #1: NO Liveness/Readiness Probes (CONFIRMED CRITICAL)
**The hermes-agent deployment has ZERO probes defined.**
This means:
- Kubernetes marks pods "Ready" immediately after container start
- The Gateway routes webhook traffic before the process has bound to port 9118
- During restarts, there is no graceful drain
- **A crashed or hung pod stays in the endpoint list forever**
### The /health endpoint problem
The Telegram webhook server runs on **port 9118** via python-telegram-bot's `start_webhook()`. This internally starts an aiohttp server that **ONLY registers the webhook path** (`/telegram/webhook/default`). It does NOT expose a `/health` endpoint.
So a probe like this would FAIL:
```yaml
# THIS WON'T WORK - port 9118 has no /health
readinessProbe:
httpGet:
path: /health
port: 9118
```
### What DOES have a health endpoint?
The **generic webhook adapter** on **port 8644** IS a Hermes-managed server. From the source code, it exposes `/health`. This IS enabled via `WEBHOOK_ENABLED: "true"`.
### The Fix
Add probes targeting port 8644 (the generic webhook server that IS healthy when the gateway is running):
```yaml
readinessProbe:
httpGet:
path: /health
port: 8644
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /health
port: 8644
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3
```
Alternatively, enable the API server on port 8642 and probe there (it also has `/health`).
---
## Root Cause #2: NO Resource Limits (HIGH)
The **new deployment dropped all resource limits** that existed in the old deployment.
**Old deployment (`deployment.yaml`):**
```yaml
resources:
requests:
memory: 2Gi
cpu: "1"
limits:
memory: 3Gi
cpu: "2"
```
**New deployment (`new-deployment.yaml`):**
```yaml
# NO resources block for the hermes-agent container
```
Only the `hermes-webui` sidecar has limits (500Mi-1Gi memory, 100m-500m CPU).
### Impact
- The hermes-agent container can consume unbounded memory
- With `agent.max_turns: 90` and `gateway_timeout: 1800` (30 min), a single agent run can eat massive memory
- Pod may be OOMKilled by the node, but K8s won't restart it gracefully without probes
- **This is a likely cause of intermittent failures**
### Fix
Add resource limits back:
```yaml
resources:
requests:
memory: 2Gi
cpu: "1"
limits:
memory: 4Gi
cpu: "2"
```
---
## Root Cause #3: Path Handling Concern (MEDIUM)
**Source code** (`/opt/hermes/gateway/platforms/telegram.py:1213`):
```python
webhook_path = urlparse(webhook_url).path or "/telegram"
```
For `TELEGRAM_WEBHOOK_URL=https://ws.siriusdevops.com/telegram/webhook/default`:
- `webhook_path` = `/telegram/webhook/default`
- PTB's aiohttp server registers a handler at exactly `/telegram/webhook/default` on port 9118
**HTTPRoute** (`hermes-webhook.yaml`):
```yaml
rules:
- matches:
- path:
type: PathPrefix
value: /telegram/webhook
backendRefs:
- name: http-tele-webhook
port: 9118
```
**The question:** Does the GKE Gateway strip the `/telegram/webhook` prefix before forwarding?
- If it does NOT strip: Pod receives `/telegram/webhook/default` -> **OK**
- If it DOES strip to `/default`: Pod receives `/default` -> **404**
- If it strips to `/`: Pod receives `/` -> **404**
Most GatewayAPI implementations pass the full original path by default, but some ingress controllers strip the matched prefix. **Verify this on the running cluster.**
---
## Root Cause #4: API Server Not Enabled (MEDIUM)
The old deployment had:
```yaml
- name: API_SERVER_ENABLED
value: "true"
- name: API_SERVER_HOST
value: "0.0.0.0"
- name: API_SERVER_PORT
value: "8642"
- name: API_SERVER_KEY
valueFrom: {secretKeyRef: ...}
- name: API_SERVER_MODEL_NAME
value: "hermes-agent"
```
**These are ALL absent from `new-deployment.yaml`.** Port 8642 is declared as a containerPort but nothing listens on it because `API_SERVER_ENABLED` is not set.
The API server exposes `/health` and `/health/detailed` endpoints. Without it, you lose a convenient health check and the API server interface.
---
## Other Findings
### 5. TELEGRAM_WEBHOOK_ENABLED is Ignored (INFO)
The code only checks if `TELEGRAM_WEBHOOK_URL` is set (non-empty). It does NOT read `TELEGRAM_WEBHOOK_ENABLED`. The env var in the deployment is a dead config — set but ignored.
### 6. vLLM Service Name Mismatch (INFO)
```yaml
OPENAI_BASE_URL: "http://openclaw-brain-service.customer1.svc.cluster.local:8000/v1"
```
But the active vLLM deployment (`rtx6000-vllm.yaml`) creates a service named `rtx6000-brain-service`. If Hermes ever switches from xAI to the openai provider, local vLLM is unreachable.
### 7. SOPS Secrets (VERIFY)
Both `hermes-secret.yaml` and `tele-webhook.yaml` are SOPS-encrypted. Verify they are decrypted in the cluster:
```bash
kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d
kubectl get secret telegram-webhook -n customer1 -o jsonpath='{.data.TELEGRAM_WEBHOOK_SECRET}' | base64 -d
```
### 8. Container Env Vars Summary
| Variable | Value | Notes |
|----------|-------|-------|
| TELEGRAM_WEBHOOK_URL | https://ws.siriusdevops.com/telegram/webhook/default | OK |
| TELEGRAM_WEBHOOK_PORT | 9118 | OK |
| TELEGRAM_WEBHOOK_SECRET | From SOPS secret | Verify decrypted |
| TELEGRAM_WEBHOOK_ENABLED | "true" | **Ignored by code** |
| TELEGRAM_BOT_TOKEN | From SOPS secret | Verify decrypted |
| TELEGRAM_ALLOWED_USERS | 7528130947 | OK |
| WEBHOOK_ENABLED | "true" | OK (enables port 8644 /health) |
| WEBHOOK_PORT | 8644 | OK |
| API_SERVER_ENABLED | **NOT SET** | Port 8642 has no listener |
| HERMES_MODEL_PROVIDER | xai | OK (uses Grok) |
| HERMES_MODEL | grok-4.20-0309-reasoning | OK |
---
## Recommended Fix (Priority Order)
### P0 - Add Probes (will detect and restart hung/crashed pods)
Add to `new-deployment.yaml` under the hermes-agent container spec:
```yaml
readinessProbe:
httpGet:
path: /health
port: 8644
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /health
port: 8644
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3
```
### P1 - Add Resource Limits (prevents OOM kills)
```yaml
resources:
requests:
memory: 2Gi
cpu: "1"
limits:
memory: 4Gi
cpu: "2"
```
### P2 - Verify Path Handling
Test if the Gateway passes the full path:
```bash
# From inside the pod, check what the webhook server receives
kubectl exec -n customer1 deploy/hermes-agent -- curl -s http://localhost:9118/telegram/webhook/default -X POST -H "Content-Type: application/json" -d '{}'
```
### P3 - Enable API Server (optional, gives /health on 8642)
Add back the API_SERVER_ENABLED env var if you want the API server health endpoint.
---
## Diagnostic Commands
Run these on the cluster RIGHT NOW to confirm the current state:
```bash
# 1. Check pod status and restart count
kubectl get pods -n customer1 -l app=hermes-agent -o wide
# 2. Check events for OOMKilled or probe failures
kubectl describe pod -n customer1 -l app=hermes-agent | grep -A5 -i 'oom\|probe\|restart'
# 3. Check if the webhook port is actually listening
kubectl exec -n customer1 deploy/hermes-agent -- ss -tlnp | grep 9118
# 4. Check if port 8644 health endpoint works
kubectl exec -n customer1 deploy/hermes-agent -- curl -s http://localhost:8644/health
# 5. Check logs for webhook startup messages
kubectl logs -n customer1 deploy/hermes-agent --tail=50 | grep -i 'webhook\|listening\|9118'
# 6. Verify SOPS secrets are decrypted
kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d && echo
kubectl get secret telegram-webhook -n customer1 -o jsonpath='{.data.TELEGRAM_WEBHOOK_SECRET}' | base64 -d && echo
# 7. Check if the webhook is registered with Telegram (using the pod's bot token)
BOT_TOKEN=$(kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d)
curl -s "https://api.telegram.org/bot${BOT_TOKEN}/getWebhookInfo" | python3 -m json.tool
# 8. Test the full path through the Gateway
curl -v https://ws.siriusdevops.com/telegram/webhook/default -X POST -H "Content-Type: application/json" -d '{}' 2>&1
# 9. Check container memory usage
kubectl top pod -n customer1 -l app=hermes-agent 2>/dev/null || echo "metrics-server not available"
# 10. Check resource limits on the container
kubectl get pod -n customer1 -l app=hermes-agent -o jsonpath='{.items[0].spec.containers[0].resources}'
```
---
## Files to Modify
1. `apps/base/customer1/hermes-agent/new-deployment.yaml` - Add probes + resource limits
2. Optionally: Re-enable API server env vars in `new-deployment.yaml`

345
analyses/telegram-webhook-failure-analysis.md
View file

@ -1,345 +0,0 @@
# Telegram Webhook Failure Analysis - Hermes Agent on GKE
**Date:** 2026-05-10
**Repo:** https://github.com/sirius0xdev/gcloud-lab
**Cluster:** devops-lab (GKE)
**Namespace:** customer1
**Webhook URL:** https://ws.siriusdevops.com/telegram/webhook/default
---
## Executive Summary
The Telegram webhook is failing because **there is no valid TLS certificate for `ws.siriusdevops.com`** at the Gateway layer. Telegram strictly requires HTTPS with a publicly-trusted certificate for webhook delivery. The Gateway listener declares `protocol: HTTPS` but has no `tls.certificateRefs` and relies on a GKE `CertMap` annotation referencing `gateway-cert-map` — a resource that **does not exist** in the repository. Cert-manager is configured but disconnected from the GatewayAPI setup (HTTP-01 solver points to a non-existent Traefik ingress class).
**TL;DR:** Telegram tries to POST to `https://ws.siriusdevops.com/...`, but the Gateway has no certificate to present during the TLS handshake. Connection fails before it ever reaches the hermes-agent pod.
---
## Root Cause #1: Missing TLS Certificate (CRITICAL)
### What's happening
**File:** `infrastructure/gatewayapi/apigateway.yaml`
```yaml
listeners:
- name: https
protocol: HTTPS
port: 443
allowedRoutes:
namespaces:
from: All
```
The listener says HTTPS but has **no `tls:` block**. It relies entirely on this annotation:
```yaml
annotations:
networking.gke.io/certmap: gateway-cert-map
```
### The problem
- **No `CertMap` or `CertMapEntry` resource** exists anywhere in the repository for `gateway-cert-map`
- Without it, GKE has no managed certificate to attach to the Gateway
- Telegram's webhook delivery gets a TLS handshake failure or no certificate
### Cert-manager is also broken
**File:** `infrastructure/controllers/base/certmanager/clusterissuer.yaml`
```yaml
solvers:
- http01:
ingress:
class: traefik
```
- HTTP-01 solver references `class: traefik`, but **no Traefik ingress controller exists** in the cluster
- **No `Certificate` CRs** exist for `ws.siriusdevops.com` or any other domain
- Cert-manager is completely disconnected from the GatewayAPI setup
### How to fix (choose ONE approach)
**Option A: GKE Managed Certificates (recommended for GatewayAPI)**
Create a `ManagedCertificate` + `BackendConfig` or `CertMap`/`CertMapEntry`:
```yaml
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
name: hermes-webhook-cert
namespace: customer1
spec:
domains:
- ws.siriusdevops.com
- brain.siriusdevops.com
- paaas.siriusdevops.com
```
Then add `tls.certificateRefs` to the Gateway listener:
```yaml
listeners:
- name: https
protocol: HTTPS
port: 443
tls:
certificateRefs:
- name: hermes-webhook-cert
group: networking.gke.io
```
**Option B: Fix Cert-manager with DNS-01**
Switch ClusterIssuer from HTTP-01/Traefik to DNS-01 (e.g., Cloudflare, GCP DNS, or Route53), then create `Certificate` resources for each domain.
---
## Root Cause #2: No Readiness/Liveness Probes (HIGH)
**File:** `apps/base/customer1/hermes-agent/new-deployment.yaml`
No readiness or liveness probes are defined on any hermes-agent deployment.
### Impact
- Pods are marked "Ready" immediately after container start
- Gateway routes traffic to the webhook port (9118) before the process has bound to it
- During restarts, traffic hits pods that haven't initialized
### Fix
Add probes to the deployment:
```yaml
readinessProbe:
httpGet:
path: /health
port: 9118
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
httpGet:
path: /health
port: 9118
initialDelaySeconds: 15
periodSeconds: 30
```
---
## Root Cause #3: SOPS Encrypted Secrets (HIGH)
**Files:**
- `apps/base/customer1/hermes-agent/hermes-secret.yaml` (SOPS encrypted)
- `apps/base/customer1/hermes-agent/tele-webhook.yaml` (SOPS encrypted)
These contain `TELEGRAM_BOT_TOKEN` and `TELEGRAM_WEBHOOK_SECRET`.
### Risk
- If the deployment pipeline (Flux/Kustomize controller) is **not decrypting SOPS secrets**, pods receive literal `ENC[...]` strings
- The bot token would be invalid, so even if TLS worked, Telegram auth would fail
- The webhook secret would not match, causing Telegram to reject payloads
### Verify
Run `kubectl get secret hermes-secrets -n customer1 -o yaml` and check if values are base64-encoded real tokens or `ENC[...]` strings.
---
## Secondary Issues
### 3a. Service Name Mismatch (vLLM Integration)
**File:** `apps/base/customer1/hermes-agent/new-deployment.yaml`
```yaml
OPENAI_BASE_URL: "http://openclaw-brain-service.customer1.svc.cluster.local:8000/v1"
```
But the active vLLM deployment (`rtx6000-vllm.yaml`) creates a service named **`rtx6000-brain-service`**.
- `OPENAI_BASE_URL` points to `openclaw-brain-service` which may not exist
- If Hermes ever switches to the `openai` provider (instead of `xai`), local vLLM is unreachable
- The model provider defaults to `HERMES_MODEL_PROVIDER: xai` (Grok external API)
### 3b. Duplicate HTTPRoute Deployment
The webhook HTTPRoute (`hermes-webhook.yaml`) is included in **two** kustomization trees:
1. `infrastructure/gatewayapi/gateway-routes/kustomization.yaml` -> deployed via Flux
2. `apps/base/customer1/hermes-agent/kustomization.yaml` -> deployed via Flux
Same resource (`http-telegram-webhook` in `customer1`) from two sources. This may cause Flux reconciliation conflicts.
### 3c. Empty HF_TOKEN in vLLM Deployments
All vLLM deployments have:
```yaml
- name: HF_TOKEN
value: ""
```
If the model `edp1096/Huihui-Qwen3.6-27B-abliterated-FP8` is a gated model on HuggingFace, it will fail to download.
### 3d. KEDA Scale-to-Zero
**File:** `infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml`
```yaml
minReplicaCount: 0
maxReplicaCount: 1
```
- vLLM scales to **zero** when idle
- First request after cold start incurs full model load time (30-60 seconds)
- For real-time Telegram responses, this causes visible latency
### 3e. PVC Name Collision
Both `rtx6000-vllm.yaml` and `a100-vllm.yaml` define a PVC named `vllm-model-qwen3.6-27b-uncensored` in namespace `customer1`. If both are ever active simultaneously, they conflict.
---
## Architecture Overview
```
Internet
|
v
[GKE External LB]
|
Gateway: external-http-gateway
(port 443/HTTPS, NO TLS cert!)
|
+-----------+-----------+
| | |
ws.siriusdevops.com brain.siriusdevops.com paaas.siriusdevops.com
| | |
v v v
/telegram/webhook / /
| | |
v v v
http-tele-webhook rtx6000- paaas-landing
:9118 brain- :8080
service:8000
|
v
hermes-agent pod
(ports: 8642, 8644, 9118)
|
v
Model Provider: xai (Grok via external API)
Fallback: OPENAI_BASE_URL -> openclaw-brain-service (MISMATCHED)
```
---
## vLLM Server Status
| Server | GPU | Model | Quantization | Status |
|--------|-----|-------|-------------|--------|
| RTX 6000 | 1x Pro 6000 (96GB) | edp1096/Huihui-Qwen3.6-27B-abliterated-FP8 | FP8 | **ACTIVE** |
| A100 | 1x A100 (80GB) | Youssofal/Qwen3.6-27B-Abliterated-Heretic-Uncensored-BF16 | BF16 | Commented out |
| L4 | 1x L4 (24GB) | p-e-w/Qwen3-8B-heretic | auto | Commented out |
| Gemma | 1x A100 (80GB) | coder3101/Qwen3.5-27B-heretic | BF16 | Not in kustomization |
**Note:** The architecture plan (`plans/AI_ARCHITECTURE.md`) describes a dual-tier L4 dispatcher + A100 deep thinker setup, but the active deployment only has RTX 6000.
---
## Action Plan (Priority Order)
### P0 - Fix TLS (will unblock Telegram webhooks)
1. **Create a `CertMap`/`CertMapEntry`** or **`ManagedCertificate`** resource for `ws.siriusdevops.com`
2. **Add `tls.certificateRefs`** to the Gateway listener in `apigateway.yaml`
3. Verify with: `curl -vI https://ws.siriusdevops.com/telegram/webhook/default`
4. If cert is valid, Telegram should start delivering webhooks
### P1 - Verify Secrets
5. Check if SOPS secrets are actually decrypted in the cluster
6. Run: `kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d`
### P2 - Add Probes
7. Add readiness/liveness probes to hermes-agent deployment
8. Redeploy
### P3 - Clean Up GatewayAPI
9. Remove duplicate `hermes-webhook.yaml` reference from one kustomization
10. Fix or remove Traefik-referencing ClusterIssuers
### P4 - Fix vLLM Integration
11. Fix `OPENAI_BASE_URL` service name or update kustomization to create `openclaw-brain-service`
12. Set real `HF_TOKEN` in vLLM deployments
13. Consider setting `minReplicaCount: 1` in KEDA for consistent response times
---
## Files Referenced
### GatewayAPI & TLS
- `infrastructure/gatewayapi/apigateway.yaml` - Gateway definition (missing TLS)
- `infrastructure/gatewayapi/gateway-routes/hermes-webhook.yaml` - Webhook HTTPRoute
- `infrastructure/gatewayapi/gateway-routes/route.yaml` - General route
- `infrastructure/controllers/base/certmanager/clusterissuer.yaml` - Cert-manager (Traefik mismatch)
### Hermes Agent Deployment
- `apps/base/customer1/hermes-agent/new-deployment.yaml` - Active deployment + webhook service
- `apps/base/customer1/hermes-agent/deployment.yaml` - Old deployment (no webhook config)
- `apps/base/customer1/hermes-agent/configmap.yaml` - Hermes config
- `apps/base/customer1/hermes-agent/hermes-secret.yaml` - SOPS encrypted secrets
- `apps/base/customer1/hermes-agent/tele-webhook.yaml` - SOPS encrypted webhook secret
- `apps/base/customer1/hermes-agent/kustomization.yaml` - Kustomize composition
### vLLM Infrastructure
- `infrastructure/gpus/base/vllm-servers/rtx6000-vllm.yaml` - Active vLLM server
- `infrastructure/gpus/base/keda-gpu-scaling/keda-vllm.yaml` - KEDA scaling
- `infrastructure/gpus/base/keda-gpu-scaling/vllm-route.yaml` - vLLM external route
### Architecture Plans
- `plans/AI_ARCHITECTURE.md` - AI architecture plan
- `plans/models-to-try.md` - Models being considered
---
## Quick Diagnostic Commands
Run these on the cluster to confirm findings:
```bash
# 1. Check if TLS cert exists for the webhook domain
curl -vI https://ws.siriusdevops.com/telegram/webhook/default 2>&1 | grep -E 'SSL|certificate|subject|issuer'
# 2. Check Gateway status
kubectl get gateway external-http-gateway -n customer1 -o yaml
# 3. Check HTTPRoute status (attached/programmed)
kubectl get httproute http-telegram-webhook -n customer1 -o yaml
# 4. Verify secrets are decrypted
kubectl get secret hermes-secrets -n customer1 -o jsonpath='{.data.TELEGRAM_BOT_TOKEN}' | base64 -d && echo
# 5. Check if pods are actually ready
kubectl get pods -n customer1 -l app=hermes-agent -o wide
# 6. Check CertMap exists
kubectl get certmap gateway-cert-map -A 2>/dev/null || echo "CertMap NOT FOUND"
# 7. Check managed certificates
kubectl get managedcertificate -A 2>/dev/null || echo "No ManagedCertificates"
# 8. Check cert-manager certificates
kubectl get certificate -A 2>/dev/null || echo "No Certificates"
# 9. Check vLLM pod status
kubectl get pods -n customer1 -l app=rtx6000-brain-vllm -o wide
# 10. Test internal webhook endpoint
kubectl exec -n customer1 deploy/hermes-agent -- curl -s http://localhost:9118/health || echo "Health check failed"
```

43
apps/base/customer1/trading-platform/secrets/trading-secrets.yaml
View file

@ -1,32 +1,27 @@
apiVersion: ENC[AES256_GCM,data:j4g=,iv:Bb3E3dbyD1MUIsthCltT2rRNNorduAL7QmPG8oRy31g=,tag:htxjh/00z6RGQNNyKCUviw==,type:str]
kind: ENC[AES256_GCM,data:k3FRKBzN,iv:Pvtu7bnGWQNl9pkjtU8GpHjpeg00oFXatf3jxxtQEpw=,tag:zj/yAy/DpCfLO7MdH3hiMA==,type:str]
apiVersion: v1
kind: Secret
metadata:
name: ENC[AES256_GCM,data:ZpbqQd2YyEfnugJ27JVAmqBf8Q7m75B0,iv:4njbS+2AYJuxzNERSOjOH8wmoI1KM0ocUT7OgeoGEgo=,tag:spwlHcpeQM0G8O3VPHoDOg==,type:str]
namespace: ENC[AES256_GCM,data:4ErEXcBaI8Yp,iv:IKHgZ6Gm5X21Atnnm2xOFU11IgSfw5X5Wdnl25EDyOI=,tag:znkdsNDkIb4mxBY4yJbX9g==,type:str]
type: ENC[AES256_GCM,data:dd7uKLw+,iv:qRkV8K+ytp55rLGNIP1lG2yZ+LENt/FkdDiWzi/1tik=,tag:aUWQ+ZOQS7/hXcnceCyrTQ==,type:str]
name: trading-platform-secrets
namespace: customer1
type: Opaque
stringData:
#ENC[AES256_GCM,data:Xol7d8ednDll9VKfZ62jZRdARcmz8UJ/6ovDaw8OHrPb72aUdA==,iv:TvGk+LiK0+maCioF5daeWDTNKOSRA5pBFf9PgKL/9Z4=,tag:34kee3TQNsCdIpyEtE12mA==,type:comment]
news-api-key: ENC[AES256_GCM,data:eNMLhs55u3bwVqD4l2tQhgq1+wp/9rCa,iv:7BjlJqgJbg6BqdXxNphnIWKA/LYFZJc3qqJ360/EteY=,tag:hs5vfj+zP9OCq2tDucItIg==,type:str]
market-data-api-key: ENC[AES256_GCM,data:ol3aAC9ijFcFUo7jEVQv7mKjCLpUXBuaVThwRzG/fQ==,iv:YouuIdmj6aK0tuYJtDuiOt91gnUtK2xdshVZxy+gCH4=,tag:MAs8mS/+ndC8AYdI+WdFcA==,type:str]
#ENC[AES256_GCM,data:Si0AKl5/sWrYDNSYiC35iD7vtvP9CzAb,iv:cyBrgR5cFSta+bPdyyCUXrKH68Hi84UbGFqiWskQb7s=,tag:w9+sotMjBTuyfcGYIHfN+w==,type:comment]
service-auth-token: ENC[AES256_GCM,data:NCu/mBKdGIFNXTA5n7M53UXjMaZy9UdzImC90GPU,iv:JPLeNfTDWV3VLbVIUidL6w1IrJN09LTGIyPn0tJlj90=,tag:RIzC8FfGXBmE2PRZzgaAww==,type:str]
#ENC[AES256_GCM,data:X6HnX4CXbHydreDklg2LsCfBN1i1dp7HFifHR4rN2A581dEbYw==,iv:5MocUK+2zYn1qxhayaVnwVR+5FLLdJRKo9uBvW6RiTE=,tag:v13dWH9t/K8Whbs9EXcgnQ==,type:comment]
news-api-key: ENC[AES256_GCM,data:Ren1WNxCGBcaV49M+W5Gl3FjfZJh3o/X,iv:Ly1v+YdGodfeQHLkYIJra+2+qqa06wAcu2RGv6cueRE=,tag:IuWIJB1mzipgmnP5rD2c1A==,type:str]
market-data-api-key: ENC[AES256_GCM,data:2x2vPbOu/T+4albtkeRrktBffX1i06vXQmnn31685g==,iv:LqfpNmK6d9tt2MeaaozI+PVim8KUK8TflEalSxptfjM=,tag:Tan91JSeh5aUhGiExns4PQ==,type:str]
#ENC[AES256_GCM,data:jVRQi+JQNlDUQ+bCe189YrOcsXjYYTGF,iv:SnCbB2w1jnPnXqWGPEivK/b9bWnYA6qL1Xkx6n8XzQ8=,tag:SYJbW0Aea/evPgvGNCqw9Q==,type:comment]
service-auth-token: ENC[AES256_GCM,data:Xrqq82eI5obC86s+FJpi4DbX9+hDdTlYdmfL6DlW,iv:lDhfIuT+On7W6Xk36FKw7fE7zBheRPaMZwQBn7DF5qc=,tag:CdE0OWzQ9OLkSsDL8gq9gA==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age1uuxf066xuuqgvjppxfcmqkwfcufnwp3wcwnl9h20g9k4l8nkw9jsaungf7
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFTzBrN3V5elJmNjNGZ28y
Q1JOQUt5eURqSFZNZkJUNWMrV1NTZEFiYkgwCnlDSTFYTW8zcU02M0NJdXZjYSti
Ylh0bWpJd0k2MWt6VjNNTUlNU0pTUVUKLS0tIENHUEU4VElXbC96bXBGRmo3QXpQ
ZkxxUDRubGt0dnRoQXVtS2xFSnhTRkUKs+rcKiZvgA7mffGo7GkkFL4vWnTIGAIn
RXwlbDNPEhiK+6lh/TgkV2CYXDBt1Hwfk4fzhZknYjY3Psp0ufvY2Q==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBOWHFUcm9aNFlzRXk3bmtY
b3hMRC85a2x3R1lIVmZoTk4xT1JtMDF0MEZFCmhjRGdUYWJpVnFkMTZpTVg4U01Z
c0trekthejd6QUk3REZ2Rms0Y2hyOFUKLS0tIExseng1M0RNU3E2d1NXNlJQVGdh
andDeDdxM2pjU05GL0pKdXIwUzc2M00KBfnTHcj5QMYGlQLRVO34sbtPsjRz37ZJ
LO1q7fngycQGm0oNgZAZnxQNMArlNIUduH3Y8M365XushwPc9BHuQw==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-05-21T04:08:04Z"
mac: ENC[AES256_GCM,data:EEuPQ1n7qAab7xkQYNt4rxzNy+u6PSYn+hFUTv3ZwmKbMHW7zQY3BgkwalDVrTC4ZNOLy3tGjVDB6v0KXmwXiXwvYL0Y17h8zRiU4id+zQl+oeZTMCFoUZ5Piz69DxO06cMaZF7+6K+9uQ0JLkZsnY3hb82xKAKbl9E/MFdvr6s=,iv:ldIAz5IKNFnbvcNpzo9qX6n0evix7tsLcTPiouB8lfk=,tag:KXDYrz4vfXkWH0cUHsOUdw==,type:str]
pgp: []
unencrypted_suffix: _unencrypted
version: 3.9.4
lastmodified: "2026-05-22T02:10:53Z"
mac: ENC[AES256_GCM,data:xpMKA/p5pEtYSjOZFwznuKO2eUE+xXNzV8Yggxt0IgzLiNSiGuHvFRaLihrgdT6/HVVEYxAbYjtnDHBY0+mVulk+p9TRUO7Z3CtlzEKfW23lpkExeN0CblWwEIh+xL86ip++BAGsI7n2LSgjVUgqInmYgi3oYAyv9wwc0+ajZzo=,iv:noqbjzz0nEUPQ/3p0rK3dHyKthz4Hm3w0GLRlya3zag=,tag:rFvLcn7QBApQs+qPrjRvKw==,type:str]
encrypted_regex: ^(data|stringData)$
version: 3.11.0