Data service needs to reach Helius WSS, Solana RPC, Jupiter API,
Hyperliquid — all external. Network policy had no egress rule for
external traffic, so all API/websocket connections were blocked.
Data-service pods use NetworkPolicy egress selector app: trading-kafka
but Strimzi Kafka pods are labeled strimzi.io/name=trading-kafka.
This silently blocked all Kafka traffic causing NoBrokersAvailable.
- Remove helius/jupiter placeholders from trading-secrets.yaml
- Create new solana-api-keys.yaml with helius-api-key and jupiter-api-key
- Update kustomization.yaml to include solana-api-keys.yaml
Convention: each secret group gets its own SOPS-encrypted file
instead of adding new keys to an already encrypted file.
- Replace Gemini API and A100 vLLM references with central rtx6000-brain vLLM
endpoint (http://rtx6000-brain-service.customer1.svc.cluster.local:8000/v1)
- Update all ConfigMaps to use model edp1096/Huihui-Qwen3.6-27B-abliterated-FP8
- Add LLM_API_KEY: sk-dummy to all ConfigMaps
- Remove ollama sidecars, initContainers, PVC volumes from all analyst CronJobs
- Remove GPU nodeSelector, tolerations, and GPU resource requests from CronJobs
- Clean up kustomization.yaml: remove ollama-storage.yaml, deepseek-storage.yaml,
gemini-apikey.yaml references
- Activate deepseek-summarizer.yaml in kustomization, keep others commented
- Create solana-ingester Deployment + Service in trading-platform/
(uses same data-service image, runs WebSocket ingester mode)
- Add solana-ingester ConfigMap (HELIUS_RPC_ENDPOINT, JUPITER_API_URL,
KAFKA_BROKER, KAFKA_TOPIC)
- Add helius-api-key and jupiter-api-key placeholders to trading-secrets.yaml
(TODO: replace with real values and re-encrypt with SOPS)
- Update kustomization to include new resources
Kraft mode fails with AccessDeniedException on meta.properties
because PVC mounts with root ownership. Adding fsGroup: 1000
ensures the volume group matches the Kafka container user.
- Removed template.pod.brokers (not valid in Strimzi 1.0.0 Kafka schema)
- Moved broker resources to KafkaNodePool.spec.resources
- Added controller role to KafkaNodePool (KRaft requires both controller+broker)
strimzi.io/install/1.0.0 is not a valid git repo or kustomize remote URL.
Download the full install bundle (strimzi-1.0.0.yaml) and reference it
locally in the strimzi kustomization.
Split Strimzi operator deployment into its own Flux Kustomization so
CRDs are applied before the Kafka CR in customer1.
Chain: infrastructure-controllers -> customer1-strimzi -> customer1
customer1 now has force: true to skip dry-run validation when CRDs are
recently applied but not yet visible to the APIServer cache.
Flux dry-run fails when CRDs are in the same kustomization as resources
that reference them - CRDs need to exist before validation.
Two kustomizations now:
strimzi/ - Strimzi 1.0.0 CRDs + operator
siriusdevops-db - DBs, Kafka CR (depends on strimzi CRDs)
Wire them in Flux:
Kustomization 'siriusdevops-strimzi' -> path: strimzi/
Kustomization 'siriusdevops-db' -> path: siriusdevops-db/
dependsOn: [{name: siriusdevops-strimzi}]
Reference Strimzi 1.0.0 install bundle directly from kustomization.
No separate install step needed — kubectl apply -k handles CRDs + operator
in one shot before the Kafka CR.
Pinned to 1.0.0 for reproducible deployments.
/etc/kafka doesn't exist in the apache/kafka image - subPath could not
create the parent directory. Switch to /opt/kafka/config/ which exists
in the image, avoiding both the directory conflict and the missing path.
Container fails with 'read-only file system' when K8s tries to mount
a secret into /etc/kafka/secrets alongside the ConfigMap at /etc/kafka.
Switch to subPath mount of server.properties directly.
Trading services depend on Kafka (trading-kafka.customer1.svc.cluster.local:9092)
but it was never deployed — lived under hermes-db kustomization which isn't
applied. Move to siriusdevops-db where trading infra belongs.
Also remove stale redis-cluster.yaml reference from hermes-db (file missing).
Migrate trading platform DB egress rule from hermes-pgdb to
siriusdevops-pgdb to complete the trading DB migration. The
configmaps already point to siriusdevops-pgdb-rw but the
network policy still allowed egress to the old cluster.
- news-app-db.yaml: point Database CR to siriusdevops-pgdb
- All ConfigMaps: DB_HOST=siriusdevops-pgdb-rw, LLM_BASE_URL for vLLM
- hermes-pgdb now dedicated to pgvector/RAG for agents only
- Aligns with CNPG extension pattern and active kanban (devops startupProbes, backend exec-service)
PR to master per repo rules.
- Add build-pgvector-image.yml GH Actions workflow for GCR
- Set imageName to gcr.io/devops-lab-cluster/postgres-pgvector:15.2-0.8.0
- Image change only (CNPG rule: no config changes in same update)
- Add 'trading' managed role to siriusdevops-pgdb cluster
- Move trading-db-credentials from hermes-db to siriusdevops-db
- Create trading-data-db.yaml (Database kind) pointing to siriusdevops-pgdb
- Update DB_HOST in all 4 trading-platform configmaps to siriusdevops-pgdb-rw
- Remove trading-data-db.yaml and trading-db-credentials.yaml from hermes-db
- Update siriusdevops-db kustomization.yaml to include new resources
Short hostname "trade" (matches /trade HTTPRoute) instead of "trading-dashboard".
This fixes the "Could not resolve host: trading-dashboard" error on tailnet devices while keeping the same annotation style as Grafana.
The /health/ready endpoint calls await initialize() on Hyperliquid and Solana executors. Without explicit timeout, the default 1s probe timeout caused repeated failures even though the app was running.
Added timeoutSeconds: 10 + reduced readiness failureThreshold.
This is the final piece for the "running but not ready" state.
Next.js standalone mode needs a writable /tmp directory for cache, temp files, and logs.
With , the container was crashing after "Ready in 228ms" (EOF in log stream).
Added the standard emptyDir volumeMount used by other services.
This should make the dashboard stable and accessible on tailnet.
The service was 'running but not ready' because:
- startupProbe used /health (always-ok) with only failureThreshold: 6 (~30s grace)
- Executor initialization (Hyperliquid SDK network calls + Solana client) takes longer
Now uses /health/ready (checks actual executor state) with failureThreshold: 30 (~5min grace), matching the pattern used for data-service and dashboard.
Also updated Helm template for consistency.
Matches the exact annotation pattern used for other tailnet services (Grafana, trade-dashboard, hermes-webui, etc.).
This ensures the Tailscale operator correctly maps the exposed port after the 3000 change.
