Flux dry-run fails when CRDs are in the same kustomization as resources
that reference them - CRDs need to exist before validation.
Two kustomizations now:
strimzi/ - Strimzi 1.0.0 CRDs + operator
siriusdevops-db - DBs, Kafka CR (depends on strimzi CRDs)
Wire them in Flux:
Kustomization 'siriusdevops-strimzi' -> path: strimzi/
Kustomization 'siriusdevops-db' -> path: siriusdevops-db/
dependsOn: [{name: siriusdevops-strimzi}]
Reference Strimzi 1.0.0 install bundle directly from kustomization.
No separate install step needed — kubectl apply -k handles CRDs + operator
in one shot before the Kafka CR.
Pinned to 1.0.0 for reproducible deployments.
/etc/kafka doesn't exist in the apache/kafka image - subPath could not
create the parent directory. Switch to /opt/kafka/config/ which exists
in the image, avoiding both the directory conflict and the missing path.
Container fails with 'read-only file system' when K8s tries to mount
a secret into /etc/kafka/secrets alongside the ConfigMap at /etc/kafka.
Switch to subPath mount of server.properties directly.
Trading services depend on Kafka (trading-kafka.customer1.svc.cluster.local:9092)
but it was never deployed — lived under hermes-db kustomization which isn't
applied. Move to siriusdevops-db where trading infra belongs.
Also remove stale redis-cluster.yaml reference from hermes-db (file missing).
Migrate trading platform DB egress rule from hermes-pgdb to
siriusdevops-pgdb to complete the trading DB migration. The
configmaps already point to siriusdevops-pgdb-rw but the
network policy still allowed egress to the old cluster.
- news-app-db.yaml: point Database CR to siriusdevops-pgdb
- All ConfigMaps: DB_HOST=siriusdevops-pgdb-rw, LLM_BASE_URL for vLLM
- hermes-pgdb now dedicated to pgvector/RAG for agents only
- Aligns with CNPG extension pattern and active kanban (devops startupProbes, backend exec-service)
PR to master per repo rules.
- Add build-pgvector-image.yml GH Actions workflow for GCR
- Set imageName to gcr.io/devops-lab-cluster/postgres-pgvector:15.2-0.8.0
- Image change only (CNPG rule: no config changes in same update)
- Add 'trading' managed role to siriusdevops-pgdb cluster
- Move trading-db-credentials from hermes-db to siriusdevops-db
- Create trading-data-db.yaml (Database kind) pointing to siriusdevops-pgdb
- Update DB_HOST in all 4 trading-platform configmaps to siriusdevops-pgdb-rw
- Remove trading-data-db.yaml and trading-db-credentials.yaml from hermes-db
- Update siriusdevops-db kustomization.yaml to include new resources
Short hostname "trade" (matches /trade HTTPRoute) instead of "trading-dashboard".
This fixes the "Could not resolve host: trading-dashboard" error on tailnet devices while keeping the same annotation style as Grafana.
The /health/ready endpoint calls await initialize() on Hyperliquid and Solana executors. Without explicit timeout, the default 1s probe timeout caused repeated failures even though the app was running.
Added timeoutSeconds: 10 + reduced readiness failureThreshold.
This is the final piece for the "running but not ready" state.
Next.js standalone mode needs a writable /tmp directory for cache, temp files, and logs.
With , the container was crashing after "Ready in 228ms" (EOF in log stream).
Added the standard emptyDir volumeMount used by other services.
This should make the dashboard stable and accessible on tailnet.
The service was 'running but not ready' because:
- startupProbe used /health (always-ok) with only failureThreshold: 6 (~30s grace)
- Executor initialization (Hyperliquid SDK network calls + Solana client) takes longer
Now uses /health/ready (checks actual executor state) with failureThreshold: 30 (~5min grace), matching the pattern used for data-service and dashboard.
Also updated Helm template for consistency.
Matches the exact annotation pattern used for other tailnet services (Grafana, trade-dashboard, hermes-webui, etc.).
This ensures the Tailscale operator correctly maps the exposed port after the 3000 change.
Gives ~5min grace period for heavy lifespan init (Redis, DB, executors, model loading).
Addresses 'up but not ready' state on data-service and dashboard (0/1 Ready).
Live manifests in apps/base/customer1/... updated (source of truth). Helm templates can be synced later.
See gke-hermes-infrastructure skill (trading-startup-probe-fix-2026-05-26.md).
Previous PR #139 added the secret to trading-platform/deploy/k8s/base/
but the live cluster uses apps/base/customer1/trading-platform/ kustomize.
- Create execute-service-jwt-secret.yaml (SOPS encrypted) in secrets/
- Add to secrets/ kustomization.yaml resources
- Wire JWT_SECRET_KEY env var in execute-service deployment
- Points to execute-service-jwt-secret via secretKeyRef
- Remove imageName (was never applied, live cluster uses default CNPG image)
- Remove maintenance_work_mem (live cluster has no custom parameters)
- Set postgresql: {} to match live state
- Keep memory role superuser: true (the only needed change)
Fixes CNPG admission webhook: 'Can't change image name and configuration at the same time'
CNPG version on cluster does not support .spec.sql on Database CR.
The rag-init-job.yaml already handles RAG schema initialization separately.
Unblocks Flux dry-run for agent-memory-rag.
- Add allowedExtensions: ['*'] so roles can CREATE EXTENSION
- Grant superuser to 'memory' role (was createdb/createrole, insufficient)
- Needed for mem0 plugin pgvector backend on agent_memory DB
Unblocks kanban task t_3d94b392
- Add trading_dashboard role + Database CR on siriusdevops-pgdb
- Create SOPS-encrypted credentials secret (template)
- Update trade-dashboard to use the new dedicated DB
- New host: siriusdevops-pgdb-rw
- New database: trading_dashboard
- New credentials secret
- Add REDIS_URL env var using variable expansion from the existing
trading-data-service-config ConfigMap (REDIS_HOST + REDIS_PORT).
- This resolves the 'connecting to localhost:6379' crash on startup.
- Matches the actual infrastructure configuration in customer1.
