Kraft mode fails with AccessDeniedException on meta.properties
because PVC mounts with root ownership. Adding fsGroup: 1000
ensures the volume group matches the Kafka container user.
- Removed template.pod.brokers (not valid in Strimzi 1.0.0 Kafka schema)
- Moved broker resources to KafkaNodePool.spec.resources
- Added controller role to KafkaNodePool (KRaft requires both controller+broker)
strimzi.io/install/1.0.0 is not a valid git repo or kustomize remote URL.
Download the full install bundle (strimzi-1.0.0.yaml) and reference it
locally in the strimzi kustomization.
Flux dry-run fails when CRDs are in the same kustomization as resources
that reference them - CRDs need to exist before validation.
Two kustomizations now:
strimzi/ - Strimzi 1.0.0 CRDs + operator
siriusdevops-db - DBs, Kafka CR (depends on strimzi CRDs)
Wire them in Flux:
Kustomization 'siriusdevops-strimzi' -> path: strimzi/
Kustomization 'siriusdevops-db' -> path: siriusdevops-db/
dependsOn: [{name: siriusdevops-strimzi}]
Reference Strimzi 1.0.0 install bundle directly from kustomization.
No separate install step needed — kubectl apply -k handles CRDs + operator
in one shot before the Kafka CR.
Pinned to 1.0.0 for reproducible deployments.
/etc/kafka doesn't exist in the apache/kafka image - subPath could not
create the parent directory. Switch to /opt/kafka/config/ which exists
in the image, avoiding both the directory conflict and the missing path.
Container fails with 'read-only file system' when K8s tries to mount
a secret into /etc/kafka/secrets alongside the ConfigMap at /etc/kafka.
Switch to subPath mount of server.properties directly.
Trading services depend on Kafka (trading-kafka.customer1.svc.cluster.local:9092)
but it was never deployed — lived under hermes-db kustomization which isn't
applied. Move to siriusdevops-db where trading infra belongs.
Also remove stale redis-cluster.yaml reference from hermes-db (file missing).
Migrate trading platform DB egress rule from hermes-pgdb to
siriusdevops-pgdb to complete the trading DB migration. The
configmaps already point to siriusdevops-pgdb-rw but the
network policy still allowed egress to the old cluster.
- news-app-db.yaml: point Database CR to siriusdevops-pgdb
- All ConfigMaps: DB_HOST=siriusdevops-pgdb-rw, LLM_BASE_URL for vLLM
- hermes-pgdb now dedicated to pgvector/RAG for agents only
- Aligns with CNPG extension pattern and active kanban (devops startupProbes, backend exec-service)
PR to master per repo rules.
- Add build-pgvector-image.yml GH Actions workflow for GCR
- Set imageName to gcr.io/devops-lab-cluster/postgres-pgvector:15.2-0.8.0
- Image change only (CNPG rule: no config changes in same update)
- Add 'trading' managed role to siriusdevops-pgdb cluster
- Move trading-db-credentials from hermes-db to siriusdevops-db
- Create trading-data-db.yaml (Database kind) pointing to siriusdevops-pgdb
- Update DB_HOST in all 4 trading-platform configmaps to siriusdevops-pgdb-rw
- Remove trading-data-db.yaml and trading-db-credentials.yaml from hermes-db
- Update siriusdevops-db kustomization.yaml to include new resources
Short hostname "trade" (matches /trade HTTPRoute) instead of "trading-dashboard".
This fixes the "Could not resolve host: trading-dashboard" error on tailnet devices while keeping the same annotation style as Grafana.
The /health/ready endpoint calls await initialize() on Hyperliquid and Solana executors. Without explicit timeout, the default 1s probe timeout caused repeated failures even though the app was running.
Added timeoutSeconds: 10 + reduced readiness failureThreshold.
This is the final piece for the "running but not ready" state.
Next.js standalone mode needs a writable /tmp directory for cache, temp files, and logs.
With , the container was crashing after "Ready in 228ms" (EOF in log stream).
Added the standard emptyDir volumeMount used by other services.
This should make the dashboard stable and accessible on tailnet.
The service was 'running but not ready' because:
- startupProbe used /health (always-ok) with only failureThreshold: 6 (~30s grace)
- Executor initialization (Hyperliquid SDK network calls + Solana client) takes longer
Now uses /health/ready (checks actual executor state) with failureThreshold: 30 (~5min grace), matching the pattern used for data-service and dashboard.
Also updated Helm template for consistency.
Matches the exact annotation pattern used for other tailnet services (Grafana, trade-dashboard, hermes-webui, etc.).
This ensures the Tailscale operator correctly maps the exposed port after the 3000 change.
Gives ~5min grace period for heavy lifespan init (Redis, DB, executors, model loading).
Addresses 'up but not ready' state on data-service and dashboard (0/1 Ready).
Live manifests in apps/base/customer1/... updated (source of truth). Helm templates can be synced later.
See gke-hermes-infrastructure skill (trading-startup-probe-fix-2026-05-26.md).
Previous PR #139 added the secret to trading-platform/deploy/k8s/base/
but the live cluster uses apps/base/customer1/trading-platform/ kustomize.
- Create execute-service-jwt-secret.yaml (SOPS encrypted) in secrets/
- Add to secrets/ kustomization.yaml resources
- Wire JWT_SECRET_KEY env var in execute-service deployment
- Points to execute-service-jwt-secret via secretKeyRef
- Remove imageName (was never applied, live cluster uses default CNPG image)
- Remove maintenance_work_mem (live cluster has no custom parameters)
- Set postgresql: {} to match live state
- Keep memory role superuser: true (the only needed change)
Fixes CNPG admission webhook: 'Can't change image name and configuration at the same time'
CNPG version on cluster does not support .spec.sql on Database CR.
The rag-init-job.yaml already handles RAG schema initialization separately.
Unblocks Flux dry-run for agent-memory-rag.
- Add allowedExtensions: ['*'] so roles can CREATE EXTENSION
- Grant superuser to 'memory' role (was createdb/createrole, insufficient)
- Needed for mem0 plugin pgvector backend on agent_memory DB
Unblocks kanban task t_3d94b392
