Commit graph

966 commits

Author SHA1 Message Date
sirius0xdev
c5e2ef16d2 Move Redis cluster to trading-platform and wire into kustomization 2026-05-26 09:52:16 -09:00
sirius0xdev
777f98a5ff
Update kustomization.yaml 2026-05-26 00:10:34 -04:00
sirius0xdev
f9ac2d553b
Merge pull request #175 from sirius0xdev/fix/kafka-fsgroup-permissions
fix(kafka): add fsGroup to pod securityContext for PVC write permissions
2026-05-25 23:34:42 -04:00
Hermes Agent
308d54cc0b fix(kafka): add fsGroup to pod securityContext for PVC write permissions
Kraft mode fails with AccessDeniedException on meta.properties
because PVC mounts with root ownership. Adding fsGroup: 1000
ensures the volume group matches the Kafka container user.
2026-05-26 03:33:43 +00:00
sirius0xdev
0f5488f8e3
Merge pull request #174 from sirius0xdev/fix/kafka-version-4.1.0
bump Kafka version from 3.8.0 to 4.1.0
2026-05-25 23:11:03 -04:00
Hermes Agent
207733f4df bump Kafka version from 3.8.0 to 4.1.0 (supported by Strimzi 1.0.0) 2026-05-26 03:01:40 +00:00
sirius0xdev
472ccc4943
Merge pull request #173 from sirius0xdev/fix/strimzi-add-controller-role
fix trading-kafka: add controller role + move resources to KafkaNodePool
2026-05-25 22:48:44 -04:00
Hermes Agent
f8b97537d7 fix trading-kafka: remove invalid template.pod.brokers, add controller role to KafkaNodePool
- Removed template.pod.brokers (not valid in Strimzi 1.0.0 Kafka schema)
- Moved broker resources to KafkaNodePool.spec.resources
- Added controller role to KafkaNodePool (KRaft requires both controller+broker)
2026-05-26 02:47:43 +00:00
sirius0xdev
506b093377
Merge pull request #172 from sirius0xdev/fix/strimzi-template-fix
fix trading-kafka: move broker resources to KafkaNodePool
2026-05-25 22:29:24 -04:00
Hermes Agent
9eb36d7292 fix trading-kafka: move broker resources to KafkaNodePool 2026-05-26 02:29:04 +00:00
sirius0xdev
631a47cc5c
Merge pull request #171 from sirius0xdev/fix/strimzi-v1-api-v2
bump trading-kafka to Strimzi 1.0.0 v1 API (KafkaNodePool)
2026-05-25 22:19:48 -04:00
Hermes Agent
c477e80790 bump trading-kafka to Strimzi 1.0.0 v1 API (KafkaNodePool) 2026-05-26 02:11:25 +00:00
sirius0xdev
289acdf7d6
Merge pull request #170 from sirius0xdev/fix/strimzi-v1-api
bump trading-kafka API from v1beta2 to v1 for Strimzi 1.0.0
2026-05-25 22:05:46 -04:00
Hermes Agent
b70f0514f4 bump trading-kafka API from v1beta2 to v1 for Strimzi 1.0.0 2026-05-26 02:05:24 +00:00
Hermes Agent
8865b76154 Revert "bump trading-kafka API from v1beta2 to v1 for Strimzi 1.0.0"
This reverts commit cbdea178cf.
2026-05-26 02:05:15 +00:00
Hermes Agent
cbdea178cf bump trading-kafka API from v1beta2 to v1 for Strimzi 1.0.0 2026-05-26 02:02:35 +00:00
sirius0xdev
d21a272b2e
Merge pull request #169 from sirius0xdev/fix/strimzi-vendor-local
vendor Strimzi 1.0.0 YAML locally instead of remote URL
2026-05-25 21:58:59 -04:00
Hermes Agent
7e9b563257 vendor Strimzi 1.0.0 YAML locally instead of remote URL
strimzi.io/install/1.0.0 is not a valid git repo or kustomize remote URL.
Download the full install bundle (strimzi-1.0.0.yaml) and reference it
locally in the strimzi kustomization.
2026-05-26 01:53:06 +00:00
sirius0xdev
b439052079
Merge pull request #168 from sirius0xdev/feat/replace-kafka-with-strimzi
Replace manual Kafka StatefulSet with Strimzi operator
2026-05-25 21:36:41 -04:00
Hermes Agent
d920cc7c63 add customer1-strimzi Flux Kustomization + force:true on customer1
Split Strimzi operator deployment into its own Flux Kustomization so
CRDs are applied before the Kafka CR in customer1.

Chain: infrastructure-controllers -> customer1-strimzi -> customer1

customer1 now has force: true to skip dry-run validation when CRDs are
recently applied but not yet visible to the APIServer cache.
2026-05-26 01:32:57 +00:00
Hermes Agent
47aae85ed3 split Strimzi operator into separate kustomization for Flux dependsOn
Flux dry-run fails when CRDs are in the same kustomization as resources
that reference them - CRDs need to exist before validation.

Two kustomizations now:
  strimzi/        - Strimzi 1.0.0 CRDs + operator
  siriusdevops-db - DBs, Kafka CR (depends on strimzi CRDs)

Wire them in Flux:
  Kustomization 'siriusdevops-strimzi' -> path: strimzi/
  Kustomization 'siriusdevops-db'      -> path: siriusdevops-db/
    dependsOn: [{name: siriusdevops-strimzi}]
2026-05-26 01:32:57 +00:00
Hermes Agent
94f6b04c28 add Strimzi operator as kustomize remote URL (GitOps)
Reference Strimzi 1.0.0 install bundle directly from kustomization.
No separate install step needed — kubectl apply -k handles CRDs + operator
in one shot before the Kafka CR.

Pinned to 1.0.0 for reproducible deployments.
2026-05-26 01:32:57 +00:00
Hermes Agent
25850b5b04 replace manual Kafka StatefulSet with Strimzi operator CR
Remove hand-rolled KRaft StatefulSet that kept failing (mount conflicts,
advertised.listeners issues). Replace with a single Strimzi Kafka CR.

Strimzi handles:
- KRaft setup, storage, scaling
- Broker config & advertised listeners (auto-resolved)
- Topic Operator for proper topic lifecycle
- Bootstrap service at trading-kafka-kafka-bootstrap.customer1.svc.cluster.local:9092

Requires Strimzi 1.0.0 operator installed in the cluster first:
  kubectl apply -f https://strimzi.io/install/latest?namespace=customer1 -n customer1
2026-05-26 01:32:57 +00:00
sirius0xdev
e060e7a108
Merge pull request #167 from sirius0xdev/feat/strimzi-kafka-replacement
Replace manual Kafka StatefulSet with Strimzi operator
2026-05-25 21:15:18 -04:00
Hermes Agent
1d6f33d449 split Strimzi operator into separate kustomization for Flux dependsOn
Flux dry-run fails when CRDs are in the same kustomization as resources
that reference them - CRDs need to exist before validation.

Two kustomizations now:
  strimzi/        - Strimzi 1.0.0 CRDs + operator
  siriusdevops-db - DBs, Kafka CR (depends on strimzi CRDs)

Wire them in Flux:
  Kustomization 'siriusdevops-strimzi' -> path: strimzi/
  Kustomization 'siriusdevops-db'      -> path: siriusdevops-db/
    dependsOn: [{name: siriusdevops-strimzi}]
2026-05-26 01:09:56 +00:00
Hermes Agent
07e4b07608 add Strimzi operator as kustomize remote URL (GitOps)
Reference Strimzi 1.0.0 install bundle directly from kustomization.
No separate install step needed — kubectl apply -k handles CRDs + operator
in one shot before the Kafka CR.

Pinned to 1.0.0 for reproducible deployments.
2026-05-26 00:49:08 +00:00
sirius0xdev
55b2a6a306
Merge pull request #166 from sirius0xdev/feat/strimzi-kafka-replacement
replace manual Kafka StatefulSet with Strimzi operator CR
2026-05-25 20:40:59 -04:00
Hermes Agent
d012cac7f4 replace manual Kafka StatefulSet with Strimzi operator CR
Remove hand-rolled KRaft StatefulSet that kept failing (mount conflicts,
advertised.listeners issues). Replace with a single Strimzi Kafka CR.

Strimzi handles:
- KRaft setup, storage, scaling
- Broker config & advertised listeners (auto-resolved)
- Topic Operator for proper topic lifecycle
- Bootstrap service at trading-kafka-kafka-bootstrap.customer1.svc.cluster.local:9092

Requires Strimzi 1.0.0 operator installed in the cluster first:
  kubectl apply -f https://strimzi.io/install/latest?namespace=customer1 -n customer1
2026-05-26 00:38:57 +00:00
sirius0xdev
572f9f2c0f
Merge pull request #165 from sirius0xdev/fix/kafka-mount-path
fix Kafka config mount: use /opt/kafka/config/ path instead of /etc/k…
2026-05-25 20:17:14 -04:00
Hermes Agent
68033faf4e fix Kafka config mount: use /opt/kafka/config/ path instead of /etc/kafka
/etc/kafka doesn't exist in the apache/kafka image - subPath could not
create the parent directory. Switch to /opt/kafka/config/ which exists
in the image, avoiding both the directory conflict and the missing path.
2026-05-26 00:16:42 +00:00
sirius0xdev
9c90b77210
Merge pull request #164 from sirius0xdev/fix/kafka-config-mount-subpath
fix Kafka config mount: use subPath to avoid /etc/kafka/secrets conflict
2026-05-25 19:55:35 -04:00
Hermes Agent
5ab0098e34 fix Kafka config mount: use subPath to avoid /etc/kafka/secrets conflict
Container fails with 'read-only file system' when K8s tries to mount
a secret into /etc/kafka/secrets alongside the ConfigMap at /etc/kafka.
Switch to subPath mount of server.properties directly.
2026-05-25 23:53:55 +00:00
sirius0xdev
8fc6bc16c3
Merge pull request #163 from sirius0xdev/fix/kafka-env-var-mismatch
fix configmap Kafka env var keys to match app code
2026-05-25 19:42:11 -04:00
Hermes Agent
32a855c457 fix configmap Kafka env var keys to match app code
App code reads KAFKA_BOOTSTRAP_SERVERS (all services), but configmaps
had KAFKA_BROKER (data) and EXECUTE_KAFKA_BOOTSTRAP_SERVERS (execute).
Services silently fell through to hardcoded defaults — Kafka unreachable.

- data-service: KAFKA_BROKER -> KAFKA_BOOTSTRAP_SERVERS
- execute-service: EXECUTE_KAFKA_BOOTSTRAP_SERVERS -> KAFKA_BOOTSTRAP_SERVERS
2026-05-25 23:40:15 +00:00
sirius0xdev
a2a4d9d732
Merge pull request #162 from sirius0xdev/fix/move-kafka-to-siriusdevops-db
move Kafka broker from hermes-db to siriusdevops-db kustomization
2026-05-25 19:37:28 -04:00
Hermes Agent
691d7c68e7 move Kafka broker from hermes-db to siriusdevops-db kustomization
Trading services depend on Kafka (trading-kafka.customer1.svc.cluster.local:9092)
but it was never deployed — lived under hermes-db kustomization which isn't
applied. Move to siriusdevops-db where trading infra belongs.

Also remove stale redis-cluster.yaml reference from hermes-db (file missing).
2026-05-25 23:36:14 +00:00
sirius0xdev
c92cbafd75
Merge pull request #161 from sirius0xdev/fix/pgvector-sa-key-auth
fix: switch pgvector image build to SA key auth
2026-05-25 18:24:48 -04:00
Hermes DevOps
eec93f6216 fix: switch pgvector image build to SA key auth (bypass broken WIF) 2026-05-25 21:19:44 +00:00
sirius0xdev
3643cf40de
Merge pull request #160 from sirius0xdev/fix/data-service-probes
fix(data-service): probe /health/ready instead of /health
2026-05-25 17:18:57 -04:00
Hermes Agent
9cbf21173f fix(data-service): probe /health/ready instead of /health 2026-05-25 21:17:10 +00:00
sirius0xdev
c98a69580f
Merge pull request #159 from sirius0xdev/fix/pgvector-workflow-wif
fix: use correct workload identity provider for GCR auth
2026-05-25 16:40:56 -04:00
sirius0xdev
6877736916
Merge pull request #157 from sirius0xdev/migrate-trading-db-network-policy
fix(trading): update network policy egress to siriusdevops-pgdb
2026-05-25 16:40:42 -04:00
Hermes Agent
1b62af55df fix: use correct workload identity provider for GCR auth 2026-05-25 20:29:49 +00:00
sirius0xdev
33118b04eb
fix: add id-token permission for OIDC auth to GCR (#158)
Co-authored-by: Hermes Agent <hermes-agent@sirius0xdev.com>
2026-05-25 16:24:46 -04:00
sirius0xdev
e28d6ccf10 fix(trading): update network policy egress to siriusdevops-pgdb
Migrate trading platform DB egress rule from hermes-pgdb to
siriusdevops-pgdb to complete the trading DB migration. The
configmaps already point to siriusdevops-pgdb-rw but the
network policy still allowed egress to the old cluster.
2026-05-25 20:08:07 +00:00
sirius0xdev
558a5a7c9e turn on gpu 2026-05-25 19:08:02 +00:00
sirius0xdev
c357b69743
Merge pull request #156 from sirius0xdev/fix/pgvector-image-and-cluster
chore(news-bot): switch DB to siriusdevops-pgdb (main apps cluster)
2026-05-25 14:56:56 -04:00
Hermes Agent
22498cd97b chore(news-bot): switch to siriusdevops-pgdb as main apps cluster
- news-app-db.yaml: point Database CR to siriusdevops-pgdb
- All ConfigMaps: DB_HOST=siriusdevops-pgdb-rw, LLM_BASE_URL for vLLM
- hermes-pgdb now dedicated to pgvector/RAG for agents only
- Aligns with CNPG extension pattern and active kanban (devops startupProbes, backend exec-service)

PR to master per repo rules.
2026-05-25 18:56:08 +00:00
sirius0xdev
67410a8ac0
Merge pull request #155 from sirius0xdev/fix/pgvector-image-and-cluster
feat: add pgvector image build and update cluster imageName
2026-05-25 14:03:31 -04:00
sirius0xdev
6ca9d9532f
Merge pull request #154 from sirius0xdev/feat/migrate-trading-to-siriusdevops-pgdb
feat: migrate all trading-platform services from hermes-pgdb to siriusdevops-pgdb
2026-05-25 14:03:20 -04:00