Sirius DevOps
627990efde
Add NASA FIRMS active-fire ingest + /api/fires; API keys management page
...
Coherent merge of two coordinated features on the shared working tree:
FIRMS fire heatmap (backend, t_6e404c14):
- app/fire_sources.py: fetch FIRMS VIIRS area CSV (free MAP_KEY) -> NATS events.fire
- fires hypertable (TimescaleDB, 1-day chunks) with natural-key PK
(latitude, longitude, acq_time, satellite); idempotent ON CONFLICT DO NOTHING
- alembic/versions/002_fires.py; GET /api/fires?bbox=&since= (JSON only)
- POST /api/ingest/fires; ~15 min poll loop (FIRMS_INTERVAL=900) in ingester
- env-driven config (FIRMS_MAP_KEY/DATASET/BBOX/INTERVAL); docs/firms.md covers
the zero-cost GIBS VIIRS_SNPP_Thermal_Anomalies_375m_All tile alternative
- 18 tests (parser, mapping, idempotency, API contract) verified vs real
TimescaleDB+PostGIS (localhost/osint-dashboard-pg image)
API keys page (frontend, t_4433cff2):
- app/keystore.py: api_keys table (self-creating), FIRMS/GEMINI/TELEGRAM
registry with format validation, ****last4 masking, get_api_key()
- GET/POST/DELETE /api/keys (never returns full values); Keys tab in index.html
DB_NULL_POOL env switch in app/database.py enables a NullPool for tests /
short-lived processes that open a fresh event loop per unit.
2026-08-24 15:37:42 -04:00
sirius0xdev
bce2d5c3bd
CI: trigger run with compose-plugin-mounted runner (no SSH deploy)
build-and-deploy / build (push) Failing after 5s
2026-07-09 19:35:29 -04:00
sirius0xdev
6ddd773a75
CI: deploy directly from checkout workspace (local docker), no cp/SSH; explicit COMPOSE_PROJECT_NAME
build-and-deploy / build (push) Failing after 19s
2026-07-09 19:33:18 -04:00
sirius0xdev
650ad27349
CI: deploy locally on the Pi (no SSH/scp) — runner already has docker socket + /opt/siriusdevops mounted
build-and-deploy / build (push) Failing after 20s
2026-07-09 19:22:30 -04:00
sirius0xdev
8c7f17f645
Harden ingest_event: coerce non-UUID source_id to NULL (tolerates legacy messages)
build-and-deploy / build (push) Successful in 4m15s
2026-07-07 19:44:13 -04:00
sirius0xdev
67fee36c7c
Wire ingestion: long-running ingester (sources->NATS->Postgres), fix nats-py API (durable=/LIMITS), drop UUID source_id on RSS
build-and-deploy / build (push) Successful in 4m18s
2026-07-07 19:35:28 -04:00
sirius0xdev
5496023285
Add long-running ingester (sources->NATS->Postgres) as compose 'ingest' profile service
build-and-deploy / build (push) Failing after 2m58s
2026-07-07 19:18:37 -04:00
sirius0xdev
0b344760dc
Add .gitignore (pycache, venv, .env)
build-and-deploy / build (push) Successful in 3m18s
2026-07-07 18:46:06 -04:00
sirius0xdev
17d39cf622
CI: build all compose services (app+db) via docker compose build before deploy
build-and-deploy / build (push) Failing after 10s
2026-07-07 18:36:12 -04:00
sirius0xdev
8eabe3b5e8
Fix SQLAlchemy 2.0 count().filter(); add search_vector trigger for full-text search
build-and-deploy / build (push) Failing after 15s
2026-07-07 18:32:02 -04:00
sirius0xdev
49d5756ceb
Make events PK composite (id, ingested_at) for TimescaleDB hypertable partitioning requirement
build-and-deploy / build (push) Failing after 15s
2026-07-07 18:29:50 -04:00
sirius0xdev
f7de9555a9
Fix migration: drop redundant explicit CREATE TYPE (collides with sa.Enum name=)
build-and-deploy / build (push) Failing after 19s
2026-07-07 18:28:02 -04:00
sirius0xdev
0a6da2f9c1
Run migrations via entrypoint (alembic upgrade head) instead of nested asyncio.run in startup hook
build-and-deploy / build (push) Failing after 16s
2026-07-07 18:24:01 -04:00
sirius0xdev
985fedec7b
Preload timescaledb via db command (conf.d ignored by official postgres entrypoint)
build-and-deploy / build (push) Failing after 15s
2026-07-07 18:21:48 -04:00
sirius0xdev
b40f83f393
Fix Dockerfile.pg: mkdir conf.d before writing timescaledb preload
build-and-deploy / build (push) Failing after 15s
2026-07-07 18:17:31 -04:00
sirius0xdev
56ed3d7b7e
Rebase custom PG image on official postgres:16 (honors POSTGRES_* env + md5 auth); add timescaledb/postgis via apt
build-and-deploy / build (push) Failing after 16s
2026-07-07 18:13:40 -04:00
sirius0xdev
5b879dfc6b
Fix PYTHONPATH: app modules live in /app/app, not /app
build-and-deploy / build (push) Failing after 17s
2026-07-07 18:06:40 -04:00
sirius0xdev
14a9178f8c
Use custom arm64 PG16+PostGIS+TimescaleDB image (no upstream arm64 postgis-timescale tag exists)
build-and-deploy / build (push) Failing after 17s
2026-07-07 18:00:45 -04:00
sirius0xdev
3d53ed2c1c
Containerize for Pi: env-var config, compose stack, CI workflow
...
build-and-deploy / build (push) Failing after 18s
- Add app/config.py (12-factor, env-driven: DB/NATS/MinIO, URL-encoded pw)
- Route database/ingestor/sources/alembic through config; kill hardcoded
svc.cluster.local hostnames
- Drop unused redis dependency
- Add docker-compose.yml (timescaledb-postgis pg13 arm64 + optional nats)
- Add .env.example, .dockerignore, .forgejo/workflows/build.yml (arm64 build+SSH deploy)
2026-07-07 17:50:51 -04:00
sirius0xdev
93b8b89639
first commit
2026-06-04 20:30:04 -04:00
