Commit graph

8 commits

Author SHA1 Message Date
Sirius DevOps
cf98af8101 Merge remote-tracking branch 'forgejo/master'
Some checks failed
build-and-deploy / build (push) Failing after 4s
# Conflicts:
#	.env.example
2026-08-24 15:40:31 -04:00
Sirius DevOps
627990efde Add NASA FIRMS active-fire ingest + /api/fires; API keys management page
Coherent merge of two coordinated features on the shared working tree:

FIRMS fire heatmap (backend, t_6e404c14):
- app/fire_sources.py: fetch FIRMS VIIRS area CSV (free MAP_KEY) -> NATS events.fire
- fires hypertable (TimescaleDB, 1-day chunks) with natural-key PK
  (latitude, longitude, acq_time, satellite); idempotent ON CONFLICT DO NOTHING
- alembic/versions/002_fires.py; GET /api/fires?bbox=&since= (JSON only)
- POST /api/ingest/fires; ~15 min poll loop (FIRMS_INTERVAL=900) in ingester
- env-driven config (FIRMS_MAP_KEY/DATASET/BBOX/INTERVAL); docs/firms.md covers
  the zero-cost GIBS VIIRS_SNPP_Thermal_Anomalies_375m_All tile alternative
- 18 tests (parser, mapping, idempotency, API contract) verified vs real
  TimescaleDB+PostGIS (localhost/osint-dashboard-pg image)

API keys page (frontend, t_4433cff2):
- app/keystore.py: api_keys table (self-creating), FIRMS/GEMINI/TELEGRAM
  registry with format validation, ****last4 masking, get_api_key()
- GET/POST/DELETE /api/keys (never returns full values); Keys tab in index.html

DB_NULL_POOL env switch in app/database.py enables a NullPool for tests /
short-lived processes that open a fresh event loop per unit.
2026-08-24 15:37:42 -04:00
Sirius DevOps
69a30115cc Add open IP camera discovery scraper (OSINT map)
Some checks failed
build-and-deploy / build (push) Failing after 5s
- camera_scraper: public-directory-only discovery (Insecam-style HTML +
  plain-text lists), hard private-range guard (fail closed), per-host
  rate limiting, Nominatim geocoding at <=1 req/s, TTL'd local snapshot
  cache, sha256 url_hash dedupe with Postgres upsert
- cameras table (migration 002) + /api/cameras?bbox= + snapshot endpoint
  with cache passthrough in main.py
- run_camera_service: long-running cycle worker following NATS->ingester
  pattern; publishes events.camera for shared ingester
- docker-compose camera-service profile, .env.example knobs

Verified E2E against TimescaleDB+PostGIS: private-range entries dropped,
dedupe across cycles holds, bbox query returns expected rows.
2026-08-24 14:54:51 -04:00
sirius0xdev
8eabe3b5e8
Fix SQLAlchemy 2.0 count().filter(); add search_vector trigger for full-text search
Some checks failed
build-and-deploy / build (push) Failing after 15s
2026-07-07 18:32:02 -04:00
sirius0xdev
49d5756ceb
Make events PK composite (id, ingested_at) for TimescaleDB hypertable partitioning requirement
Some checks failed
build-and-deploy / build (push) Failing after 15s
2026-07-07 18:29:50 -04:00
sirius0xdev
f7de9555a9
Fix migration: drop redundant explicit CREATE TYPE (collides with sa.Enum name=)
Some checks failed
build-and-deploy / build (push) Failing after 19s
2026-07-07 18:28:02 -04:00
sirius0xdev
3d53ed2c1c
Containerize for Pi: env-var config, compose stack, CI workflow
Some checks failed
build-and-deploy / build (push) Failing after 18s
- Add app/config.py (12-factor, env-driven: DB/NATS/MinIO, URL-encoded pw)
- Route database/ingestor/sources/alembic through config; kill hardcoded
  svc.cluster.local hostnames
- Drop unused redis dependency
- Add docker-compose.yml (timescaledb-postgis pg13 arm64 + optional nats)
- Add .env.example, .dockerignore, .forgejo/workflows/build.yml (arm64 build+SSH deploy)
2026-07-07 17:50:51 -04:00
sirius0xdev
93b8b89639
first commit 2026-06-04 20:30:04 -04:00