Vendor the newsPipeline scraper + summarizer into the repo and wire them into
docker-compose against the EXISTING osint-db (no second Postgres), replacing
the upstream k8s CronJobs with in-compose wall-clock loops (:00 scrape, :05
summarize).
- news/scraper: vendored Scrapy project (257 RSS feeds) + hourly loop
scheduler (run_news_scraper.py)
- news/summerizer: vendored Gemini map-reduce summarizer, cleaned:
* fix broken google-genai response handling (_extract_text, defensive)
* fix malformed INSERT/GRANT query in save_summary_to_db
* OSINT-neutral default MAP_PROMPT; futures/markets language gated behind
INCLUDE_FUTURES=0 (yfinance lazy-imported)
* env-configurable model, batch size, lookback window
+ hourly loop scheduler (run_news_summarizer.py, :05)
- alembic 003_news: idempotent articles + article_summaries tables
- API: GET /api/news and GET /api/news/summaries (+ models, schemas)
- tests/test_api_news.py: 5 DB-backed contract tests (all pass vs real PG)
- docs/news.md + .env.example updates
Both services run under the `ingest` compose profile (matching the
ingester/camera-scraper pattern) and build arm64 on the Pi via the existing
Forgejo CI workflow. telebot left out of scope (reserved env only).
Coherent merge of two coordinated features on the shared working tree:
FIRMS fire heatmap (backend, t_6e404c14):
- app/fire_sources.py: fetch FIRMS VIIRS area CSV (free MAP_KEY) -> NATS events.fire
- fires hypertable (TimescaleDB, 1-day chunks) with natural-key PK
(latitude, longitude, acq_time, satellite); idempotent ON CONFLICT DO NOTHING
- alembic/versions/002_fires.py; GET /api/fires?bbox=&since= (JSON only)
- POST /api/ingest/fires; ~15 min poll loop (FIRMS_INTERVAL=900) in ingester
- env-driven config (FIRMS_MAP_KEY/DATASET/BBOX/INTERVAL); docs/firms.md covers
the zero-cost GIBS VIIRS_SNPP_Thermal_Anomalies_375m_All tile alternative
- 18 tests (parser, mapping, idempotency, API contract) verified vs real
TimescaleDB+PostGIS (localhost/osint-dashboard-pg image)
API keys page (frontend, t_4433cff2):
- app/keystore.py: api_keys table (self-creating), FIRMS/GEMINI/TELEGRAM
registry with format validation, ****last4 masking, get_api_key()
- GET/POST/DELETE /api/keys (never returns full values); Keys tab in index.html
DB_NULL_POOL env switch in app/database.py enables a NullPool for tests /
short-lived processes that open a fresh event loop per unit.
