Continuous whole-IPv4 rolling sweep for open TCP 554, feeding the same
cameras table as the passive scraper (discovery_source=masscan).
- masscan_config.py: env-driven knobs (range, ports, rate, retries, excludes)
- masscan_scanner.py: JSON-lines parser, rtsp://IP/ URL + url_hash dedupe,
ip-api geolocation, insert/refresh, NATS publish for new finds
- run_masscan_service.py: long-lived rolling-sweep runner (streams results
in, restarts on pass completion); fails closed without an excludefile
- deploy/: systemd unit + README + excludes file for the Pi host
- .env.example: masscan section
Verified end-to-end against a local Postgres: parse, insert, and dedupe
(0 new on re-ingest) all pass.
FIRMS area-CSV requires an explicit [1..5] day range — requests without
it return 'Invalid day range. Expects [1..5].' as a 400. New
FIRMS_DAYS env (default 1).
Key saved via the dashboard Keys page sat unused in api_keys while
ingest_fires only checked the env var. Now env first, keystore fallback,
picked up on next poll without a restart.
- Vendored leaflet.heat 0.2.0; fires overlay is now a real heatmap
(L.heatLayer) with intensity driven by brightness or VIIRS confidence
(color mode selector) and a since= time filter (6h/24h/48h/7d/30d/all).
- Camera markers now open popups with the live snapshot thumbnail proxied
through GET /api/cameras/{id}/snapshot (backend TTL cache) + full
metadata (vendor, device, discovery source, first/last seen, coords,
source link); missing-snapshot cams show a placeholder instead of 404.
- New collapsible Layers panel (dark neon) over the map: per-layer
visibility toggles, opacity sliders (basemap/fires/cameras), live
counts legend, and a fire heat-intensity ramp that swaps for the
confidence legend in confidence mode.
- Clamp currentBBox() to world bounds so /api/cameras doesn't 422 at low
zoom (worldCopyJump spans multiple copies).
- Skip overlay reloads on moveend while a popup is open — popup autopan
no longer rebuilds the marker group and closes the popup.
- Zoom control moved to top-right so it doesn't collide with the panel.
Interactive world map panel for the dashboard:
- Vendored Leaflet 1.9.4 under app/static/vendor/leaflet/ (served via
new /static mount on the FastAPI app)
- New Map tab: GIBS WMTS raster basemap (BlueMarble_ShadedRelief_Bathymetry,
VIIRS/MODIS/Aqua CorrectedReflectance_TrueColor, VIIRS_DayNightBand),
layer picker + UTC date selector (Latest/-7d/-30d quick picks)
- /api/map/layers: curated GIBS catalog (tms, format, has_time, max_zoom)
- /api/map/times: per-layer date windows from GIBS Domains XML (6h cache,
graceful 502 on GIBS hiccup; static layers 422)
- Fire hotspots (existing /api/fires) + open cameras (/api/cameras) as
toggleable bbox-scoped overlays; dark UI matching dashboard theme
Vendor the newsPipeline scraper + summarizer into the repo and wire them into
docker-compose against the EXISTING osint-db (no second Postgres), replacing
the upstream k8s CronJobs with in-compose wall-clock loops (:00 scrape, :05
summarize).
- news/scraper: vendored Scrapy project (257 RSS feeds) + hourly loop
scheduler (run_news_scraper.py)
- news/summerizer: vendored Gemini map-reduce summarizer, cleaned:
* fix broken google-genai response handling (_extract_text, defensive)
* fix malformed INSERT/GRANT query in save_summary_to_db
* OSINT-neutral default MAP_PROMPT; futures/markets language gated behind
INCLUDE_FUTURES=0 (yfinance lazy-imported)
* env-configurable model, batch size, lookback window
+ hourly loop scheduler (run_news_summarizer.py, :05)
- alembic 003_news: idempotent articles + article_summaries tables
- API: GET /api/news and GET /api/news/summaries (+ models, schemas)
- tests/test_api_news.py: 5 DB-backed contract tests (all pass vs real PG)
- docs/news.md + .env.example updates
Both services run under the `ingest` compose profile (matching the
ingester/camera-scraper pattern) and build arm64 on the Pi via the existing
Forgejo CI workflow. telebot left out of scope (reserved env only).
Coherent merge of two coordinated features on the shared working tree:
FIRMS fire heatmap (backend, t_6e404c14):
- app/fire_sources.py: fetch FIRMS VIIRS area CSV (free MAP_KEY) -> NATS events.fire
- fires hypertable (TimescaleDB, 1-day chunks) with natural-key PK
(latitude, longitude, acq_time, satellite); idempotent ON CONFLICT DO NOTHING
- alembic/versions/002_fires.py; GET /api/fires?bbox=&since= (JSON only)
- POST /api/ingest/fires; ~15 min poll loop (FIRMS_INTERVAL=900) in ingester
- env-driven config (FIRMS_MAP_KEY/DATASET/BBOX/INTERVAL); docs/firms.md covers
the zero-cost GIBS VIIRS_SNPP_Thermal_Anomalies_375m_All tile alternative
- 18 tests (parser, mapping, idempotency, API contract) verified vs real
TimescaleDB+PostGIS (localhost/osint-dashboard-pg image)
API keys page (frontend, t_4433cff2):
- app/keystore.py: api_keys table (self-creating), FIRMS/GEMINI/TELEGRAM
registry with format validation, ****last4 masking, get_api_key()
- GET/POST/DELETE /api/keys (never returns full values); Keys tab in index.html
DB_NULL_POOL env switch in app/database.py enables a NullPool for tests /
short-lived processes that open a fresh event loop per unit.
