osint-dashboard/docs/firms.md

FIRMS active fire / hotspot heatmap — data source

The OSINT map's fire overlay is fed by NASA FIRMS (Fire Information for Resource Management System). Two zero-cost options exist; this repo implements option A (ingested vector points served as JSON), and option B (GIBS raster tiles) is documented below for a no-storage frontend-only alternative.

---

A. FIRMS area CSV → Postgres/TimescaleDB → GET /api/fires

Data flow

NASA FIRMS area CSV  ──►  app/fire_sources.py  ──►  NATS events.fire
                                                          │
                                              app/ingestor.py (ingest_fire_row)
                                                          │
                                          fires hypertable (idempotent PK)
                                                          │
                                          GET /api/fires?bbox=&since=  (JSON)

• app/fire_sources.py fetches the VIIRS active-fire CSV for a bounding box, normalises each row (combining acq_date + acq_time into a UTC timestamp), and publishes one message per hotspot to NATS JetStream subject events.fire.
• The long-running ingester (app/run_ingester.py) polls FIRMS on its own ~15-minute cadence (FIRMS_INTERVAL, default 900 s) and shares the existing NATS consumer. ingest_event routes source_type == "fire" messages to ingest_fire_row, which writes to the fires table.
• Idempotency: the fires primary key IS the natural key (latitude, longitude, acq_time, satellite). Inserts use ON CONFLICT DO NOTHING, so a hotspot re-delivered on a later poll is silently ignored — no duplicates, no upsert churn.
• The fires table is a TimescaleDB hypertable partitioned on acq_time (1-day chunks), so retention is one drop_chunks call away.

Endpoint

GET /api/fires?bbox=<minlon,minlat,maxlon,maxlat>&since=<ISO-8601-UTC>&limit=<N>

Query param	Meaning	Default
bbox	"minlon,minlat,maxlon,maxlat" to bound the result (e.g. -125,24,-66,50).	all stored detections
since	only hotspots acquired at/after this UTC instant (e.g. 2026-08-24T12:00:00Z).	none
limit	max rows returned.	2000 (max 10000)

Response is a plain JSON array (frontend renders it as the heatmap overlay):

[
  {
    "latitude": 39.45678,
    "longitude": -121.12345,
    "brightness": 341.4,        // bright_ti4, Kelvin
    "confidence": "h",          // VIIRS: n (nominal) / l (low) / h (high)
    "acq_time": "2026-08-24T18:10:00Z",
    "satellite": "N",           // N (S-NPP), N20, N21
    "instrument": "VIIRS",
    "bright_ti5": 310.2,        // 12µm brightness, Kelvin
    "frp": 12.4,                // fire radiative power, MW
    "daynight": "D"             // D / N
  }
]

A manual poll can also be triggered with POST /api/ingest/fires?bbox=....

Configuration (all via env / .env)

Var	Default	Notes
FIRMS_MAP_KEY	(blank)	Required for live data. Free key: https://firms.modaps.eosdis.nasa.gov/api/map_key_info/ (1-minute signup). Until set, the fire loop logs a warning and stays idle — it never crashes the ingester.
FIRMS_DATASET	VIIRS_SNPP_NRT	NRT VIIRS S-NPP 375 m active fire detection.
FIRMS_BBOX	-180,-60,180,75	Poll area "minlon,minlat,maxlon,maxlat". Narrow it (e.g. -125,24,-66,50) to cut payload and write volume.
FIRMS_INTERVAL	900	Poll cadence in seconds (~15 min; FIRMS NRT refreshes every ~5–10 min).
INGEST_FIRES	1	Set 0 to disable the fire loop entirely.
FIRMS_TIMEOUT	60	Outbound HTTP timeout for the CSV download.

Tests

tests/ — parser/normalisation/mapping unit tests run anywhere; DB-backed idempotency + API contract tests are marked integration and auto-skip without a reachable TimescaleDB+PostGIS (the repo's localhost/osint-dashboard-pg image or a local postgis/postgis):

DB_HOST=... DB_PORT=... DB_USER=osint DB_PASSWORD=... DB_NAME=osint_data \
  pytest tests/ -v

DB_NULL_POOL=1 is set by the test suite (fresh connection per event loop).

Live verification

Live end-to-end verification (real FIRMS fetch → NATS → Postgres → API) is blocked until FIRMS_MAP_KEY is set in .env. Everything else — CSV parsing, idempotent storage, the API contract — is verified against a real TimescaleDB instance in the test suite.

---

B. GIBS thermal-anomaly tiles — zero-cost, no key, no storage

If you want a fire layer with zero backend work (raster tiles rendered by the map library directly, no ingest, no DB, no API key), NASA GIBS serves the same VIIRS S-NPP detections as WMTS tiles:

• Layer: VIIRS_SNPP_Thermal_Anomalies_375m_All (375 m VIIRS S-NPP thermal anomalies / active fires). Sibling layers exist for day/night-only views (..._Day, ..._Night).

• REST tile URL (Web Mercator, EPSG:3857 — what Leaflet/MapLibre use):

  https://gibs.earthdata.nasa.gov/wmts/epsg3857/best/VIIRS_SNPP_Thermal_Anomalies_375m_All/default/{Time}/GoogleMapsCompatible_Level{Z}/{Y}/{X}.png
  

• {Time} is a date like 2026-08-24 (or a time-of-day string); the list of available times comes from the WMTS capabilities: https://gibs.earthdata.nasa.gov/wmts/epsg3857/best/1.0.0/WMTSCapabilities.xml (search for the layer, read its Dimension → Value).

• Leaflet/MapLibre example:

  L.tileLayer(
    'https://gibs.earthdata.nasa.gov/wmts/epsg3857/best/VIIRS_SNPP_Thermal_Anomalies_375m_All/default/{time}/GoogleMapsCompatible_Level{z}/{y}/{x}.png',
    { attribution: 'NASA GIBS / FIRMS', maxZoom: 9 }
  ).addTo(map);
  

Trade-offs vs. option A:

	A — FIRMS CSV ingest	B — GIBS WMTS tiles
Data in our DB	yes (queryable, filterable)	no (pixels only)
Per-hotspot attributes (brightness, FRP, confidence)	yes	no (colour-coded only)
Time range / since filtering server-side	yes	tile {Time} per snapshot
Backend cost	ingest service + DB rows	none
API key	free FIRMS_MAP_KEY	none

GIBS is the right choice when the map only needs "where are fires right now". The FIRMS ingest is right when you want to query, aggregate, or persist the detections (e.g. "fires near X in the last 24 h").