# FIRMS active fire / hotspot heatmap — data source

The OSINT map's fire overlay is fed by **NASA FIRMS** (Fire Information for
Resource Management System). Two zero-cost options exist; this repo implements
option A (ingested vector points served as JSON), and option B (GIBS raster
tiles) is documented below for a no-storage frontend-only alternative.

---

## A. FIRMS area CSV → Postgres/TimescaleDB → `GET /api/fires`

### Data flow

```
NASA FIRMS area CSV  ──►  app/fire_sources.py  ──►  NATS events.fire
                                                          │
                                              app/ingestor.py (ingest_fire_row)
                                                          │
                                          fires hypertable (idempotent PK)
                                                          │
                                          GET /api/fires?bbox=&since=  (JSON)
```

* `app/fire_sources.py` fetches the VIIRS active-fire CSV for a bounding box,
  normalises each row (combining `acq_date` + `acq_time` into a UTC timestamp),
  and publishes one message per hotspot to NATS JetStream subject `events.fire`.
* The long-running ingester (`app/run_ingester.py`) polls FIRMS on its own
  ~15-minute cadence (`FIRMS_INTERVAL`, default 900 s) and shares the existing
  NATS consumer. `ingest_event` routes `source_type == "fire"` messages to
  `ingest_fire_row`, which writes to the `fires` table.
* **Idempotency:** the `fires` primary key IS the natural key
  `(latitude, longitude, acq_time, satellite)`. Inserts use
  `ON CONFLICT DO NOTHING`, so a hotspot re-delivered on a later poll is
  silently ignored — no duplicates, no upsert churn.
* The `fires` table is a **TimescaleDB hypertable** partitioned on `acq_time`
  (1-day chunks), so retention is one `drop_chunks` call away.

### Endpoint

```
GET /api/fires?bbox=<minlon,minlat,maxlon,maxlat>&since=<ISO-8601-UTC>&limit=<N>
```

| Query param | Meaning | Default |
|---|---|---|
| `bbox` | `"minlon,minlat,maxlon,maxlat"` to bound the result (e.g. `-125,24,-66,50`). | all stored detections |
| `since` | only hotspots acquired at/after this UTC instant (e.g. `2026-08-24T12:00:00Z`). | none |
| `limit` | max rows returned. | `2000` (max `10000`) |

Response is a plain JSON array (frontend renders it as the heatmap overlay):

```json
[
  {
    "latitude": 39.45678,
    "longitude": -121.12345,
    "brightness": 341.4,        // bright_ti4, Kelvin
    "confidence": "h",          // VIIRS: n (nominal) / l (low) / h (high)
    "acq_time": "2026-08-24T18:10:00Z",
    "satellite": "N",           // N (S-NPP), N20, N21
    "instrument": "VIIRS",
    "bright_ti5": 310.2,        // 12µm brightness, Kelvin
    "frp": 12.4,                // fire radiative power, MW
    "daynight": "D"             // D / N
  }
]
```

A manual poll can also be triggered with `POST /api/ingest/fires?bbox=...`.

### Configuration (all via env / `.env`)

| Var | Default | Notes |
|---|---|---|
| `FIRMS_MAP_KEY` | *(blank)* | **Required for live data.** Free key: <https://firms.modaps.eosdis.nasa.gov/api/map_key_info/> (1-minute signup). Until set, the fire loop logs a warning and stays idle — it never crashes the ingester. |
| `FIRMS_DATASET` | `VIIRS_SNPP_NRT` | NRT VIIRS S-NPP 375 m active fire detection. |
| `FIRMS_BBOX` | `-180,-60,180,75` | Poll area `"minlon,minlat,maxlon,maxlat"`. Narrow it (e.g. `-125,24,-66,50`) to cut payload and write volume. |
| `FIRMS_INTERVAL` | `900` | Poll cadence in seconds (~15 min; FIRMS NRT refreshes every ~5–10 min). |
| `INGEST_FIRES` | `1` | Set `0` to disable the fire loop entirely. |
| `FIRMS_TIMEOUT` | `60` | Outbound HTTP timeout for the CSV download. |

### Tests

`tests/` — parser/normalisation/mapping unit tests run anywhere; DB-backed
idempotency + API contract tests are marked `integration` and auto-skip without
a reachable TimescaleDB+PostGIS (the repo's `localhost/osint-dashboard-pg`
image or a local `postgis/postgis`):

```bash
DB_HOST=... DB_PORT=... DB_USER=osint DB_PASSWORD=... DB_NAME=osint_data \
  pytest tests/ -v
```

`DB_NULL_POOL=1` is set by the test suite (fresh connection per event loop).

### Live verification

Live end-to-end verification (real FIRMS fetch → NATS → Postgres → API) is
**blocked until `FIRMS_MAP_KEY` is set** in `.env`. Everything else — CSV
parsing, idempotent storage, the API contract — is verified against a real
TimescaleDB instance in the test suite.

---

## B. GIBS thermal-anomaly tiles — zero-cost, no key, no storage

If you want a fire layer with **zero backend work** (raster tiles rendered by
the map library directly, no ingest, no DB, no API key), NASA GIBS serves the
same VIIRS S-NPP detections as WMTS tiles:

* Layer: **`VIIRS_SNPP_Thermal_Anomalies_375m_All`** (375 m VIIRS S-NPP thermal
  anomalies / active fires). Sibling layers exist for day/night-only views
  (`..._Day`, `..._Night`).
* REST tile URL (Web Mercator, EPSG:3857 — what Leaflet/MapLibre use):

  ```
  https://gibs.earthdata.nasa.gov/wmts/epsg3857/best/VIIRS_SNPP_Thermal_Anomalies_375m_All/default/{Time}/GoogleMapsCompatible_Level{Z}/{Y}/{X}.png
  ```

* `{Time}` is a date like `2026-08-24` (or a time-of-day string); the list of
  available times comes from the WMTS capabilities:
  `https://gibs.earthdata.nasa.gov/wmts/epsg3857/best/1.0.0/WMTSCapabilities.xml`
  (search for the layer, read its `Dimension` → `Value`).
* Leaflet/MapLibre example:

  ```js
  L.tileLayer(
    'https://gibs.earthdata.nasa.gov/wmts/epsg3857/best/VIIRS_SNPP_Thermal_Anomalies_375m_All/default/{time}/GoogleMapsCompatible_Level{z}/{y}/{x}.png',
    { attribution: 'NASA GIBS / FIRMS', maxZoom: 9 }
  ).addTo(map);
  ```

**Trade-offs vs. option A:**

| | A — FIRMS CSV ingest | B — GIBS WMTS tiles |
|---|---|---|
| Data in our DB | yes (queryable, filterable) | no (pixels only) |
| Per-hotspot attributes (brightness, FRP, confidence) | yes | no (colour-coded only) |
| Time range / `since` filtering server-side | yes | tile `{Time}` per snapshot |
| Backend cost | ingest service + DB rows | none |
| API key | free FIRMS_MAP_KEY | none |

GIBS is the right choice when the map only needs "where are fires right now".
The FIRMS ingest is right when you want to query, aggregate, or persist the
detections (e.g. "fires near X in the last 24 h").