146 lines
6.2 KiB
Markdown
146 lines
6.2 KiB
Markdown
|
# FIRMS active fire / hotspot heatmap — data source
|
|||
|
|
|
|||
|
|
The OSINT map's fire overlay is fed by **NASA FIRMS** (Fire Information for
|
|||
|
|
Resource Management System). Two zero-cost options exist; this repo implements
|
|||
|
|
option A (ingested vector points served as JSON), and option B (GIBS raster
|
|||
|
|
tiles) is documented below for a no-storage frontend-only alternative.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## A. FIRMS area CSV → Postgres/TimescaleDB → `GET /api/fires`
|
|||
|
|
|
|||
|
|
### Data flow
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
NASA FIRMS area CSV ──► app/fire_sources.py ──► NATS events.fire
|
|||
|
|
│
|
|||
|
|
app/ingestor.py (ingest_fire_row)
|
|||
|
|
│
|
|||
|
|
fires hypertable (idempotent PK)
|
|||
|
|
│
|
|||
|
|
GET /api/fires?bbox=&since= (JSON)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
* `app/fire_sources.py` fetches the VIIRS active-fire CSV for a bounding box,
|
|||
|
|
normalises each row (combining `acq_date` + `acq_time` into a UTC timestamp),
|
|||
|
|
and publishes one message per hotspot to NATS JetStream subject `events.fire`.
|
|||
|
|
* The long-running ingester (`app/run_ingester.py`) polls FIRMS on its own
|
|||
|
|
~15-minute cadence (`FIRMS_INTERVAL`, default 900 s) and shares the existing
|
|||
|
|
NATS consumer. `ingest_event` routes `source_type == "fire"` messages to
|
|||
|
|
`ingest_fire_row`, which writes to the `fires` table.
|
|||
|
|
* **Idempotency:** the `fires` primary key IS the natural key
|
|||
|
|
`(latitude, longitude, acq_time, satellite)`. Inserts use
|
|||
|
|
`ON CONFLICT DO NOTHING`, so a hotspot re-delivered on a later poll is
|
|||
|
|
silently ignored — no duplicates, no upsert churn.
|
|||
|
|
* The `fires` table is a **TimescaleDB hypertable** partitioned on `acq_time`
|
|||
|
|
(1-day chunks), so retention is one `drop_chunks` call away.
|
|||
|
|
|
|||
|
|
### Endpoint
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
GET /api/fires?bbox=<minlon,minlat,maxlon,maxlat>&since=<ISO-8601-UTC>&limit=<N>
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
| Query param | Meaning | Default |
|
|||
|
|
|---|---|---|
|
|||
|
|
| `bbox` | `"minlon,minlat,maxlon,maxlat"` to bound the result (e.g. `-125,24,-66,50`). | all stored detections |
|
|||
|
|
| `since` | only hotspots acquired at/after this UTC instant (e.g. `2026-08-24T12:00:00Z`). | none |
|
|||
|
|
| `limit` | max rows returned. | `2000` (max `10000`) |
|
|||
|
|
|
|||
|
|
Response is a plain JSON array (frontend renders it as the heatmap overlay):
|
|||
|
|
|
|||
|
|
```json
|
|||
|
|
[
|
|||
|
|
{
|
|||
|
|
"latitude": 39.45678,
|
|||
|
|
"longitude": -121.12345,
|
|||
|
|
"brightness": 341.4, // bright_ti4, Kelvin
|
|||
|
|
"confidence": "h", // VIIRS: n (nominal) / l (low) / h (high)
|
|||
|
|
"acq_time": "2026-08-24T18:10:00Z",
|
|||
|
|
"satellite": "N", // N (S-NPP), N20, N21
|
|||
|
|
"instrument": "VIIRS",
|
|||
|
|
"bright_ti5": 310.2, // 12µm brightness, Kelvin
|
|||
|
|
"frp": 12.4, // fire radiative power, MW
|
|||
|
|
"daynight": "D" // D / N
|
|||
|
|
}
|
|||
|
|
]
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
A manual poll can also be triggered with `POST /api/ingest/fires?bbox=...`.
|
|||
|
|
|
|||
|
|
### Configuration (all via env / `.env`)
|
|||
|
|
|
|||
|
|
| Var | Default | Notes |
|
|||
|
|
|---|---|---|
|
|||
|
|
| `FIRMS_MAP_KEY` | *(blank)* | **Required for live data.** Free key: <https://firms.modaps.eosdis.nasa.gov/api/map_key_info/> (1-minute signup). Until set, the fire loop logs a warning and stays idle — it never crashes the ingester. |
|
|||
|
|
| `FIRMS_DATASET` | `VIIRS_SNPP_NRT` | NRT VIIRS S-NPP 375 m active fire detection. |
|
|||
|
|
| `FIRMS_BBOX` | `-180,-60,180,75` | Poll area `"minlon,minlat,maxlon,maxlat"`. Narrow it (e.g. `-125,24,-66,50`) to cut payload and write volume. |
|
|||
|
|
| `FIRMS_INTERVAL` | `900` | Poll cadence in seconds (~15 min; FIRMS NRT refreshes every ~5–10 min). |
|
|||
|
|
| `INGEST_FIRES` | `1` | Set `0` to disable the fire loop entirely. |
|
|||
|
|
| `FIRMS_TIMEOUT` | `60` | Outbound HTTP timeout for the CSV download. |
|
|||
|
|
|
|||
|
|
### Tests
|
|||
|
|
|
|||
|
|
`tests/` — parser/normalisation/mapping unit tests run anywhere; DB-backed
|
|||
|
|
idempotency + API contract tests are marked `integration` and auto-skip without
|
|||
|
|
a reachable TimescaleDB+PostGIS (the repo's `localhost/osint-dashboard-pg`
|
|||
|
|
image or a local `postgis/postgis`):
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
DB_HOST=... DB_PORT=... DB_USER=osint DB_PASSWORD=... DB_NAME=osint_data \
|
|||
|
|
pytest tests/ -v
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
`DB_NULL_POOL=1` is set by the test suite (fresh connection per event loop).
|
|||
|
|
|
|||
|
|
### Live verification
|
|||
|
|
|
|||
|
|
Live end-to-end verification (real FIRMS fetch → NATS → Postgres → API) is
|
|||
|
|
**blocked until `FIRMS_MAP_KEY` is set** in `.env`. Everything else — CSV
|
|||
|
|
parsing, idempotent storage, the API contract — is verified against a real
|
|||
|
|
TimescaleDB instance in the test suite.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## B. GIBS thermal-anomaly tiles — zero-cost, no key, no storage
|
|||
|
|
|
|||
|
|
If you want a fire layer with **zero backend work** (raster tiles rendered by
|
|||
|
|
the map library directly, no ingest, no DB, no API key), NASA GIBS serves the
|
|||
|
|
same VIIRS S-NPP detections as WMTS tiles:
|
|||
|
|
|
|||
|
|
* Layer: **`VIIRS_SNPP_Thermal_Anomalies_375m_All`** (375 m VIIRS S-NPP thermal
|
|||
|
|
anomalies / active fires). Sibling layers exist for day/night-only views
|
|||
|
|
(`..._Day`, `..._Night`).
|
|||
|
|
* REST tile URL (Web Mercator, EPSG:3857 — what Leaflet/MapLibre use):
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
https://gibs.earthdata.nasa.gov/wmts/epsg3857/best/VIIRS_SNPP_Thermal_Anomalies_375m_All/default/{Time}/GoogleMapsCompatible_Level{Z}/{Y}/{X}.png
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
* `{Time}` is a date like `2026-08-24` (or a time-of-day string); the list of
|
|||
|
|
available times comes from the WMTS capabilities:
|
|||
|
|
`https://gibs.earthdata.nasa.gov/wmts/epsg3857/best/1.0.0/WMTSCapabilities.xml`
|
|||
|
|
(search for the layer, read its `Dimension` → `Value`).
|
|||
|
|
* Leaflet/MapLibre example:
|
|||
|
|
|
|||
|
|
```js
|
|||
|
|
L.tileLayer(
|
|||
|
|
'https://gibs.earthdata.nasa.gov/wmts/epsg3857/best/VIIRS_SNPP_Thermal_Anomalies_375m_All/default/{time}/GoogleMapsCompatible_Level{z}/{y}/{x}.png',
|
|||
|
|
{ attribution: 'NASA GIBS / FIRMS', maxZoom: 9 }
|
|||
|
|
).addTo(map);
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**Trade-offs vs. option A:**
|
|||
|
|
|
|||
|
|
| | A — FIRMS CSV ingest | B — GIBS WMTS tiles |
|
|||
|
|
|---|---|---|
|
|||
|
|
| Data in our DB | yes (queryable, filterable) | no (pixels only) |
|
|||
|
|
| Per-hotspot attributes (brightness, FRP, confidence) | yes | no (colour-coded only) |
|
|||
|
|
| Time range / `since` filtering server-side | yes | tile `{Time}` per snapshot |
|
|||
|
|
| Backend cost | ingest service + DB rows | none |
|
|||
|
|
| API key | free FIRMS_MAP_KEY | none |
|
|||
|
|
|
|||
|
|
GIBS is the right choice when the map only needs "where are fires right now".
|
|||
|
|
The FIRMS ingest is right when you want to query, aggregate, or persist the
|
|||
|
|
detections (e.g. "fires near X in the last 24 h").
|