627990efde
Coherent merge of two coordinated features on the shared working tree: FIRMS fire heatmap (backend, t_6e404c14): - app/fire_sources.py: fetch FIRMS VIIRS area CSV (free MAP_KEY) -> NATS events.fire - fires hypertable (TimescaleDB, 1-day chunks) with natural-key PK (latitude, longitude, acq_time, satellite); idempotent ON CONFLICT DO NOTHING - alembic/versions/002_fires.py; GET /api/fires?bbox=&since= (JSON only) - POST /api/ingest/fires; ~15 min poll loop (FIRMS_INTERVAL=900) in ingester - env-driven config (FIRMS_MAP_KEY/DATASET/BBOX/INTERVAL); docs/firms.md covers the zero-cost GIBS VIIRS_SNPP_Thermal_Anomalies_375m_All tile alternative - 18 tests (parser, mapping, idempotency, API contract) verified vs real TimescaleDB+PostGIS (localhost/osint-dashboard-pg image) API keys page (frontend, t_4433cff2): - app/keystore.py: api_keys table (self-creating), FIRMS/GEMINI/TELEGRAM registry with format validation, ****last4 masking, get_api_key() - GET/POST/DELETE /api/keys (never returns full values); Keys tab in index.html DB_NULL_POOL env switch in app/database.py enables a NullPool for tests / short-lived processes that open a fresh event loop per unit.
145 lines
6.2 KiB
Markdown
145 lines
6.2 KiB
Markdown
# FIRMS active fire / hotspot heatmap — data source
|
||
|
||
The OSINT map's fire overlay is fed by **NASA FIRMS** (Fire Information for
|
||
Resource Management System). Two zero-cost options exist; this repo implements
|
||
option A (ingested vector points served as JSON), and option B (GIBS raster
|
||
tiles) is documented below for a no-storage frontend-only alternative.
|
||
|
||
---
|
||
|
||
## A. FIRMS area CSV → Postgres/TimescaleDB → `GET /api/fires`
|
||
|
||
### Data flow
|
||
|
||
```
|
||
NASA FIRMS area CSV ──► app/fire_sources.py ──► NATS events.fire
|
||
│
|
||
app/ingestor.py (ingest_fire_row)
|
||
│
|
||
fires hypertable (idempotent PK)
|
||
│
|
||
GET /api/fires?bbox=&since= (JSON)
|
||
```
|
||
|
||
* `app/fire_sources.py` fetches the VIIRS active-fire CSV for a bounding box,
|
||
normalises each row (combining `acq_date` + `acq_time` into a UTC timestamp),
|
||
and publishes one message per hotspot to NATS JetStream subject `events.fire`.
|
||
* The long-running ingester (`app/run_ingester.py`) polls FIRMS on its own
|
||
~15-minute cadence (`FIRMS_INTERVAL`, default 900 s) and shares the existing
|
||
NATS consumer. `ingest_event` routes `source_type == "fire"` messages to
|
||
`ingest_fire_row`, which writes to the `fires` table.
|
||
* **Idempotency:** the `fires` primary key IS the natural key
|
||
`(latitude, longitude, acq_time, satellite)`. Inserts use
|
||
`ON CONFLICT DO NOTHING`, so a hotspot re-delivered on a later poll is
|
||
silently ignored — no duplicates, no upsert churn.
|
||
* The `fires` table is a **TimescaleDB hypertable** partitioned on `acq_time`
|
||
(1-day chunks), so retention is one `drop_chunks` call away.
|
||
|
||
### Endpoint
|
||
|
||
```
|
||
GET /api/fires?bbox=<minlon,minlat,maxlon,maxlat>&since=<ISO-8601-UTC>&limit=<N>
|
||
```
|
||
|
||
| Query param | Meaning | Default |
|
||
|---|---|---|
|
||
| `bbox` | `"minlon,minlat,maxlon,maxlat"` to bound the result (e.g. `-125,24,-66,50`). | all stored detections |
|
||
| `since` | only hotspots acquired at/after this UTC instant (e.g. `2026-08-24T12:00:00Z`). | none |
|
||
| `limit` | max rows returned. | `2000` (max `10000`) |
|
||
|
||
Response is a plain JSON array (frontend renders it as the heatmap overlay):
|
||
|
||
```json
|
||
[
|
||
{
|
||
"latitude": 39.45678,
|
||
"longitude": -121.12345,
|
||
"brightness": 341.4, // bright_ti4, Kelvin
|
||
"confidence": "h", // VIIRS: n (nominal) / l (low) / h (high)
|
||
"acq_time": "2026-08-24T18:10:00Z",
|
||
"satellite": "N", // N (S-NPP), N20, N21
|
||
"instrument": "VIIRS",
|
||
"bright_ti5": 310.2, // 12µm brightness, Kelvin
|
||
"frp": 12.4, // fire radiative power, MW
|
||
"daynight": "D" // D / N
|
||
}
|
||
]
|
||
```
|
||
|
||
A manual poll can also be triggered with `POST /api/ingest/fires?bbox=...`.
|
||
|
||
### Configuration (all via env / `.env`)
|
||
|
||
| Var | Default | Notes |
|
||
|---|---|---|
|
||
| `FIRMS_MAP_KEY` | *(blank)* | **Required for live data.** Free key: <https://firms.modaps.eosdis.nasa.gov/api/map_key_info/> (1-minute signup). Until set, the fire loop logs a warning and stays idle — it never crashes the ingester. |
|
||
| `FIRMS_DATASET` | `VIIRS_SNPP_NRT` | NRT VIIRS S-NPP 375 m active fire detection. |
|
||
| `FIRMS_BBOX` | `-180,-60,180,75` | Poll area `"minlon,minlat,maxlon,maxlat"`. Narrow it (e.g. `-125,24,-66,50`) to cut payload and write volume. |
|
||
| `FIRMS_INTERVAL` | `900` | Poll cadence in seconds (~15 min; FIRMS NRT refreshes every ~5–10 min). |
|
||
| `INGEST_FIRES` | `1` | Set `0` to disable the fire loop entirely. |
|
||
| `FIRMS_TIMEOUT` | `60` | Outbound HTTP timeout for the CSV download. |
|
||
|
||
### Tests
|
||
|
||
`tests/` — parser/normalisation/mapping unit tests run anywhere; DB-backed
|
||
idempotency + API contract tests are marked `integration` and auto-skip without
|
||
a reachable TimescaleDB+PostGIS (the repo's `localhost/osint-dashboard-pg`
|
||
image or a local `postgis/postgis`):
|
||
|
||
```bash
|
||
DB_HOST=... DB_PORT=... DB_USER=osint DB_PASSWORD=... DB_NAME=osint_data \
|
||
pytest tests/ -v
|
||
```
|
||
|
||
`DB_NULL_POOL=1` is set by the test suite (fresh connection per event loop).
|
||
|
||
### Live verification
|
||
|
||
Live end-to-end verification (real FIRMS fetch → NATS → Postgres → API) is
|
||
**blocked until `FIRMS_MAP_KEY` is set** in `.env`. Everything else — CSV
|
||
parsing, idempotent storage, the API contract — is verified against a real
|
||
TimescaleDB instance in the test suite.
|
||
|
||
---
|
||
|
||
## B. GIBS thermal-anomaly tiles — zero-cost, no key, no storage
|
||
|
||
If you want a fire layer with **zero backend work** (raster tiles rendered by
|
||
the map library directly, no ingest, no DB, no API key), NASA GIBS serves the
|
||
same VIIRS S-NPP detections as WMTS tiles:
|
||
|
||
* Layer: **`VIIRS_SNPP_Thermal_Anomalies_375m_All`** (375 m VIIRS S-NPP thermal
|
||
anomalies / active fires). Sibling layers exist for day/night-only views
|
||
(`..._Day`, `..._Night`).
|
||
* REST tile URL (Web Mercator, EPSG:3857 — what Leaflet/MapLibre use):
|
||
|
||
```
|
||
https://gibs.earthdata.nasa.gov/wmts/epsg3857/best/VIIRS_SNPP_Thermal_Anomalies_375m_All/default/{Time}/GoogleMapsCompatible_Level{Z}/{Y}/{X}.png
|
||
```
|
||
|
||
* `{Time}` is a date like `2026-08-24` (or a time-of-day string); the list of
|
||
available times comes from the WMTS capabilities:
|
||
`https://gibs.earthdata.nasa.gov/wmts/epsg3857/best/1.0.0/WMTSCapabilities.xml`
|
||
(search for the layer, read its `Dimension` → `Value`).
|
||
* Leaflet/MapLibre example:
|
||
|
||
```js
|
||
L.tileLayer(
|
||
'https://gibs.earthdata.nasa.gov/wmts/epsg3857/best/VIIRS_SNPP_Thermal_Anomalies_375m_All/default/{time}/GoogleMapsCompatible_Level{z}/{y}/{x}.png',
|
||
{ attribution: 'NASA GIBS / FIRMS', maxZoom: 9 }
|
||
).addTo(map);
|
||
```
|
||
|
||
**Trade-offs vs. option A:**
|
||
|
||
| | A — FIRMS CSV ingest | B — GIBS WMTS tiles |
|
||
|---|---|---|
|
||
| Data in our DB | yes (queryable, filterable) | no (pixels only) |
|
||
| Per-hotspot attributes (brightness, FRP, confidence) | yes | no (colour-coded only) |
|
||
| Time range / `since` filtering server-side | yes | tile `{Time}` per snapshot |
|
||
| Backend cost | ingest service + DB rows | none |
|
||
| API key | free FIRMS_MAP_KEY | none |
|
||
|
||
GIBS is the right choice when the map only needs "where are fires right now".
|
||
The FIRMS ingest is right when you want to query, aggregate, or persist the
|
||
detections (e.g. "fires near X in the last 24 h").
|