613 lines
28 KiB
Markdown
613 lines
28 KiB
Markdown
# Free data streams for the OSINT dashboard
|
||
|
||
Builder brief for backend + frontend. Researched 2026-08-27. Every endpoint below was either live-probed from this machine or taken from the provider’s current docs. Prefer **free, no-key, CORS-open** sources first. Keys are called out explicitly.
|
||
|
||
This is **not** a camera-discovery / masscan change. Existing camera rules still apply: never emit `rtsp://` hrefs; masscan pins go through `/api/cameras/{id}/snapshot`; HTTP directory cams use `/stream` MJPEG.
|
||
|
||
---
|
||
|
||
## 0. What is already in the repo (do not rebuild)
|
||
|
||
| Layer | Status | Notes |
|
||
|---|---|---|
|
||
| NASA FIRMS VIIRS hotspots | Ingested (`app/fire_sources.py` → NATS `events.fire` → `fires` hypertable → `GET /api/fires`) | Needs free `FIRMS_MAP_KEY`. See `docs/firms.md`. |
|
||
| NASA GIBS basemaps | Frontend tiles via `app/gibs_map.py` | No key. CORS `*`. |
|
||
| GIBS VIIRS thermal tiles | Documented, not wired as overlay | Same GIBS stack; no key. |
|
||
| Cameras | Scraper + masscan → `cameras` table | Defaults already include ALERTWest JPEGs + Live-Environment-Streams HLS/YouTube GeoJSON. |
|
||
| News / RSS / GDELT / USGS quakes | Ingest | Out of scope for this brief. |
|
||
|
||
**Action for existing fire ingest:** NASA will stop Suomi NPP product delivery on **2026-11-01**. Switch `FIRMS_DATASET` from `VIIRS_SNPP_NRT` to `VIIRS_NOAA20_NRT` and/or `VIIRS_NOAA21_NRT` before then.[20]
|
||
|
||
---
|
||
|
||
## 1. Architecture rules (backend + frontend)
|
||
|
||
Moving objects (aircraft, vessels, trains) and alerts are **vectors**. Radar / satellite / GIBS fire tiles are **rasters**.
|
||
|
||
| Kind | Where it runs | Persist? | API shape |
|
||
|---|---|---|---|
|
||
| Radar, GOES, GIBS, OpenRailwayMap tiles | **Browser** `L.tileLayer` (CORS `*` on the recommended ones) | No | Tile URL template |
|
||
| Aircraft / trains / last-known ships | **Backend poll** → last-known row → `GET /api/{layer}?bbox=` | Last-known only (Timescale, 1-day chunks, drop after 24–72 h) | JSON array of points |
|
||
| AIS live stream | **Backend WebSocket** (aisstream forbids browsers)[5] | Last-known + optional 15-min trail | Same `/api/vessels` |
|
||
| NWS / SPC polygons | Backend poll **or** browser (NWS CORS `*`)[17] | Optional 24 h | GeoJSON FeatureCollection |
|
||
| Official JPEG/HLS cameras | Existing `cameras` scraper | Yes, same table | Existing `/stream` + HLS |
|
||
|
||
**Do not** hit third-party APIs from the Leaflet client when: (a) an API key would leak, (b) CORS is closed, (c) the provider rate-limits by IP and many map users would share one Pi egress. Proxy those through FastAPI.
|
||
|
||
**Pi / home-uplink constraints:** viewport bbox only. Never poll global ADS-B or global AIS. Cap aircraft poll to 5–10 s, trains 15–30 s, FIRMS already 900 s. Do not store 1 Hz AIS history.
|
||
|
||
Suggested NATS subjects (match `events.fire`): `events.aircraft`, `events.vessel`, `events.train`, `events.alert`.
|
||
|
||
Suggested last-known tables (natural keys):
|
||
|
||
```
|
||
aircraft(icao24 PK, callsign, lat, lon, alt_baro_m, gs_kt, track_deg, squawk, src, seen_at)
|
||
vessels(mmsi PK, name, lat, lon, sog_kt, cog_deg, navstat, dest, src, seen_at)
|
||
trains(train_id PK, route, lat, lon, heading, late_min, src, seen_at)
|
||
```
|
||
|
||
---
|
||
|
||
## 2. Build order (what to ship first)
|
||
|
||
### Phase A — frontend-only, no new tables (1–2 days)
|
||
|
||
1. RainViewer radar overlay (global, no key, CORS `*`).[13][14]
|
||
2. IEM NEXRAD CONUS overlay as US-quality alternative (no key, CORS `*`).[15]
|
||
3. GIBS `VIIRS_SNPP_Thermal_Anomalies_375m_All` fire-pixel overlay (already documented in `docs/firms.md`).
|
||
4. NWS active alerts GeoJSON overlay (no key, User-Agent required, CORS `*`).[17]
|
||
5. IEM current storm-based warnings GeoJSON.[16]
|
||
|
||
### Phase B — backend last-known overlays
|
||
|
||
6. ADSB.lol viewport aircraft (no key today).[3]
|
||
7. OpenSky bbox as fallback / research-grade (anonymous works; OAuth recommended).[1]
|
||
8. Amtraker trains (no key; **must** send identifying `User-Agent`).[9]
|
||
9. WFIGS incident points + current perimeters (no key).[21]
|
||
10. Caltrans CWWP2 cameras into existing `cameras` scraper (JPEG + HLS).[27]
|
||
|
||
### Phase C — free keys, streams
|
||
|
||
11. AISStream WebSocket (free GitHub-login key; **server-side only**).[5]
|
||
12. Optional OpenSky OAuth client for 4,000 credits/day vs 400 anonymous.[1]
|
||
13. Optional Transitland key later for multi-agency GTFS-RT (10k REST/month free).[12]
|
||
|
||
Skip paid: MarineTraffic, ADS-B Exchange, FlightAware, TrafficLand, OpenWeatherMap One Call paid tiers.
|
||
|
||
---
|
||
|
||
## 3. Air traffic
|
||
|
||
### 3.1 ADSB.lol — **recommended primary for the map**
|
||
|
||
| | |
|
||
|---|---|
|
||
| Cost / key | Free, no key today. Docs say a feeder key *may* be required in the future; contact them before treating this as a production SLA.[3] |
|
||
| License | ODbL 1.0 (same family as OSM; share-alike on the database).[4] |
|
||
| CORS | Treat as **backend-only**. Do not assume browser CORS. |
|
||
| Live probe 2026-08-27 | `GET https://api.adsb.lol/v2/lat/35.88/lon/-78.79/dist/50` → HTTP 200, 27 aircraft near RDU. |
|
||
|
||
Useful endpoints:[3]
|
||
|
||
```
|
||
GET /v2/lat/{lat}/lon/{lon}/dist/{radius_nm} # radius cap 250 nm
|
||
GET /v2/point/{lat}/{lon}/{radius}
|
||
GET /v2/sqk/{squawk} # e.g. 7700/7600/7500
|
||
GET /v2/mil
|
||
GET /v2/hex/{icao}
|
||
GET /v2/callsign/{callsign}
|
||
```
|
||
|
||
Sample fields from live payload: `hex, flight, r, t, lat, lon, alt_baro, gs, track, squawk, emergency, category, seen_pos`.
|
||
|
||
**Backend:** poll the **map viewport** (center + radius from bbox, clamp ≤ 150 nm) every 5–10 s. Dedup on `hex`. Do not poll `/v2/mil` on a timer unless the user toggles a mil filter.
|
||
|
||
**Frontend:** circleMarker or rotating icon, tooltip = `flight`/`hex`/`alt`. Cluster above ~200 markers. Color by altitude or `emergency != none`.
|
||
|
||
### 3.2 OpenSky Network — research-grade fallback
|
||
|
||
| | |
|
||
|---|---|
|
||
| Cost / key | Free for non-commercial / research. Anonymous OK. Authenticated uses OAuth2 **client credentials** (not basic auth). Create `client_id` + `client_secret` on the account page.[1][26] |
|
||
| Terms | Cite the OpenSky paper + URL for publications. Commercial needs a license. They may block AWS/hyperscaler IPs.[2] |
|
||
| Live probe | `GET /api/states/all?lamin=33.8&lomin=-84.5&lamax=36.6&lomax=-75.4` → HTTP 200, **270** state vectors over NC in 0.63 s. |
|
||
|
||
Root: `https://opensky-network.org/api`[1]
|
||
|
||
```
|
||
GET /states/all?lamin=&lomin=&lamax=&lomax=&extended=1
|
||
```
|
||
|
||
State vector indexes: `0 icao24, 1 callsign, 2 origin_country, 5 lon, 6 lat, 7 baro_alt m, 8 on_ground, 9 velocity m/s, 10 true_track, 14 squawk`.[1]
|
||
|
||
Credits (independent buckets for `/states/*`, `/tracks/*`, `/flights/*`):[1]
|
||
|
||
| Tier | Credits / refill |
|
||
|---|---|
|
||
| Anonymous | 400 / day |
|
||
| Standard user (OAuth) | 4,000 / day |
|
||
| Active feeder ≥30% uptime | 8,000 / day |
|
||
|
||
`/states/all` costs **1–4 credits** by bbox area (≤25 sq° = 1 credit; global = 4). Exhaustion → HTTP 429 + `X-Rate-Limit-Retry-After-Seconds`.[1]
|
||
|
||
Anonymous: live only, 10 s time resolution. Authenticated: up to 1 hour of history, 5 s resolution.[1]
|
||
|
||
Token:
|
||
|
||
```
|
||
POST https://auth.opensky-network.org/auth/realms/opensky-network/protocol/openid-connect/token
|
||
grant_type=client_credentials&client_id=&client_secret=
|
||
Authorization: Bearer <access_token> # ~30 min TTL
|
||
```
|
||
|
||
**Backend:** use OpenSky as fallback when ADSB.lol 429s, or for a “research” toggle. Always send a bbox. NC-sized box is ~25 sq° → 1 credit; 10 s poll as anonymous = 8,640 credits/day → **will 429**. Authenticated 4,000/day ≈ one poll every ~22 s if cost=1. Prefer ADSB.lol for the live map; keep OpenSky for sparse/historical.
|
||
|
||
**Do not** run this from AWS/the Pi if OpenSky starts blocking the IP; the docs warn about hyperscalers.[2]
|
||
|
||
### 3.3 AviationWeather.gov METARs (context, not tracks)
|
||
|
||
No key. User-Agent required. **100 req/min**. **No CORS** — backend only.[22]
|
||
|
||
```
|
||
GET https://aviationweather.gov/api/data/metar?ids=KRDU&format=json
|
||
```
|
||
|
||
Live probe: KRDU thunderstorm METAR returned JSON with `lat/lon/temp/rawOb`. Use as airport popup, not as an aircraft layer. Prefer the gzip cache files for bulk.
|
||
|
||
### 3.4 Skip
|
||
|
||
- ADS-B Exchange — paid gateway / API key.
|
||
- FlightAware / AeroAPI — paid.
|
||
- OpenSky Trino historical — overkill for the live map.
|
||
|
||
---
|
||
|
||
## 4. Boat / AIS traffic
|
||
|
||
### 4.1 AISStream — **recommended live ships**
|
||
|
||
| | |
|
||
|---|---|
|
||
| Cost / key | Free. Sign in with GitHub, create key at `/account`. Shown once; rotate from Account.[5][6] |
|
||
| Transport | `wss://stream.aisstream.io/v0/stream` |
|
||
| Browser | **Forbidden.** Connect from the FastAPI/ingest container and proxy a downsampled last-known feed.[5] |
|
||
| Limits | 3 connections / account, 3 / IP. Subscribe JSON **within 3 seconds** or the socket closes. Subscription replace ≤ 1/s. No SLA, no replay.[5] |
|
||
| Compression | Negotiate `permessage-deflate`. Uncompressed connections get bandwidth caps starting **September 2026**.[5] |
|
||
|
||
Subscribe (bbox is **\[\[lat, lon\], \[lat, lon\]\]** — not lon/lat):[5]
|
||
|
||
```json
|
||
{
|
||
"APIKey": "<AISSTREAM_API_KEY>",
|
||
"BoundingBoxes": [[[33.8, -84.5], [36.6, -75.4]]],
|
||
"FilterMessageTypes": [
|
||
"PositionReport",
|
||
"StandardClassBPositionReport",
|
||
"ExtendedClassBPositionReport",
|
||
"ShipStaticData"
|
||
]
|
||
}
|
||
```
|
||
|
||
Frames are **binary WebSocket** containing UTF-8 JSON. Decode bytes, then `json.loads`.[5]
|
||
|
||
Position envelope (docs + site examples): `MessageType`, `MetaData.{MMSI,ShipName,Latitude,Longitude}`, `Message.PositionReport.{Sog,Cog,TrueHeading,NavigationalStatus}`.[5][6]
|
||
|
||
**Backend:** one long-lived WS in the `ingest` profile. Filter to the dashboard’s configured bbox (default CONUS coasts + Great Lakes, not world). Upsert last-known by MMSI. Expose `GET /api/vessels?bbox=`. Reconnect with jittered backoff; resend full subscription in <3 s. Do not fan the raw stream to browsers.
|
||
|
||
**Frontend:** circleMarker, color by nav status / SOG, tooltip name+MMSI. Optional 5-minute trail from in-memory ring, not DB.
|
||
|
||
### 4.2 AISHub — only if you feed a receiver
|
||
|
||
Free **if you contribute a live AIS UDP feed**. Then JSON/XML/CSV at most **once per minute**.[7][8]
|
||
|
||
```
|
||
https://data.aishub.net/ws.php?username=USER&format=1&output=json&latmin=&latmax=&lonmin=&lonmax=
|
||
```
|
||
|
||
`format=1` = human-readable degrees / knots. Faster than once/minute returns empty.[8]
|
||
|
||
Not useful on the Pi unless Lance later stands up an AIS dongle. Keep as a footnote.
|
||
|
||
### 4.3 Skip / later
|
||
|
||
- MarineTraffic REST — commercial.
|
||
- Global Fishing Watch — free **non-commercial** token, fishing-effort tiles / vessel identity, not a general AIS live layer.[24] Optional Phase D for a “fishing effort” overlay.
|
||
- AISHub without a feeder — no data.
|
||
|
||
---
|
||
|
||
## 5. Train traffic
|
||
|
||
There is **no free global live train API**. US intercity is solvable; European freight is not, cheaply.
|
||
|
||
### 5.1 Amtraker — **recommended US passenger trains**
|
||
|
||
Community API that rehosts Amtrak + Brightline + VIA Rail. Free. **Identifying `User-Agent` required or the request is blocked.**[9] Data license ODC-By 1.0; attribute “Amtraker” on the map (collapsed OSM-style credit is OK).[9][10]
|
||
|
||
Live probe: `GET https://api.amtraker.com/v3/trains` and `https://api-v3.amtraker.com/v3/trains` both HTTP 200, **~1.3 MB**, **197 train numbers**. Sample: Sunset Limited `#1`, `lat/lon`, `heading`, `stations[]`, `iconColor`.
|
||
|
||
```
|
||
GET https://api.amtraker.com/v3/trains
|
||
GET https://api.amtraker.com/v3/trains/:trainId # e.g. 5-9
|
||
GET https://api.amtraker.com/v3/stations
|
||
GET https://api.amtraker.com/v3/stale # stale=true if Amtrak >15 min old
|
||
```
|
||
|
||
Poll **15–30 s**. Response is `{ "1": [Train, ...], "5": [...], ... }` — flatten to one row per `trainID`. Do not hammer; the author asks not to spam.[10]
|
||
|
||
**Frontend:** train icon, color from `iconColor` (or compute lateness). Popup: route, number, next station, delay.
|
||
|
||
### 5.2 Transitland — multi-agency GTFS-RT (optional, key)
|
||
|
||
Free plan: **10,000 REST queries/month**, 100k vector tiles/month, 1,000 routing/month. Key via Interline signup (`apikey=` query or header; **not** `Authorization`). Attribution required.[11][12]
|
||
|
||
```
|
||
https://transit.land/api/v2/rest/
|
||
```
|
||
|
||
Use later if you want commuter rail / subway vehicle positions (LA Metro, MTA, etc.). 10k/month is only ~13 queries/hour — **cannot** poll every agency every 15 s. Cache GTFS-RT protobufs yourself from the **agency’s own feed URL** (those are usually free, no Transitland meter). Transitland is a directory; the live positions should come from the agency GTFS-RT URL listed in the Atlas.
|
||
|
||
### 5.3 OpenRailwayMap — infrastructure tiles only
|
||
|
||
`https://tiles.openrailwaymap.org/{style}/{z}/{x}/{y}.png` — OSM-derived tracks, not live trains.[25] This probe’s HEAD got **HTTP 403** (UA/hotlink?). Treat as optional overlay; do not depend on it. No vehicle data.
|
||
|
||
### 5.4 Skip
|
||
|
||
- Official Amtrak Track-A-Train JSON — unofficial reverse-engineering; Amtraker already does this legally enough for a hobby map, with a documented UA policy.
|
||
- Deutsche Bahn / Network Rail — free-ish developer portals but EU-centric and key + ToS heavy. Out of scope unless the map goes global-rail.
|
||
|
||
---
|
||
|
||
## 6. Open video / camera feeds (official public only)
|
||
|
||
Do **not** add Insecam-style random IP cams as a new source. The scraper already has a public list + masscan; this section is **agency-published** JPEG/HLS.
|
||
|
||
### 6.1 Already wired
|
||
|
||
- ALERTWest / ALERTCalifornia: `https://api.cdn.prod.alertwest.com/api/getCameraDataByLoc` — live probe HTTP 200, **6.6 MB** JSON with `data.locs` + `data.cams`. Snapshots at `https://img.cdn.prod.alertwest.com/data/img/{cid}/...` (see `parse_alertwest_json`).
|
||
- Live-Environment-Streams GeoJSON (HLS / YouTube / JPEG).
|
||
- `public-ip-cams` README list.
|
||
|
||
### 6.2 Caltrans CWWP2 — **add to CAMERA_SOURCE_URLS**
|
||
|
||
Official CA DOT CCTV. No key.
|
||
|
||
```
|
||
https://cwwp2.dot.ca.gov/data/d{N}/cctv/cctvStatusD{NN}.json
|
||
```
|
||
|
||
Live probe D4: HTTP 200, **2.4 MB**, **749 cameras**.
|
||
|
||
Each `data[].cctv`:
|
||
|
||
- `location.{latitude,longitude,locationName,nearbyPlace,district,route,county,direction}`
|
||
- `inService` (`"true"`/`"false"`)
|
||
- `imageData.static.currentImageURL` — JPEG, updates ~5 s (`https://cwwp2.dot.ca.gov/data/d4/cctv/image/...jpg`)
|
||
- `imageData.streamingVideoURL` — **HLS** `https://wzmedia.dot.ca.gov/D4/....stream/playlist.m3u8`
|
||
|
||
Districts 1–12 exist with the same schema (D4 verified). Backend: new parser next to `parse_alertwest_json`, skip `inService != true`, store JPEG as `snapshot_url`, HLS as stream URL (dashboard already vendors `hls.min.js`). Frontend: same camera popup; prefer JPEG for map thumbs, HLS in the preview pane. Never RTSP.
|
||
|
||
Politeness: scrape hourly like other sources (`CAMERA_SCRAPE_INTERVAL=3600`). Do not scrape every JPEG every cycle — store URLs, let the snapshot cache fetch on demand.
|
||
|
||
### 6.3 IEM webcams — small, Iowa-centric bonus
|
||
|
||
```
|
||
GET https://mesonet.agron.iastate.edu/geojson/webcam.geojson
|
||
```
|
||
|
||
No key, CORS permissive.[23] Live probe: HTTP 200, **3** cameras at probe time (not a national network). Fields: `cid, name, imgurl, utc_valid, state`. Nice extra source, low value vs ALERTWest/Caltrans.
|
||
|
||
### 6.4 YouTube live (already in Live-Environment-Streams)
|
||
|
||
Do not call the YouTube Data API unless you want search. Embedding existing stream URLs from the GeoJSON is enough. YouTube Data API **requires a Google key** and quota.
|
||
|
||
### 6.5 Skip
|
||
|
||
- Insecam / random “public IP cam” aggregators — ToS / privacy / already covered by masscan ethics.
|
||
- TrafficLand, EarthCam commercial APIs.
|
||
- SkylineWebcams — scraping, not an API.
|
||
|
||
---
|
||
|
||
## 7. Fire maps (beyond current FIRMS points)
|
||
|
||
FIRMS CSV = hotspot **points** (already ingested). WFIGS = **incident identity + perimeters**. GIBS = **pixel overlay**. Use all three; they answer different questions.
|
||
|
||
### 7.1 WFIGS current incident locations — **recommended vector layer**
|
||
|
||
No key. ArcGIS FeatureServer, GeoJSON.
|
||
|
||
```
|
||
https://services3.arcgis.com/T4QMspbfLg3qTGWY/arcgis/rest/services/WFIGS_Incident_Locations_Current/FeatureServer/0/query
|
||
?where=1=1&outFields=IncidentName,IncidentSize,FireDiscoveryDateTime,POOState,PercentContained,IncidentTypeCategory,FireCause
|
||
&outSR=4326&f=geojson
|
||
```
|
||
|
||
Live probe: **635** current locations. Sample: `Foster Bridge`, 675 acres, `US-FL`, 100% contained, cause Human. `FireDiscoveryDateTime` is **epoch ms**.
|
||
|
||
Poll 5–15 min (NIFC says related perimeter service refreshes ~5 min).[21] Backend can proxy `GET /api/fire-incidents` so the keyless URL is cached. Frontend: labeled point, popup acres / contained %.
|
||
|
||
Some similarly named services (`Current_WildlandFire_Locations`) return **Token Required** — stick to `WFIGS_Incident_Locations_Current`.
|
||
|
||
### 7.2 WFIGS current perimeters — **recommended polygon overlay**
|
||
|
||
```
|
||
https://services3.arcgis.com/T4QMspbfLg3qTGWY/arcgis/rest/services/WFIGS_Interagency_Perimeters_Current/FeatureServer/0/query
|
||
?where=1=1&outFields=poly_IncidentName,poly_GISAcres,attr_IncidentSize,attr_PercentContained,attr_FireDiscoveryDateTime
|
||
&outSR=4326&f=geojson
|
||
```
|
||
|
||
Live probe: **236** current perimeters. Historic `WFIGS_Interagency_Perimeters` counted **41,078** — do not fetch that unfiltered. Use Current only, or `resultRecordCount` + bbox `geometry` + `spatialRel=esriSpatialRelIntersects`.
|
||
|
||
Authoritative US wildland-fire polygons; not every incident has a perimeter. Refresh ~5 min, fall-off rules drop stale small fires.[21]
|
||
|
||
**Frontend:** `L.geoJSON` fill red/orange by acres. Click for name/acres. This is the layer that makes FIRMS dots make sense.
|
||
|
||
### 7.3 GIBS thermal tiles (no key, no storage)
|
||
|
||
Already in `docs/firms.md`. Leaflet:
|
||
|
||
```
|
||
https://gibs.earthdata.nasa.gov/wmts/epsg3857/best/VIIRS_SNPP_Thermal_Anomalies_375m_All/default/{time}/GoogleMapsCompatible_Level{z}/{y}/{x}.png
|
||
```
|
||
|
||
Max zoom ~9. `{time}` = `YYYY-MM-DD`. Same GIBS CORS `*` as existing basemaps. Use as a toggle under FIRMS points.
|
||
|
||
### 7.4 FIRMS dataset note
|
||
|
||
Keep the CSV ingest. Add NOAA-20/21 datasets. Ultra-real-time US/Canada is available on those products (seconds-scale for much of US/Canada).[20]
|
||
|
||
---
|
||
|
||
## 8. Weather radar + alerts
|
||
|
||
### 8.1 RainViewer — **recommended global radar overlay**
|
||
|
||
No key. Personal / educational / small community. Attribution: “Weather data by RainViewer” + link.[13]
|
||
|
||
Live probe: `GET https://api.rainviewer.com/public/weather-maps.json` → version `2.0`, host `https://tilecache.rainviewer.com`, **13** past frames (~2 h, 10 min steps). Tile HEAD: HTTP 200, CORS `*`, **`x-ratelimit-limit: 500` / 60 s**, burst 300.
|
||
|
||
Tile URL:[14]
|
||
|
||
```
|
||
{host}{path}/{size}/{z}/{x}/{y}/{color}/{smooth}_{snow}.png
|
||
```
|
||
|
||
Example: `https://tilecache.rainviewer.com/v2/radar/cb581daa2c0f/256/{z}/{x}/{y}/2/1_1.png`
|
||
|
||
- `{size}` 256 or 512
|
||
- `{color}` scheme id (2 is a common default)
|
||
- `{smooth}_{snow}` e.g. `1_1`
|
||
- **max zoom 7** (coarse vs IEM)
|
||
|
||
**Frontend:** fetch the JSON every 5 min, set `L.tileLayer` to the latest `past[].path`. Optional animation through `past` frames. No backend.
|
||
|
||
### 8.2 IEM NEXRAD / MRMS — **recommended CONUS quality radar**
|
||
|
||
Iowa Environmental Mesonet. No key. Free including commercial, as-is, no SLA.[16] CORS `*` on tiles (live HEAD confirmed).
|
||
|
||
Current CONUS base reflectivity TMS:[15]
|
||
|
||
```
|
||
https://mesonet.agron.iastate.edu/cache/tile.py/1.0.0/nexrad-n0q/{z}/{x}/{y}.png
|
||
```
|
||
|
||
Cache-Control 5 min. Aliases `mesonet1/2/3.agron.iastate.edu` for parallel browser tile fetches.[15]
|
||
|
||
Loop (past 5–55 min, step 5): layer `nexrad-n0q-mXXm` e.g. `nexrad-n0q-m05m`.
|
||
|
||
Also useful:
|
||
|
||
| Layer | Meaning |
|
||
|---|---|
|
||
| `nexrad-eet` | Echo tops |
|
||
| `q2-hsr` | MRMS seamless hybrid-scan reflectivity |
|
||
| `q2-n1p` / `q2-p24h` | MRMS 1 h / 24 h precip |
|
||
| `goes_east_conus_ch02` | GOES-East visible |
|
||
| `goes_east_conus_ch13` | GOES-East IR |
|
||
|
||
WMS alternative: `https://mesonet.agron.iastate.edu/cgi-bin/wms/nexrad/n0q.cgi?` — TMS is simpler for Leaflet.
|
||
|
||
**Frontend:** use IEM as default when the map is over CONUS (higher res than RainViewer z7); RainViewer when zoomed outside the US or as a global fallback.
|
||
|
||
### 8.3 NWS API — alerts / forecasts (no radar tiles)
|
||
|
||
`https://api.weather.gov` — US public domain, no fee. **User-Agent required** (app + contact). Rate limit unpublished; 429s clear in ~5 s. CORS `*`.[17]
|
||
|
||
Live probe: `GET /alerts/active?area=NC` → HTTP 200 GeoJSON, 4 alerts, CORS `*`.
|
||
|
||
```
|
||
GET https://api.weather.gov/alerts/active?area=NC
|
||
GET https://api.weather.gov/alerts/active?point={lat},{lon}
|
||
GET https://api.weather.gov/points/{lat},{lon} # then follow forecast URLs
|
||
```
|
||
|
||
**Frontend can hit this directly** (CORS open) **or** backend can cache 30 s to protect the Pi if many tabs are open. Prefer backend cache. Draw `features[].geometry` with color by `properties.severity` / `event`.
|
||
|
||
### 8.4 IEM storm-based warnings GeoJSON
|
||
|
||
```
|
||
GET https://mesonet.agron.iastate.edu/geojson/sbw.geojson
|
||
```
|
||
|
||
Live probe: **46** current polygons. Sample: Severe Thunderstorm Warning, WFO BGM. Good companion to NWS alerts (polygon-accurate).
|
||
|
||
### 8.5 Open-Meteo — point forecast, not radar
|
||
|
||
No key for non-commercial **< 10,000 calls/day**. CC BY 4.0 data, attribution required. Commercial needs a plan or self-host.[18][19]
|
||
|
||
```
|
||
GET https://api.open-meteo.com/v1/forecast?latitude=35.88&longitude=-78.79¤t=temperature_2m,precipitation,weather_code,wind_speed_10m
|
||
```
|
||
|
||
Live probe: RDU 28.4 °C, weather_code 80 (rain showers). Use for click-to-forecast popups, not a map overlay. Cache by grid cell.
|
||
|
||
### 8.6 NHC active storms
|
||
|
||
```
|
||
GET https://www.nhc.noaa.gov/CurrentStorms.json
|
||
```
|
||
|
||
Live probe: HTTP 200, `activeStorms[]` — Tropical Storm Dolly (`al042026`, 13.6N 38.7W, 35 kt). No key. Plot as a hurricane marker + optional NHC cone GeoJSON (separate NHC GIS products). Nice extra during Atlantic season.
|
||
|
||
---
|
||
|
||
## 9. Keys / env vars to add
|
||
|
||
Store new secrets in the existing Keys UI (`api_keys` table) **and** `.env` fallback, same planned pattern as FIRMS.
|
||
|
||
| Var | Required for | How to get | Cost |
|
||
|---|---|---|---|
|
||
| `FIRMS_MAP_KEY` | Existing fire CSV | https://firms.modaps.eosdis.nasa.gov/api/map_key_info/ | Free |
|
||
| `OPENSKY_CLIENT_ID` / `OPENSKY_CLIENT_SECRET` | OpenSky authenticated | OpenSky account → API Client | Free |
|
||
| `AISSTREAM_API_KEY` | Live AIS | GitHub login at aisstream.io/account | Free |
|
||
| `TRANSITLAND_API_KEY` | Transitland REST (optional) | Interline free plan | Free 10k/mo |
|
||
| `GFW_TOKEN` | Fishing-effort overlay (optional) | globalfishingwatch.org/our-apis/tokens | Free, non-commercial |
|
||
| *(none)* | ADSB.lol, Amtraker, RainViewer, IEM, NWS, WFIGS, Caltrans, Open-Meteo, NHC | — | Free |
|
||
|
||
Amtraker: no key, but set `OSINT_USER_AGENT` to something like `osint-dashboard/1.0 (lancewalters94@gmail.com)` — already have `USER_AGENT` in `camera_config.py`; reuse it on **all** outbound HTTP.
|
||
|
||
NWS: same User-Agent header is mandatory.[17]
|
||
|
||
---
|
||
|
||
## 10. Suggested API surface (backend engineer)
|
||
|
||
Keep FIRMS `GET /api/fires`. Add:
|
||
|
||
```
|
||
GET /api/aircraft?bbox=minlon,minlat,maxlon,maxlat
|
||
GET /api/vessels?bbox=...
|
||
GET /api/trains?bbox=...
|
||
GET /api/fire-incidents?bbox=... # WFIGS points
|
||
GET /api/fire-perimeters?bbox=... # WFIGS polygons (or proxy GeoJSON)
|
||
GET /api/alerts?area=NC|&bbox=... # cached NWS + IEM SBW
|
||
GET /api/map/radar # { provider, tileUrl, frames[] } for RainViewer metadata
|
||
GET /api/map/layers # already have GIBS; extend with radar/fire tile templates
|
||
```
|
||
|
||
All vector endpoints: default `limit=2000`, require bbox except trains (Amtrak is ~200 rows total). Return arrays of `{id, lat, lon, heading, speed, label, extra}` so the frontend can share one marker renderer.
|
||
|
||
Do **not** proxy RainViewer/IEM/GIBS tiles through the Pi (bandwidth). Return the template; browser fetches tiles.
|
||
|
||
AISStream stays inside `ingest` profile (needs a long-lived process, like FIRMS). Aircraft/trains can live in `run_ingester.py` loops.
|
||
|
||
---
|
||
|
||
## 11. Suggested UI (frontend bot)
|
||
|
||
Layer toggles on the existing Leaflet map (same pattern as GIBS + fire heatmap):
|
||
|
||
| Toggle | Default | Source |
|
||
|---|---|---|
|
||
| Radar | on (CONUS) | IEM `nexrad-n0q` if view intersects US, else RainViewer |
|
||
| NWS alerts | on | polygons |
|
||
| FIRMS heat | existing | existing |
|
||
| WFIGS perimeters | on | polygons |
|
||
| WFIGS incidents | off | points |
|
||
| Aircraft | on | `/api/aircraft` |
|
||
| Vessels | off until AIS key | `/api/vessels` |
|
||
| Trains | on | `/api/trains` |
|
||
| Cameras | existing | existing |
|
||
|
||
Viewport-driven: on `moveend` (debounced 300 ms), refetch aircraft/vessels for the new bbox. Do not refetch radar tiles except RainViewer frame JSON every 5 min.
|
||
|
||
Marker performance: canvas renderer (`L.canvas()`), cluster at zoom < 7, hide labels until zoom ≥ 8.
|
||
|
||
Camera preview: Caltrans HLS via existing hls.js path; JPEG thumb from `currentImageURL`. Still never emit RTSP.
|
||
|
||
Attribution bar (required): OpenSky / ADSB.lol ODbL / Amtraker / RainViewer / IEM / NWS / NASA FIRMS-GIBS / NIFC WFIGS / OSM as applicable.
|
||
|
||
---
|
||
|
||
## 12. Legal / ethics (non-negotiable)
|
||
|
||
- Masscan / RTSP policy unchanged.
|
||
- AISStream: server-side only; do not put the key in JS.[5]
|
||
- OpenSky: non-commercial unless licensed; cite if you publish.[2]
|
||
- ADSB.lol: ODbL share-alike on derived databases.[4]
|
||
- Amtraker: ODC-By, visible credit, identifying UA.[9]
|
||
- RainViewer: personal/educational; credit + link.[13]
|
||
- Open-Meteo: non-commercial 10k/day on the public instance.[18]
|
||
- GFW: non-commercial.[24]
|
||
- Caltrans / NWS / NIFC / NASA / IEM: US public or explicitly open; still cache politely.
|
||
- Do not scrape MarineTraffic / FlightRadar24 / FlightAware map tiles.
|
||
|
||
---
|
||
|
||
## 13. Verification log (this research session)
|
||
|
||
| Endpoint | Result |
|
||
|---|---|
|
||
| OpenSky `/states/all` NC bbox | 200, 270 aircraft, 0.63 s |
|
||
| ADSB.lol `/v2/lat/35.88/lon/-78.79/dist/50` | 200, 27 aircraft |
|
||
| RainViewer `weather-maps.json` | 200, 13 past frames |
|
||
| RainViewer tile | 200, CORS `*`, 500 req/min header |
|
||
| Open-Meteo forecast RDU | 200, 28.4 °C, code 80 |
|
||
| NWS `/alerts/active?area=NC` | 200, 4 alerts, CORS `*` |
|
||
| IEM `nexrad-n0q` tile | 200, CORS `*`, max-age 300 |
|
||
| IEM `sbw.geojson` | 200, 46 polygons |
|
||
| IEM `webcam.geojson` | 200, 3 cams |
|
||
| WFIGS locations count | 635 |
|
||
| WFIGS current perimeters count | 236 |
|
||
| ALERTWest `getCameraDataByLoc` | 200, 6.6 MB |
|
||
| Caltrans D4 CCTV JSON | 200, 749 cams, JPEG + HLS |
|
||
| Amtraker `/v3/trains` | 200, ~197 numbers, 1.3 MB |
|
||
| AviationWeather METAR KRDU | 200 JSON |
|
||
| NHC `CurrentStorms.json` | 200, TS Dolly |
|
||
| OpenRailwayMap tile HEAD | **403** — do not rely on |
|
||
|
||
---
|
||
|
||
## 14. Concrete next tickets
|
||
|
||
**Backend**
|
||
|
||
1. `FIRMS_DATASET` dual-write NOAA-20/21; document S-NPP sunset.
|
||
2. `aircraft` ingest: ADSB.lol viewport poll + `/api/aircraft`.
|
||
3. `trains` ingest: Amtraker 30 s poll + User-Agent + `/api/trains`.
|
||
4. Camera parser: Caltrans CWWP2 D1–D12 → existing `cameras` rows (`discovery_source=caltrans`).
|
||
5. WFIGS proxy endpoints (locations + current perimeters), 5–15 min cache.
|
||
6. AISStream worker behind `AISSTREAM_API_KEY` (idle if unset, same as FIRMS).
|
||
7. `/api/alerts` 30 s cache of NWS + IEM SBW.
|
||
|
||
**Frontend**
|
||
|
||
1. Layer control: radar (IEM/RainViewer), alerts, WFIGS, aircraft, trains, vessels.
|
||
2. Shared point-layer component (id, lat, lon, heading, label).
|
||
3. RainViewer frame JSON → animated overlay.
|
||
4. Caltrans HLS in existing preview pane.
|
||
5. Attribution strip.
|
||
|
||
**Do not start with:** Transitland, GFW, AISHub, OpenSky OAuth (until ADSB.lol is live and still needs a fallback).
|
||
|
||
## Sources
|
||
|
||
[1] https://openskynetwork.github.io/opensky-api/rest.html
|
||
[2] https://openskynetwork.github.io/opensky-api
|
||
[3] https://api.adsb.lol
|
||
[4] https://www.adsb.lol/docs/open-data/api
|
||
[5] https://aisstream.io/documentation
|
||
[6] https://aisstream.io
|
||
[7] https://www.aishub.net
|
||
[8] https://www.aishub.net/api
|
||
[9] https://github.com/piemadd/amtrak
|
||
[10] https://amtraker.com/about
|
||
[11] https://www.transit.land/documentation
|
||
[12] https://www.transit.land/plans-pricing
|
||
[13] https://www.rainviewer.com/api.html
|
||
[14] https://www.rainviewer.com/api/weather-maps-api.html
|
||
[15] https://mesonet.agron.iastate.edu/ogc
|
||
[16] https://mesonet.agron.iastate.edu/api
|
||
[17] https://www.weather.gov/documentation/services-web-API
|
||
[18] https://open-meteo.com
|
||
[19] https://open-meteo.com/en/docs
|
||
[20] https://firms.modaps.eosdis.nasa.gov/api/area
|
||
[21] https://data-nifc.opendata.arcgis.com/datasets/nifc::wfigs-current-interagency-fire-perimeters/about
|
||
[22] https://aviationweather.gov/data/api
|
||
[23] https://mesonet.agron.iastate.edu/geojson/webcam.py?help
|
||
[24] https://globalfishingwatch.org/our-apis/documentation
|
||
[25] https://openrailwaymap.org
|
||
[26] https://opensky-network.org/about/faq
|
||
[27] https://cwwp2.dot.ca.gov/vm/iframemap.htm
|