Off-viewport clients that send {type:watch_geofences,ids} on /ws/live
still receive geofence_alert; AIS/ADS-B stay viewport-only.
GET /api/geofence-alerts accepts geofence_id/since/until/source_kind.
GET /api/geofences/{id}/at returns CAGG+FIRMS inside the fence at T
(404 if missing, empty lists if DB down). DELETE missing fences 404s.
Reject private/loopback/link-local hosts on POST /api/ingest/rss and
URL-shaped GDELT queries via camera_scraper.is_public_url (HTTP 400).
PATCH /api/sources/{id} only accepts name, url, config, enabled (422 else).
Pin titiler to the Pi-running digest, make workers=1 explicit for
in-memory WS/layer caches, and add PostGIS GIST indexes for events/fires
bbox pans without dropping the existing btree indexes.
GET /api/satellites?groups=stations,weather returns last-known satellite
positions propagated from CelesTrak GP JSON (OMM mean elements, not TLE)
via the real sgp4 library (TEME->geodetic), not two-body Kepler.
- Fetch gp.php?GROUP=...&FORMAT=JSON at most once per 2h per group,
cached with a last-good blob fallback for 403 / stale responses.
- SatNOGS TLE fallback only when the CelesTrak cache is empty.
- bbox viewport culling; unknown group -> 400.
- overlay_catalog() gains id=satellites (kind=points, /api/satellites).
- OMM path handles NORAD cat numbers >= 100000 (no TLE round-trip).
Right-click (long-press on touch) opens a What's here? HUD panel: reverse
geocode via GET /api/place (honest UA, 1 req/s, 60s/500-key cache) and lists
already-loaded cameras, aircraft, vessels, fires, and alerts within 5 km.
Curated static catalogue of 13 conflict theatres (Ukraine, Gaza, Sudan,
Myanmar, DRC, Yemen, Syria, Lebanon, Sahel, Somalia, Red Sea, Taiwan
Strait, Korean DMZ) with war|high|elevated severity and short factual
descriptions. GET /api/conflicts returns { zones: [{id,label,severity,
lat,lon,description,eventCount,lastUpdated}], timestamp } where
eventCount rolls up pre-existing geocoded news/GDELT//api/news/map rows
inside each zone bbox (0 when the tables are empty). Adds the conflicts
entry to overlay_catalog(). No upstream scraping, no jittered coords.
Geofences could be drawn but not removed. VesselAPI Hormuz dots vanished
on restart and DVR skipped between the 5 daily polls. Sentinel-1 re-hit
STAC on every pan and often painted a neighbouring swath. Executive
briefs truncated; ticker stayed empty unless something was critical.
- Layer-panel list + polygon popup DELETE /api/geofences/{id}
- Persist VesselAPI polls to vessels (UTC-day purge, DVR as-of, boot hydrate)
- Cache Sentinel-1 by 2° cell; pick covering scene; clip Leaflet tiles
- Retry truncated LLM JSON; ticker falls back to medium/low; 3-min HUD poll
- GET /api/map/chokepoints: static five-strait fly-to catalog (Hormuz, Bab
el-Mandeb, Suez, Malacca, Taiwan) with bbox/center/zoom/vesselapi. No
VesselAPI calls; only Hormuz flags vesselapi=true.
- GET /api/vessels?src=aisstream|vesselapi|all (default all) filters the
union store by provider so a Hormuz view skips ~5k CONUS AISStream rows.
- Tests: span validation of all five boxes, catalog 200 shape, src filter.
Add GET /api/map/sentinel1?bbox=... which queries Planetary Computer STAC
(sentinel-1-grd, last 7d, most recent), signs the vv/hh COG with a SAS token,
and returns a same-origin /titiler/... XYZ tile template. Cache keyed on
quantized bbox + UTC day (20 min TTL). 429 -> 429 (Retry-After), no imagery
-> 404, other upstream errors -> 502.
Self-host TiTiler on the Pi (ghcr.io/developmentseed/titiler, arm64, 1G cap,
host loopback 8001 -> container 8000) instead of titiler.xyz. nginx
/titiler/ proxy snippet routes browser tiles to it; TITILER_PUBLIC_BASE and
TITILER_INTERNAL_URL are env-driven (no hardcoded hostnames).
- keystore KEY_REGISTRY copy splits the two keys (open/shared US-coast WS vs
commercial Hormuz 5x/day REST); VesselAPI no longer reads as an AIS backup.
- vesselapi.py docstring drops 'fallback'; states both are first-class.
- /api/vessels docstring: last-known is the union (extra.src aisstream vs
vesselapi).
- tests: VesselAPI idles/polls independently of the AISStream key.
Add a server-side REST poller for VesselAPI (free tier 150 calls/mo) that
upserts last-known positions into the shared vessel_last_known store when
AISStream is unset. Default box is the Strait of Hormuz (span 3.6 <= 4 deg),
never CONUS/NC (AISStream owns US coasts).
- app/vesselapi.py: worker loop, 4deg span validator, position->marker
transform (skip suspected_glitch), durable Postgres daily-quota table.
- Local hard cap 5 successful 2xx/UTC day (VESSELAPI_MAX_CALLS_PER_DAY),
monthly floor from X-RateLimit-Remaining, single request limit=50, no
nextToken, no filter.sat, no retry-storm.
- keystore VESSELAPI_API_KEY registry entry; config + compose env passthrough
(app + ingest); wired next to AISStream in main.py lifespan + run_ingester.
- GET /api/vessels docstring notes AISStream and/or VesselAPI cache.
- tests/test_vesselapi.py: 20 unit tests (no network/DB).
Add planespotters.net latest-photo lookup for ADS-B aircraft.
- app/live_layers.py: fetch_planespotters_photo() (hex preferred, reg
fallback) + _normalize_planespotter_photo(); 24h TTL cache (their ToS cap).
- app/main.py: GET /api/aircraft/photo?hex=...|reg=... (422/404/502).
- app/static/index.html: .ps-photo block in ADS-B popup; lazy load on
popupopen; thumbnail links to photo page + photographer credit.
Server-side proxy, not browser fetch(): planespotters 403s any request
carrying an Origin header (every browser fetch() sends one). The thumbnail
binary is loaded by the browser straight from their CDN, never re-hosted.
Tests: 4 unit + 4 API contract (38 pass in the two files).
Summarizer still runs the 15-min analyst, plus a 24h breaking-news recap
at 23:00 America/New_York. HUD pins kind=daily_recap. Prompts ignore
futures/market tape.
Stop the live HUD reconnect storm (nginx WS snippet + backoff), copy
intel/nous_client into the summarizer image, and make event ingest
idempotent on URL. GDELT uses the DOC API; NWS no longer sends bbox;
FIRMS is one ON CONFLICT batch; GET /api/aircraft serves last-known.
Health reports freshness without 503ing docker. EONET + CISA KEV added.
Scraper loops with NEWS_SCRAPE_INTERVAL_S (default 10s after each
crawl). Summarizer runs every NEWS_SUMMARIZE_INTERVAL_S (default 900)
over the last 15 minutes of articles. Feed list replaced from the
k8s scrapy-urls configmap (334 sources).
The Vessels layer now retunes the server-side AISStream subscription to the
client viewport instead of a static AISSTREAM_BBOX. The frontend POSTs its
quantized viewport box to /api/vessels/subscribe on moveend; the ais_stream
worker coalesces and applies it at the service's 1 subscription/s cap, then
last-known positions for the new area arrive within a couple of seconds (the
frontend does one follow-up fetch after retuning). Bounds the in-memory
vessel store across regions. Key stays server-side.
Reuse one TLS pool for overlay upstreams (8s/3s timeouts). Gzip JSON
over 1 KB. Aircraft/vessels Cache-Control max-age=5, alerts/perimeters 30.
Lifespan replaces deprecated on_event startup.
Heatmap uses {lat,lon,i,c}. Camera list drops URLs (detail via GET
/api/cameras/{id}). Event blips skip body. News omits content unless
include_content=true.
Wire the free data streams from docs/free-data-streams.md into the
dashboard as layer-panel toggles. Third-party APIs are proxied/cached
in FastAPI; raster tiles (IEM, RainViewer, GIBS) stay in the browser.
- Aircraft via ADSB.lol viewport poll (bbox required, radius ≤ 150 nm)
- Amtraker trains, NHC storms, WFIGS incidents/perimeters
- NWS + IEM SBW as /api/weather-alerts (does not collide with /api/alerts)
- AISStream worker is server-side only and idles without AISSTREAM_API_KEY
- Caltrans CWWP2 D1–D12 camera parser; FIRMS dual-write NOAA-20/21
- Map is now the landing view: full-viewport NASA GIBS globe, existing
fires/cameras/HLS map machinery preserved verbatim
- New 'Event Blips' layer: geolocated ingest events, color-coded by source,
bbox/since/has_coords filters added to GET /api/events
- Weather / Flights(ADS-B) / Vessels(AIS) layer slots reserved (feed pending)
- Market ticker strip with configurable symbols — auto-promotes to LIVE when
GET /api/market returns {symbols:[...]} (contract documented in Settings)
- Breaking-news ticker: LLM exec-summary flash + headline marquee, 15-min cycle
- Dropdown nav (Map/News/Events/Alerts/Entities/Ingest/API Keys/Settings),
Settings view (localStorage: symbols, map layer defaults), System panel
- Section-9 theme: near-black navy, cyan/magenta accents, Orbitron/Rajdhani/
Share Tech Mono, chamfered HUD panels, scanlines, boot splash, UTC clock
- Fix: events.camera enum value missing from models.py/schemas.py caused 500s
on every events query once camera events flowed in (migration 004 added it
to the DB enum only)
Zooming in replaced the marker group with a bbox subset, so dense world
views (5k dots) thinned to a handful of pins — or zero on a wrapped world
copy, where currentBBox collapsed to a sliver and the API returned ~nothing.
Cameras only seemed to exist when zoomed out.
- vendor leaflet.markercluster 1.5.3; camera markers now render as numbered
neon-green clusters at low zoom and split into individual dots as you zoom,
so camera coverage is visible at every zoom level (spiderfy at max zoom)
- guard against null lat/lon rows (no phantom markers at 0,0)
- stale-response guard for loadCams/loadFires: rapid wheel zoom can resolve
fetches out of order and a stale bbox response blanked the map
- serve index.html with Cache-Control: no-cache so the browser never keeps
serving a pre-deploy copy after fixes land
Leaflet worldCopyJump at zoom 2 reports longitudes outside ±180; clamping
them independently produced west>=east or a thin sliver. Low zoom now
queries the full world, and the API ignores inverted bboxes.
Ingest ~4.2k direct HLS/YouTube feeds (VDOT, MDSHA, DelDOT, Iowa DOT,
OpenCCTV, etc.) from the public GeoJSON catalog. Popup plays HLS via a
CORS-safe playlist proxy + vendored hls.js, YouTube via embed, with
ffmpeg-MJPEG fallback. ALERTWest rows now use a stable camera id so
hourly scrapes do not duplicate.
Masscan open-554 hosts almost never have a public picture. The map now
defaults to cameras with an http(s) snapshot_url (~132 directory cams).
Ingest probes unauthenticated still/MJPEG paths first and skips the rest,
so dead RTSP-only hosts never land on the map. Pass working=false to see
unverified port-554 hits.
Masscan finds are rtsp:// with no snapshot_url, so the popup skipped the
<img> and the leftover source link handed the OS an rtsp:// URL (VLC).
- Popup always hits /api/cameras/{id}/snapshot (HTTP stills, then one
ffmpeg frame grab). No credentials. 10s hard timeout.
- rtsp:// is rendered as text, never as an href.
- ffmpeg added to the app image for the RTSP still/MJPEG path.
- Default MASSCAN_RATE 200 (1k/10k saturated the home uplink).
Interactive world map panel for the dashboard:
- Vendored Leaflet 1.9.4 under app/static/vendor/leaflet/ (served via
new /static mount on the FastAPI app)
- New Map tab: GIBS WMTS raster basemap (BlueMarble_ShadedRelief_Bathymetry,
VIIRS/MODIS/Aqua CorrectedReflectance_TrueColor, VIIRS_DayNightBand),
layer picker + UTC date selector (Latest/-7d/-30d quick picks)
- /api/map/layers: curated GIBS catalog (tms, format, has_time, max_zoom)
- /api/map/times: per-layer date windows from GIBS Domains XML (6h cache,
graceful 502 on GIBS hiccup; static layers 422)
- Fire hotspots (existing /api/fires) + open cameras (/api/cameras) as
toggleable bbox-scoped overlays; dark UI matching dashboard theme
Vendor the newsPipeline scraper + summarizer into the repo and wire them into
docker-compose against the EXISTING osint-db (no second Postgres), replacing
the upstream k8s CronJobs with in-compose wall-clock loops (:00 scrape, :05
summarize).
- news/scraper: vendored Scrapy project (257 RSS feeds) + hourly loop
scheduler (run_news_scraper.py)
- news/summerizer: vendored Gemini map-reduce summarizer, cleaned:
* fix broken google-genai response handling (_extract_text, defensive)
* fix malformed INSERT/GRANT query in save_summary_to_db
* OSINT-neutral default MAP_PROMPT; futures/markets language gated behind
INCLUDE_FUTURES=0 (yfinance lazy-imported)
* env-configurable model, batch size, lookback window
+ hourly loop scheduler (run_news_summarizer.py, :05)
- alembic 003_news: idempotent articles + article_summaries tables
- API: GET /api/news and GET /api/news/summaries (+ models, schemas)
- tests/test_api_news.py: 5 DB-backed contract tests (all pass vs real PG)
- docs/news.md + .env.example updates
Both services run under the `ingest` compose profile (matching the
ingester/camera-scraper pattern) and build arm64 on the Pi via the existing
Forgejo CI workflow. telebot left out of scope (reserved env only).
