Add parse_mdot_json for the MDOT MiDrive camera/list JSON where
coordinates/id live in the county field's map link (lat=/lon=/id=) and
the JPEG still lives in the image field's <img src>. Michigan bbox
filter, missing-coords drop, RTSP reject. discovery_source=mdot,
stable source_url keyed on camera id, url_hash dedupe. Wired into
scrape_source dispatch + CAMERA_SOURCE_URLS defaults.
Unit tests: HTML field extract, bbox drop, missing-coords/image drop,
malformed payload.
Geofences could be drawn but not removed. VesselAPI Hormuz dots vanished
on restart and DVR skipped between the 5 daily polls. Sentinel-1 re-hit
STAC on every pan and often painted a neighbouring swath. Executive
briefs truncated; ticker stayed empty unless something was critical.
- Layer-panel list + polygon popup DELETE /api/geofences/{id}
- Persist VesselAPI polls to vessels (UTC-day purge, DVR as-of, boot hydrate)
- Cache Sentinel-1 by 2° cell; pick covering scene; clip Leaflet tiles
- Retry truncated LLM JSON; ticker falls back to medium/low; 3-min HUD poll
CISA KEV republished 1685 NIST URLs every 5 min; FIRMS re-inserted
~325k global hotspots every 15 min. Producer now skips URLs already
in event_dedup and FIRMS CSVs that are unchanged (delta-only persist).
CI rebuilds only images whose paths changed and never pulls/rebuilds
Timescale or bounces osint-db unless Dockerfile.pg changes.
Fetch GET /api/map/chokepoints once. Desktop buttons / phone <select>
fly to Hormuz, Bab el-Mandeb, Suez, Malacca, Taiwan. Turns Vessels on,
uses ?src=vesselapi on Hormuz, enables Sentinel-1 there, and never
POSTs /api/vessels/subscribe outside CONUS.
- GET /api/map/chokepoints: static five-strait fly-to catalog (Hormuz, Bab
el-Mandeb, Suez, Malacca, Taiwan) with bbox/center/zoom/vesselapi. No
VesselAPI calls; only Hormuz flags vesselapi=true.
- GET /api/vessels?src=aisstream|vesselapi|all (default all) filters the
union store by provider so a Hormuz view skips ~5k CONUS AISStream rows.
- Tests: span validation of all five boxes, catalog 200 shape, src filter.
Live-probed 2026-08-29 inventory (GIBS extra layers, GOES/Himawari,
HLS, OPERA, STAC/COG paths). Ranked drop-ins for the OSINT map; skip
paid/NICFI. Does not change runtime code.
AISStream's live US-coast feed crowds vessel_last_known past _MAX_VESSELS
(6000) and the 'evict oldest seen_at' pass reaps the Hormuz VesselAPI rows,
which are only refreshed every VESSELAPI_INTERVAL (~4.8h). Result: the poller
was fetching 50 Hormuz vessels per 2xx but /api/vessels?bbox=Hormuz stayed
empty. Eviction now skips src=vesselapi rows (falls back to evictable-only,
clamped) so the blind-spot box stays resident between polls.
TiTiler /tiles/{z}/{x}/{y} takes a strict int for y; the @1x scale-suffix
template 422s on every SAR tile request (int_parsing on '21@1x').
sign_cog_url wrapped the Planetary Computer SAS token under a single
token= param, which Azure rejects (403→409); the token is a pre-encoded
query string and must be appended top-level (st=…&se=…&sig=…).
Proven live: correct-form URL renders a 200 image/png 256px tile.
Add a default-off Layers row that fetches GET /api/map/sentinel1?bbox=
only when enabled, then L.tileLayer(tileUrl) at opacity 0.8. HUD hints
for 404/429/502; moveend refetch while on. Same-origin tileUrl as-is.
Retune the 820px layout so GIBS sits as a full-width bar under the
header, LAYERS starts collapsed top-left, zoom stays top-right, and
the DVR row clears the ticker. Hide attribution and map-hint on phone
so NASA/RainViewer text no longer paints through MKT/NEWS.
Stage uses 100dvh (100vh fallback) plus safe-area so the scrollport
ends above the dock. Subviews get extra bottom padding; phone key
forms wrap with 44px tap targets.
Add GET /api/map/sentinel1?bbox=... which queries Planetary Computer STAC
(sentinel-1-grd, last 7d, most recent), signs the vv/hh COG with a SAS token,
and returns a same-origin /titiler/... XYZ tile template. Cache keyed on
quantized bbox + UTC day (20 min TTL). 429 -> 429 (Retry-After), no imagery
-> 404, other upstream errors -> 502.
Self-host TiTiler on the Pi (ghcr.io/developmentseed/titiler, arm64, 1G cap,
host loopback 8001 -> container 8000) instead of titiler.xyz. nginx
/titiler/ proxy snippet routes browser tiles to it; TITILER_PUBLIC_BASE and
TITILER_INTERNAL_URL are env-driven (no hardcoded hostnames).
- keystore KEY_REGISTRY copy splits the two keys (open/shared US-coast WS vs
commercial Hormuz 5x/day REST); VesselAPI no longer reads as an AIS backup.
- vesselapi.py docstring drops 'fallback'; states both are first-class.
- /api/vessels docstring: last-known is the union (extra.src aisstream vs
vesselapi).
- tests: VesselAPI idles/polls independently of the AISStream key.
Add a server-side REST poller for VesselAPI (free tier 150 calls/mo) that
upserts last-known positions into the shared vessel_last_known store when
AISStream is unset. Default box is the Strait of Hormuz (span 3.6 <= 4 deg),
never CONUS/NC (AISStream owns US coasts).
- app/vesselapi.py: worker loop, 4deg span validator, position->marker
transform (skip suspected_glitch), durable Postgres daily-quota table.
- Local hard cap 5 successful 2xx/UTC day (VESSELAPI_MAX_CALLS_PER_DAY),
monthly floor from X-RateLimit-Remaining, single request limit=50, no
nextToken, no filter.sat, no retry-storm.
- keystore VESSELAPI_API_KEY registry entry; config + compose env passthrough
(app + ingest); wired next to AISStream in main.py lifespan + run_ingester.
- GET /api/vessels docstring notes AISStream and/or VesselAPI cache.
- tests/test_vesselapi.py: 20 unit tests (no network/DB).
Planespotters 403s server clients whose User-Agent has no contact
email; compose default UA lacked one. Leaflet popup.update() also
re-ran the bindPopup factory and wiped the thumbnail after load.
- Send a contact UA on planespotters lookups; align compose default.
- Resize the popup without re-rendering content.
- Keep hex/src on live firefighter upserts so they get a photo too.
The reduce/recap prompts told the model to AND STOP with "No qualifying…"
whenever nothing looked breaking enough, so full article batches became
one sentence. Always write summary_en from the provided stories; ticker
and map may still be empty. Futures/market tape stay ignored.
Add planespotters.net latest-photo lookup for ADS-B aircraft.
- app/live_layers.py: fetch_planespotters_photo() (hex preferred, reg
fallback) + _normalize_planespotter_photo(); 24h TTL cache (their ToS cap).
- app/main.py: GET /api/aircraft/photo?hex=...|reg=... (422/404/502).
- app/static/index.html: .ps-photo block in ADS-B popup; lazy load on
popupopen; thumbnail links to photo page + photographer credit.
Server-side proxy, not browser fetch(): planespotters 403s any request
carrying an Origin header (every browser fetch() sends one). The thumbnail
binary is loaded by the browser straight from their CDN, never re-hosted.
Tests: 4 unit + 4 API contract (38 pass in the two files).
Summarizer still runs the 15-min analyst, plus a 24h breaking-news recap
at 23:00 America/New_York. HUD pins kind=daily_recap. Prompts ignore
futures/market tape.
A ~90k-row INSERT ... ON CONFLICT dies with InterfaceError, so fire_loop
fails every poll and /api/health stays degraded (fires age > 30 min).
Chunk 2000 rows per statement, still one commit per poll.
Stop the live HUD reconnect storm (nginx WS snippet + backoff), copy
intel/nous_client into the summarizer image, and make event ingest
idempotent on URL. GDELT uses the DOC API; NWS no longer sends bbox;
FIRMS is one ON CONFLICT batch; GET /api/aircraft serves last-known.
Health reports freshness without 503ing docker. EONET + CISA KEV added.
Scraper loops with NEWS_SCRAPE_INTERVAL_S (default 10s after each
crawl). Summarizer runs every NEWS_SUMMARIZE_INTERVAL_S (default 900)
over the last 15 minutes of articles. Feed list replaced from the
k8s scrapy-urls configmap (334 sources).
CI ran `compose up --force-recreate` without the ingest profile, so
compose tried to *create* osint-dashboard while the old name-pinned
container was still running. down + rm -f the known names, then up
--profile ingest. Volumes stay.
