Heatmap uses {lat,lon,i,c}. Camera list drops URLs (detail via GET
/api/cameras/{id}). Event blips skip body. News omits content unless
include_content=true.
World view no longer fetches cameras/NWS/WFIGS. Default Blue Marble so init
does not wait on GIBS times. News/summary/events load with their views.
hls.min.js loads only on the first HLS camera popup.
Replace the shared colored-dot renderPoints for live feeds with per-feed SVG
glyphs on heading-aware divIcons:
- Aircraft: plane silhouette rotated by heading, color-coded by alt
- Trains: railcar glyph (respecting per-train iconColor), rotated when heading present
- Vessels: sailboat glyph rotated by heading, color by speed (teal/gray)
Icons are cached per (feed, color, heading) so thousands of markers reuse a
bounded set of L.divIcon instances. Rotation is applied to the inner glyph
span only (never the marker container), keeping clustering, spiderfy, and
iconAnchor intact. Missing/NaN heading falls back to an upright glyph.
Color values are sanitized against style-attribute breakout.
Wire the free data streams from docs/free-data-streams.md into the
dashboard as layer-panel toggles. Third-party APIs are proxied/cached
in FastAPI; raster tiles (IEM, RainViewer, GIBS) stay in the browser.
- Aircraft via ADSB.lol viewport poll (bbox required, radius ≤ 150 nm)
- Amtraker trains, NHC storms, WFIGS incidents/perimeters
- NWS + IEM SBW as /api/weather-alerts (does not collide with /api/alerts)
- AISStream worker is server-side only and idles without AISSTREAM_API_KEY
- Caltrans CWWP2 D1–D12 camera parser; FIRMS dual-write NOAA-20/21
Dockerfile.pg installs TimescaleDB from packagecloud.io, which the Pi's
network cannot reach (ISP abuse-mitigation blackholes AWS ranges after the
masscan incident). Every CI deploy rebuilt the DB image and failed on the
packagecloud fetch, blocking all pipelines.
The DB image is now built once manually (docker compose build db) and
pinned as localhost/osint-dashboard-pg:latest; compose up uses it directly.
The app build only needs deb.debian.org + pypi + the docker daemon's
registry mirror, all reachable.
The Ghost-in-the-Shell overhaul was merged via PR #1; the .worktrees
gitlink was only ever a local staging worktree and must not live in the
repo — it breaks CI checkout (git submodule foreach: 'No url found for
submodule path .worktrees/frontend-overhaul').
Master's b2369bc fixed cameras vanishing after opening a popup and zooming:
Leaflet 1.9.4's Map.closePopup() never nulls map._popup, so the old
'if (map._popup) return' guard skipped every overlay reload forever after
the first popup. Port the fix into the redesigned frontend: track popup
state via popupopen/popupclose, close on user zoom/drag so moveend reloads
always run, and keep the autopan skip for the popup's own pan.
Root cause: Leaflet 1.9.4's Map.closePopup() never nulls map._popup (only
unbindPopup does), so the moveend guard 'if (map._popup) return' skipped
every overlay reload forever after the first popup was ever opened. Opening
one camera popup froze the marker layer: zoom out and the map kept showing
the stale zoomed-in subset (or nothing) until a full page refresh.
Track real popup state via popupopen/popupclose events instead, and close
the popup on user zoom/drag so the moveend reload always runs after
navigation; the autopan skip still protects the popup when it pans itself
into view.
- Map is now the landing view: full-viewport NASA GIBS globe, existing
fires/cameras/HLS map machinery preserved verbatim
- New 'Event Blips' layer: geolocated ingest events, color-coded by source,
bbox/since/has_coords filters added to GET /api/events
- Weather / Flights(ADS-B) / Vessels(AIS) layer slots reserved (feed pending)
- Market ticker strip with configurable symbols — auto-promotes to LIVE when
GET /api/market returns {symbols:[...]} (contract documented in Settings)
- Breaking-news ticker: LLM exec-summary flash + headline marquee, 15-min cycle
- Dropdown nav (Map/News/Events/Alerts/Entities/Ingest/API Keys/Settings),
Settings view (localStorage: symbols, map layer defaults), System panel
- Section-9 theme: near-black navy, cyan/magenta accents, Orbitron/Rajdhani/
Share Tech Mono, chamfered HUD panels, scanlines, boot splash, UTC clock
- Fix: events.camera enum value missing from models.py/schemas.py caused 500s
on every events query once camera events flowed in (migration 004 added it
to the DB enum only)
Zooming in replaced the marker group with a bbox subset, so dense world
views (5k dots) thinned to a handful of pins — or zero on a wrapped world
copy, where currentBBox collapsed to a sliver and the API returned ~nothing.
Cameras only seemed to exist when zoomed out.
- vendor leaflet.markercluster 1.5.3; camera markers now render as numbered
neon-green clusters at low zoom and split into individual dots as you zoom,
so camera coverage is visible at every zoom level (spiderfy at max zoom)
- guard against null lat/lon rows (no phantom markers at 0,0)
- stale-response guard for loadCams/loadFires: rapid wheel zoom can resolve
fetches out of order and a stale bbox response blanked the map
- serve index.html with Cache-Control: no-cache so the browser never keeps
serving a pre-deploy copy after fixes land
Viewing CA on a wrapped map copy clamped both edges to 180, so the API
returned the newest 2000 feeds (mostly VDOT HLS in Virginia) and the CA
viewport was empty. Normalize longitudes into [-180,180] and load 5000.
Leaflet worldCopyJump at zoom 2 reports longitudes outside ±180; clamping
them independently produced west>=east or a thin sliver. Low zoom now
queries the full world, and the API ignores inverted bboxes.
Ingest ~4.2k direct HLS/YouTube feeds (VDOT, MDSHA, DelDOT, Iowa DOT,
OpenCCTV, etc.) from the public GeoJSON catalog. Popup plays HLS via a
CORS-safe playlist proxy + vendored hls.js, YouTube via embed, with
ffmpeg-MJPEG fallback. ALERTWest rows now use a stable camera id so
hourly scrapes do not duplicate.
Adds the documented getCameraDataByLoc API (~10k public wildfire, DOT,
and FAA stills with lat/lon). Masscan will not produce viewable feeds;
this will. Map bbox fetch raised to 2000 markers.
Masscan open-554 hosts almost never have a public picture. The map now
defaults to cameras with an http(s) snapshot_url (~132 directory cams).
Ingest probes unauthenticated still/MJPEG paths first and skips the rest,
so dead RTSP-only hosts never land on the map. Pass working=false to see
unverified port-554 hits.
Masscan finds are rtsp:// with no snapshot_url, so the popup skipped the
<img> and the leftover source link handed the OS an rtsp:// URL (VLC).
- Popup always hits /api/cameras/{id}/snapshot (HTTP stills, then one
ffmpeg frame grab). No credentials. 10s hard timeout.
- rtsp:// is rendered as text, never as an href.
- ffmpeg added to the app image for the RTSP still/MJPEG path.
- Default MASSCAN_RATE 200 (1k/10k saturated the home uplink).
Continuous whole-IPv4 rolling sweep for open TCP 554, feeding the same
cameras table as the passive scraper (discovery_source=masscan).
- masscan_config.py: env-driven knobs (range, ports, rate, retries, excludes)
- masscan_scanner.py: JSON-lines parser, rtsp://IP/ URL + url_hash dedupe,
ip-api geolocation, insert/refresh, NATS publish for new finds
- run_masscan_service.py: long-lived rolling-sweep runner (streams results
in, restarts on pass completion); fails closed without an excludefile
- deploy/: systemd unit + README + excludes file for the Pi host
- .env.example: masscan section
Verified end-to-end against a local Postgres: parse, insert, and dedupe
(0 new on re-ingest) all pass.
FIRMS area-CSV requires an explicit [1..5] day range — requests without
it return 'Invalid day range. Expects [1..5].' as a 400. New
FIRMS_DAYS env (default 1).
Key saved via the dashboard Keys page sat unused in api_keys while
ingest_fires only checked the env var. Now env first, keystore fallback,
picked up on next poll without a restart.
- Vendored leaflet.heat 0.2.0; fires overlay is now a real heatmap
(L.heatLayer) with intensity driven by brightness or VIIRS confidence
(color mode selector) and a since= time filter (6h/24h/48h/7d/30d/all).
- Camera markers now open popups with the live snapshot thumbnail proxied
through GET /api/cameras/{id}/snapshot (backend TTL cache) + full
metadata (vendor, device, discovery source, first/last seen, coords,
source link); missing-snapshot cams show a placeholder instead of 404.
- New collapsible Layers panel (dark neon) over the map: per-layer
visibility toggles, opacity sliders (basemap/fires/cameras), live
counts legend, and a fire heat-intensity ramp that swaps for the
confidence legend in confidence mode.
- Clamp currentBBox() to world bounds so /api/cameras doesn't 422 at low
zoom (worldCopyJump spans multiple copies).
- Skip overlay reloads on moveend while a popup is open — popup autopan
no longer rebuilds the marker group and closes the popup.
- Zoom control moved to top-right so it doesn't collide with the panel.
The vendored spider only started on urls.txt lines containing '/rss' or
'/feed' — but the curated urls.txt holds 200 homepages, so the crawler
matched ZERO feeds and silently did nothing (2ms, 0 items). Rewrite the
spider with feed autodiscovery: fetch each homepage, find its
<link rel="alternate" type="application/rss+xml"> (or /feed|/rss link),
parse the feed, then follow each item to extract the article. Bump
DEPTH_LIMIT 1->3 (homepage -> feed -> article).
Also make the summarizer resilient to booting before the app container has
run alembic (docker-compose only guarantees `db` is up): add idempotent
ensure_tables() mirroring the scraper's CREATE TABLE IF NOT EXISTS.
Interactive world map panel for the dashboard:
- Vendored Leaflet 1.9.4 under app/static/vendor/leaflet/ (served via
new /static mount on the FastAPI app)
- New Map tab: GIBS WMTS raster basemap (BlueMarble_ShadedRelief_Bathymetry,
VIIRS/MODIS/Aqua CorrectedReflectance_TrueColor, VIIRS_DayNightBand),
layer picker + UTC date selector (Latest/-7d/-30d quick picks)
- /api/map/layers: curated GIBS catalog (tms, format, has_time, max_zoom)
- /api/map/times: per-layer date windows from GIBS Domains XML (6h cache,
graceful 502 on GIBS hiccup; static layers 422)
- Fire hotspots (existing /api/fires) + open cameras (/api/cameras) as
toggleable bbox-scoped overlays; dark UI matching dashboard theme
Vendor the newsPipeline scraper + summarizer into the repo and wire them into
docker-compose against the EXISTING osint-db (no second Postgres), replacing
the upstream k8s CronJobs with in-compose wall-clock loops (:00 scrape, :05
summarize).
- news/scraper: vendored Scrapy project (257 RSS feeds) + hourly loop
scheduler (run_news_scraper.py)
- news/summerizer: vendored Gemini map-reduce summarizer, cleaned:
* fix broken google-genai response handling (_extract_text, defensive)
* fix malformed INSERT/GRANT query in save_summary_to_db
* OSINT-neutral default MAP_PROMPT; futures/markets language gated behind
INCLUDE_FUTURES=0 (yfinance lazy-imported)
* env-configurable model, batch size, lookback window
+ hourly loop scheduler (run_news_summarizer.py, :05)
- alembic 003_news: idempotent articles + article_summaries tables
- API: GET /api/news and GET /api/news/summaries (+ models, schemas)
- tests/test_api_news.py: 5 DB-backed contract tests (all pass vs real PG)
- docs/news.md + .env.example updates
Both services run under the `ingest` compose profile (matching the
ingester/camera-scraper pattern) and build arm64 on the Pi via the existing
Forgejo CI workflow. telebot left out of scope (reserved env only).
002_fires and 002_cameras both had down_revision=001_initial, producing two
heads so 'alembic upgrade head' fails at container startup and the dashboard
never boots. Tables are independent; linearize 001 -> 002_fires -> 002_cameras.
Coherent merge of two coordinated features on the shared working tree:
FIRMS fire heatmap (backend, t_6e404c14):
- app/fire_sources.py: fetch FIRMS VIIRS area CSV (free MAP_KEY) -> NATS events.fire
- fires hypertable (TimescaleDB, 1-day chunks) with natural-key PK
(latitude, longitude, acq_time, satellite); idempotent ON CONFLICT DO NOTHING
- alembic/versions/002_fires.py; GET /api/fires?bbox=&since= (JSON only)
- POST /api/ingest/fires; ~15 min poll loop (FIRMS_INTERVAL=900) in ingester
- env-driven config (FIRMS_MAP_KEY/DATASET/BBOX/INTERVAL); docs/firms.md covers
the zero-cost GIBS VIIRS_SNPP_Thermal_Anomalies_375m_All tile alternative
- 18 tests (parser, mapping, idempotency, API contract) verified vs real
TimescaleDB+PostGIS (localhost/osint-dashboard-pg image)
API keys page (frontend, t_4433cff2):
- app/keystore.py: api_keys table (self-creating), FIRMS/GEMINI/TELEGRAM
registry with format validation, ****last4 masking, get_api_key()
- GET/POST/DELETE /api/keys (never returns full values); Keys tab in index.html
DB_NULL_POOL env switch in app/database.py enables a NullPool for tests /
short-lived processes that open a fresh event loop per unit.
