Zooming in replaced the marker group with a bbox subset, so dense world
views (5k dots) thinned to a handful of pins — or zero on a wrapped world
copy, where currentBBox collapsed to a sliver and the API returned ~nothing.
Cameras only seemed to exist when zoomed out.
- vendor leaflet.markercluster 1.5.3; camera markers now render as numbered
neon-green clusters at low zoom and split into individual dots as you zoom,
so camera coverage is visible at every zoom level (spiderfy at max zoom)
- guard against null lat/lon rows (no phantom markers at 0,0)
- stale-response guard for loadCams/loadFires: rapid wheel zoom can resolve
fetches out of order and a stale bbox response blanked the map
- serve index.html with Cache-Control: no-cache so the browser never keeps
serving a pre-deploy copy after fixes land
Viewing CA on a wrapped map copy clamped both edges to 180, so the API
returned the newest 2000 feeds (mostly VDOT HLS in Virginia) and the CA
viewport was empty. Normalize longitudes into [-180,180] and load 5000.
Leaflet worldCopyJump at zoom 2 reports longitudes outside ±180; clamping
them independently produced west>=east or a thin sliver. Low zoom now
queries the full world, and the API ignores inverted bboxes.
Ingest ~4.2k direct HLS/YouTube feeds (VDOT, MDSHA, DelDOT, Iowa DOT,
OpenCCTV, etc.) from the public GeoJSON catalog. Popup plays HLS via a
CORS-safe playlist proxy + vendored hls.js, YouTube via embed, with
ffmpeg-MJPEG fallback. ALERTWest rows now use a stable camera id so
hourly scrapes do not duplicate.
Adds the documented getCameraDataByLoc API (~10k public wildfire, DOT,
and FAA stills with lat/lon). Masscan will not produce viewable feeds;
this will. Map bbox fetch raised to 2000 markers.
Masscan open-554 hosts almost never have a public picture. The map now
defaults to cameras with an http(s) snapshot_url (~132 directory cams).
Ingest probes unauthenticated still/MJPEG paths first and skips the rest,
so dead RTSP-only hosts never land on the map. Pass working=false to see
unverified port-554 hits.
Masscan finds are rtsp:// with no snapshot_url, so the popup skipped the
<img> and the leftover source link handed the OS an rtsp:// URL (VLC).
- Popup always hits /api/cameras/{id}/snapshot (HTTP stills, then one
ffmpeg frame grab). No credentials. 10s hard timeout.
- rtsp:// is rendered as text, never as an href.
- ffmpeg added to the app image for the RTSP still/MJPEG path.
- Default MASSCAN_RATE 200 (1k/10k saturated the home uplink).
Continuous whole-IPv4 rolling sweep for open TCP 554, feeding the same
cameras table as the passive scraper (discovery_source=masscan).
- masscan_config.py: env-driven knobs (range, ports, rate, retries, excludes)
- masscan_scanner.py: JSON-lines parser, rtsp://IP/ URL + url_hash dedupe,
ip-api geolocation, insert/refresh, NATS publish for new finds
- run_masscan_service.py: long-lived rolling-sweep runner (streams results
in, restarts on pass completion); fails closed without an excludefile
- deploy/: systemd unit + README + excludes file for the Pi host
- .env.example: masscan section
Verified end-to-end against a local Postgres: parse, insert, and dedupe
(0 new on re-ingest) all pass.
FIRMS area-CSV requires an explicit [1..5] day range — requests without
it return 'Invalid day range. Expects [1..5].' as a 400. New
FIRMS_DAYS env (default 1).
Key saved via the dashboard Keys page sat unused in api_keys while
ingest_fires only checked the env var. Now env first, keystore fallback,
picked up on next poll without a restart.
- Vendored leaflet.heat 0.2.0; fires overlay is now a real heatmap
(L.heatLayer) with intensity driven by brightness or VIIRS confidence
(color mode selector) and a since= time filter (6h/24h/48h/7d/30d/all).
- Camera markers now open popups with the live snapshot thumbnail proxied
through GET /api/cameras/{id}/snapshot (backend TTL cache) + full
metadata (vendor, device, discovery source, first/last seen, coords,
source link); missing-snapshot cams show a placeholder instead of 404.
- New collapsible Layers panel (dark neon) over the map: per-layer
visibility toggles, opacity sliders (basemap/fires/cameras), live
counts legend, and a fire heat-intensity ramp that swaps for the
confidence legend in confidence mode.
- Clamp currentBBox() to world bounds so /api/cameras doesn't 422 at low
zoom (worldCopyJump spans multiple copies).
- Skip overlay reloads on moveend while a popup is open — popup autopan
no longer rebuilds the marker group and closes the popup.
- Zoom control moved to top-right so it doesn't collide with the panel.
The vendored spider only started on urls.txt lines containing '/rss' or
'/feed' — but the curated urls.txt holds 200 homepages, so the crawler
matched ZERO feeds and silently did nothing (2ms, 0 items). Rewrite the
spider with feed autodiscovery: fetch each homepage, find its
<link rel="alternate" type="application/rss+xml"> (or /feed|/rss link),
parse the feed, then follow each item to extract the article. Bump
DEPTH_LIMIT 1->3 (homepage -> feed -> article).
Also make the summarizer resilient to booting before the app container has
run alembic (docker-compose only guarantees `db` is up): add idempotent
ensure_tables() mirroring the scraper's CREATE TABLE IF NOT EXISTS.
Interactive world map panel for the dashboard:
- Vendored Leaflet 1.9.4 under app/static/vendor/leaflet/ (served via
new /static mount on the FastAPI app)
- New Map tab: GIBS WMTS raster basemap (BlueMarble_ShadedRelief_Bathymetry,
VIIRS/MODIS/Aqua CorrectedReflectance_TrueColor, VIIRS_DayNightBand),
layer picker + UTC date selector (Latest/-7d/-30d quick picks)
- /api/map/layers: curated GIBS catalog (tms, format, has_time, max_zoom)
- /api/map/times: per-layer date windows from GIBS Domains XML (6h cache,
graceful 502 on GIBS hiccup; static layers 422)
- Fire hotspots (existing /api/fires) + open cameras (/api/cameras) as
toggleable bbox-scoped overlays; dark UI matching dashboard theme
Vendor the newsPipeline scraper + summarizer into the repo and wire them into
docker-compose against the EXISTING osint-db (no second Postgres), replacing
the upstream k8s CronJobs with in-compose wall-clock loops (:00 scrape, :05
summarize).
- news/scraper: vendored Scrapy project (257 RSS feeds) + hourly loop
scheduler (run_news_scraper.py)
- news/summerizer: vendored Gemini map-reduce summarizer, cleaned:
* fix broken google-genai response handling (_extract_text, defensive)
* fix malformed INSERT/GRANT query in save_summary_to_db
* OSINT-neutral default MAP_PROMPT; futures/markets language gated behind
INCLUDE_FUTURES=0 (yfinance lazy-imported)
* env-configurable model, batch size, lookback window
+ hourly loop scheduler (run_news_summarizer.py, :05)
- alembic 003_news: idempotent articles + article_summaries tables
- API: GET /api/news and GET /api/news/summaries (+ models, schemas)
- tests/test_api_news.py: 5 DB-backed contract tests (all pass vs real PG)
- docs/news.md + .env.example updates
Both services run under the `ingest` compose profile (matching the
ingester/camera-scraper pattern) and build arm64 on the Pi via the existing
Forgejo CI workflow. telebot left out of scope (reserved env only).
002_fires and 002_cameras both had down_revision=001_initial, producing two
heads so 'alembic upgrade head' fails at container startup and the dashboard
never boots. Tables are independent; linearize 001 -> 002_fires -> 002_cameras.
Coherent merge of two coordinated features on the shared working tree:
FIRMS fire heatmap (backend, t_6e404c14):
- app/fire_sources.py: fetch FIRMS VIIRS area CSV (free MAP_KEY) -> NATS events.fire
- fires hypertable (TimescaleDB, 1-day chunks) with natural-key PK
(latitude, longitude, acq_time, satellite); idempotent ON CONFLICT DO NOTHING
- alembic/versions/002_fires.py; GET /api/fires?bbox=&since= (JSON only)
- POST /api/ingest/fires; ~15 min poll loop (FIRMS_INTERVAL=900) in ingester
- env-driven config (FIRMS_MAP_KEY/DATASET/BBOX/INTERVAL); docs/firms.md covers
the zero-cost GIBS VIIRS_SNPP_Thermal_Anomalies_375m_All tile alternative
- 18 tests (parser, mapping, idempotency, API contract) verified vs real
TimescaleDB+PostGIS (localhost/osint-dashboard-pg image)
API keys page (frontend, t_4433cff2):
- app/keystore.py: api_keys table (self-creating), FIRMS/GEMINI/TELEGRAM
registry with format validation, ****last4 masking, get_api_key()
- GET/POST/DELETE /api/keys (never returns full values); Keys tab in index.html
DB_NULL_POOL env switch in app/database.py enables a NullPool for tests /
short-lived processes that open a fresh event loop per unit.
