Scraper loops with NEWS_SCRAPE_INTERVAL_S (default 10s after each
crawl). Summarizer runs every NEWS_SUMMARIZE_INTERVAL_S (default 900)
over the last 15 minutes of articles. Feed list replaced from the
k8s scrapy-urls configmap (334 sources).
005_phase2 and 005_news_items both revised 004_camera_enum. Entrypoint
`alembic upgrade head` then failed and the app never bound :8000 (502
over Tailscale). Merge revision + `upgrade heads`.
The Vessels layer now retunes the server-side AISStream subscription to the
client viewport instead of a static AISSTREAM_BBOX. The frontend POSTs its
quantized viewport box to /api/vessels/subscribe on moveend; the ais_stream
worker coalesces and applies it at the service's 1 subscription/s cap, then
last-known positions for the new area arrive within a couple of seconds (the
frontend does one follow-up fetch after retuning). Bounds the in-memory
vessel store across regions. Key stays server-side.
moveend aborts in-flight overlay JSON and skips work when the 0.01° cell
is unchanged. Live aircraft/trains/vessels diff-update by id. Camera popup
HTML is built on open. Alert/perimeter polygons paint on canvas.
Reuse one TLS pool for overlay upstreams (8s/3s timeouts). Gzip JSON
over 1 KB. Aircraft/vessels Cache-Control max-age=5, alerts/perimeters 30.
Lifespan replaces deprecated on_event startup.
Heatmap uses {lat,lon,i,c}. Camera list drops URLs (detail via GET
/api/cameras/{id}). Event blips skip body. News omits content unless
include_content=true.
World view no longer fetches cameras/NWS/WFIGS. Default Blue Marble so init
does not wait on GIBS times. News/summary/events load with their views.
hls.min.js loads only on the first HLS camera popup.
Replace the shared colored-dot renderPoints for live feeds with per-feed SVG
glyphs on heading-aware divIcons:
- Aircraft: plane silhouette rotated by heading, color-coded by alt
- Trains: railcar glyph (respecting per-train iconColor), rotated when heading present
- Vessels: sailboat glyph rotated by heading, color by speed (teal/gray)
Icons are cached per (feed, color, heading) so thousands of markers reuse a
bounded set of L.divIcon instances. Rotation is applied to the inner glyph
span only (never the marker container), keeping clustering, spiderfy, and
iconAnchor intact. Missing/NaN heading falls back to an upright glyph.
Color values are sanitized against style-attribute breakout.
Wire the free data streams from docs/free-data-streams.md into the
dashboard as layer-panel toggles. Third-party APIs are proxied/cached
in FastAPI; raster tiles (IEM, RainViewer, GIBS) stay in the browser.
- Aircraft via ADSB.lol viewport poll (bbox required, radius ≤ 150 nm)
- Amtraker trains, NHC storms, WFIGS incidents/perimeters
- NWS + IEM SBW as /api/weather-alerts (does not collide with /api/alerts)
- AISStream worker is server-side only and idles without AISSTREAM_API_KEY
- Caltrans CWWP2 D1–D12 camera parser; FIRMS dual-write NOAA-20/21
Master's b2369bc fixed cameras vanishing after opening a popup and zooming:
Leaflet 1.9.4's Map.closePopup() never nulls map._popup, so the old
'if (map._popup) return' guard skipped every overlay reload forever after
the first popup. Port the fix into the redesigned frontend: track popup
state via popupopen/popupclose, close on user zoom/drag so moveend reloads
always run, and keep the autopan skip for the popup's own pan.
- Map is now the landing view: full-viewport NASA GIBS globe, existing
fires/cameras/HLS map machinery preserved verbatim
- New 'Event Blips' layer: geolocated ingest events, color-coded by source,
bbox/since/has_coords filters added to GET /api/events
- Weather / Flights(ADS-B) / Vessels(AIS) layer slots reserved (feed pending)
- Market ticker strip with configurable symbols — auto-promotes to LIVE when
GET /api/market returns {symbols:[...]} (contract documented in Settings)
- Breaking-news ticker: LLM exec-summary flash + headline marquee, 15-min cycle
- Dropdown nav (Map/News/Events/Alerts/Entities/Ingest/API Keys/Settings),
Settings view (localStorage: symbols, map layer defaults), System panel
- Section-9 theme: near-black navy, cyan/magenta accents, Orbitron/Rajdhani/
Share Tech Mono, chamfered HUD panels, scanlines, boot splash, UTC clock
- Fix: events.camera enum value missing from models.py/schemas.py caused 500s
on every events query once camera events flowed in (migration 004 added it
to the DB enum only)
Zooming in replaced the marker group with a bbox subset, so dense world
views (5k dots) thinned to a handful of pins — or zero on a wrapped world
copy, where currentBBox collapsed to a sliver and the API returned ~nothing.
Cameras only seemed to exist when zoomed out.
- vendor leaflet.markercluster 1.5.3; camera markers now render as numbered
neon-green clusters at low zoom and split into individual dots as you zoom,
so camera coverage is visible at every zoom level (spiderfy at max zoom)
- guard against null lat/lon rows (no phantom markers at 0,0)
- stale-response guard for loadCams/loadFires: rapid wheel zoom can resolve
fetches out of order and a stale bbox response blanked the map
- serve index.html with Cache-Control: no-cache so the browser never keeps
serving a pre-deploy copy after fixes land
Viewing CA on a wrapped map copy clamped both edges to 180, so the API
returned the newest 2000 feeds (mostly VDOT HLS in Virginia) and the CA
viewport was empty. Normalize longitudes into [-180,180] and load 5000.
Leaflet worldCopyJump at zoom 2 reports longitudes outside ±180; clamping
them independently produced west>=east or a thin sliver. Low zoom now
queries the full world, and the API ignores inverted bboxes.
Ingest ~4.2k direct HLS/YouTube feeds (VDOT, MDSHA, DelDOT, Iowa DOT,
OpenCCTV, etc.) from the public GeoJSON catalog. Popup plays HLS via a
CORS-safe playlist proxy + vendored hls.js, YouTube via embed, with
ffmpeg-MJPEG fallback. ALERTWest rows now use a stable camera id so
hourly scrapes do not duplicate.
Adds the documented getCameraDataByLoc API (~10k public wildfire, DOT,
and FAA stills with lat/lon). Masscan will not produce viewable feeds;
this will. Map bbox fetch raised to 2000 markers.
Masscan open-554 hosts almost never have a public picture. The map now
defaults to cameras with an http(s) snapshot_url (~132 directory cams).
Ingest probes unauthenticated still/MJPEG paths first and skips the rest,
so dead RTSP-only hosts never land on the map. Pass working=false to see
unverified port-554 hits.
Masscan finds are rtsp:// with no snapshot_url, so the popup skipped the
<img> and the leftover source link handed the OS an rtsp:// URL (VLC).
- Popup always hits /api/cameras/{id}/snapshot (HTTP stills, then one
ffmpeg frame grab). No credentials. 10s hard timeout.
- rtsp:// is rendered as text, never as an href.
- ffmpeg added to the app image for the RTSP still/MJPEG path.
- Default MASSCAN_RATE 200 (1k/10k saturated the home uplink).
Continuous whole-IPv4 rolling sweep for open TCP 554, feeding the same
cameras table as the passive scraper (discovery_source=masscan).
- masscan_config.py: env-driven knobs (range, ports, rate, retries, excludes)
- masscan_scanner.py: JSON-lines parser, rtsp://IP/ URL + url_hash dedupe,
ip-api geolocation, insert/refresh, NATS publish for new finds
- run_masscan_service.py: long-lived rolling-sweep runner (streams results
in, restarts on pass completion); fails closed without an excludefile
- deploy/: systemd unit + README + excludes file for the Pi host
- .env.example: masscan section
Verified end-to-end against a local Postgres: parse, insert, and dedupe
(0 new on re-ingest) all pass.
FIRMS area-CSV requires an explicit [1..5] day range — requests without
it return 'Invalid day range. Expects [1..5].' as a 400. New
FIRMS_DAYS env (default 1).
Key saved via the dashboard Keys page sat unused in api_keys while
ingest_fires only checked the env var. Now env first, keystore fallback,
picked up on next poll without a restart.
- Vendored leaflet.heat 0.2.0; fires overlay is now a real heatmap
(L.heatLayer) with intensity driven by brightness or VIIRS confidence
(color mode selector) and a since= time filter (6h/24h/48h/7d/30d/all).
- Camera markers now open popups with the live snapshot thumbnail proxied
through GET /api/cameras/{id}/snapshot (backend TTL cache) + full
metadata (vendor, device, discovery source, first/last seen, coords,
source link); missing-snapshot cams show a placeholder instead of 404.
- New collapsible Layers panel (dark neon) over the map: per-layer
visibility toggles, opacity sliders (basemap/fires/cameras), live
counts legend, and a fire heat-intensity ramp that swaps for the
confidence legend in confidence mode.
- Clamp currentBBox() to world bounds so /api/cameras doesn't 422 at low
zoom (worldCopyJump spans multiple copies).
- Skip overlay reloads on moveend while a popup is open — popup autopan
no longer rebuilds the marker group and closes the popup.
- Zoom control moved to top-right so it doesn't collide with the panel.
Interactive world map panel for the dashboard:
- Vendored Leaflet 1.9.4 under app/static/vendor/leaflet/ (served via
new /static mount on the FastAPI app)
- New Map tab: GIBS WMTS raster basemap (BlueMarble_ShadedRelief_Bathymetry,
VIIRS/MODIS/Aqua CorrectedReflectance_TrueColor, VIIRS_DayNightBand),
layer picker + UTC date selector (Latest/-7d/-30d quick picks)
- /api/map/layers: curated GIBS catalog (tms, format, has_time, max_zoom)
- /api/map/times: per-layer date windows from GIBS Domains XML (6h cache,
graceful 502 on GIBS hiccup; static layers 422)
- Fire hotspots (existing /api/fires) + open cameras (/api/cameras) as
toggleable bbox-scoped overlays; dark UI matching dashboard theme
Vendor the newsPipeline scraper + summarizer into the repo and wire them into
docker-compose against the EXISTING osint-db (no second Postgres), replacing
the upstream k8s CronJobs with in-compose wall-clock loops (:00 scrape, :05
summarize).
- news/scraper: vendored Scrapy project (257 RSS feeds) + hourly loop
scheduler (run_news_scraper.py)
- news/summerizer: vendored Gemini map-reduce summarizer, cleaned:
* fix broken google-genai response handling (_extract_text, defensive)
* fix malformed INSERT/GRANT query in save_summary_to_db
* OSINT-neutral default MAP_PROMPT; futures/markets language gated behind
INCLUDE_FUTURES=0 (yfinance lazy-imported)
* env-configurable model, batch size, lookback window
+ hourly loop scheduler (run_news_summarizer.py, :05)
- alembic 003_news: idempotent articles + article_summaries tables
- API: GET /api/news and GET /api/news/summaries (+ models, schemas)
- tests/test_api_news.py: 5 DB-backed contract tests (all pass vs real PG)
- docs/news.md + .env.example updates
Both services run under the `ingest` compose profile (matching the
ingester/camera-scraper pattern) and build arm64 on the Pi via the existing
Forgejo CI workflow. telebot left out of scope (reserved env only).
