A ~90k-row INSERT ... ON CONFLICT dies with InterfaceError, so fire_loop
fails every poll and /api/health stays degraded (fires age > 30 min).
Chunk 2000 rows per statement, still one commit per poll.
Stop the live HUD reconnect storm (nginx WS snippet + backoff), copy
intel/nous_client into the summarizer image, and make event ingest
idempotent on URL. GDELT uses the DOC API; NWS no longer sends bbox;
FIRMS is one ON CONFLICT batch; GET /api/aircraft serves last-known.
Health reports freshness without 503ing docker. EONET + CISA KEV added.
Scraper loops with NEWS_SCRAPE_INTERVAL_S (default 10s after each
crawl). Summarizer runs every NEWS_SUMMARIZE_INTERVAL_S (default 900)
over the last 15 minutes of articles. Feed list replaced from the
k8s scrapy-urls configmap (334 sources).
CI ran `compose up --force-recreate` without the ingest profile, so
compose tried to *create* osint-dashboard while the old name-pinned
container was still running. down + rm -f the known names, then up
--profile ingest. Volumes stay.
005_phase2 and 005_news_items both revised 004_camera_enum. Entrypoint
`alembic upgrade head` then failed and the app never bound :8000 (502
over Tailscale). Merge revision + `upgrade heads`.
GITHUB_TOKEN cannot write packages (401). PAT works, but layers
over https://forgejo.siriusdevops.com 413 at Cloudflare's ~100MB
POST cap. Push 127.0.0.1:3000; pull names stay public.
Push api/pg/news-scraper/news-summarizer to
forgejo.siriusdevops.com/sirius/* on every master build, then deploy
on the Pi runner. No GHCR. Helper script retags pulls to localhost/*
for compose.
The Vessels layer now retunes the server-side AISStream subscription to the
client viewport instead of a static AISSTREAM_BBOX. The frontend POSTs its
quantized viewport box to /api/vessels/subscribe on moveend; the ais_stream
worker coalesces and applies it at the service's 1 subscription/s cap, then
last-known positions for the new area arrive within a couple of seconds (the
frontend does one follow-up fetch after retuning). Bounds the in-memory
vessel store across regions. Key stays server-side.
moveend aborts in-flight overlay JSON and skips work when the 0.01° cell
is unchanged. Live aircraft/trains/vessels diff-update by id. Camera popup
HTML is built on open. Alert/perimeter polygons paint on canvas.
Reuse one TLS pool for overlay upstreams (8s/3s timeouts). Gzip JSON
over 1 KB. Aircraft/vessels Cache-Control max-age=5, alerts/perimeters 30.
Lifespan replaces deprecated on_event startup.
Heatmap uses {lat,lon,i,c}. Camera list drops URLs (detail via GET
/api/cameras/{id}). Event blips skip body. News omits content unless
include_content=true.
World view no longer fetches cameras/NWS/WFIGS. Default Blue Marble so init
does not wait on GIBS times. News/summary/events load with their views.
hls.min.js loads only on the first HLS camera popup.
Replace the shared colored-dot renderPoints for live feeds with per-feed SVG
glyphs on heading-aware divIcons:
- Aircraft: plane silhouette rotated by heading, color-coded by alt
- Trains: railcar glyph (respecting per-train iconColor), rotated when heading present
- Vessels: sailboat glyph rotated by heading, color by speed (teal/gray)
Icons are cached per (feed, color, heading) so thousands of markers reuse a
bounded set of L.divIcon instances. Rotation is applied to the inner glyph
span only (never the marker container), keeping clustering, spiderfy, and
iconAnchor intact. Missing/NaN heading falls back to an upright glyph.
Color values are sanitized against style-attribute breakout.
Wire the free data streams from docs/free-data-streams.md into the
dashboard as layer-panel toggles. Third-party APIs are proxied/cached
in FastAPI; raster tiles (IEM, RainViewer, GIBS) stay in the browser.
- Aircraft via ADSB.lol viewport poll (bbox required, radius ≤ 150 nm)
- Amtraker trains, NHC storms, WFIGS incidents/perimeters
- NWS + IEM SBW as /api/weather-alerts (does not collide with /api/alerts)
- AISStream worker is server-side only and idles without AISSTREAM_API_KEY
- Caltrans CWWP2 D1–D12 camera parser; FIRMS dual-write NOAA-20/21
Dockerfile.pg installs TimescaleDB from packagecloud.io, which the Pi's
network cannot reach (ISP abuse-mitigation blackholes AWS ranges after the
masscan incident). Every CI deploy rebuilt the DB image and failed on the
packagecloud fetch, blocking all pipelines.
The DB image is now built once manually (docker compose build db) and
pinned as localhost/osint-dashboard-pg:latest; compose up uses it directly.
The app build only needs deb.debian.org + pypi + the docker daemon's
registry mirror, all reachable.
The Ghost-in-the-Shell overhaul was merged via PR #1; the .worktrees
gitlink was only ever a local staging worktree and must not live in the
repo — it breaks CI checkout (git submodule foreach: 'No url found for
submodule path .worktrees/frontend-overhaul').
