sirius/osint-dashboard
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix: pipeline reliability — WS, summarizer, ingest, health #13

Merged
sirius merged 1 commit from fix/pipeline-reliability into master 2026-08-28 21:49:57 -04:00
Conversation 0 Commits 1 Files changed 26 +1169 -115
sirius commented 2026-08-28 21:47:28 -04:00
Owner
Copy link

Summary

Fixes the confirmed live bugs and the ranked load-time issues.

  • Summarizer image copies intel.py + nous_client.py (was crashing ModuleNotFoundError: intel on boot).
  • WebSocket: deploy/osint-ws.nginx.conf with proxy_http_version 1.1 + Upgrade/Connection. HUD reconnect uses exponential backoff instead of 4s. Nginx snippet is not applied by CI — include it in the osint vhost and reload.
  • Event idempotency: event_dedup table (Timescale cannot unique-index url alone). Alembic 007 deletes URL dupes, then compression/retention on events/fires/tracks.
  • GDELT uses official DOC API (/api/v2/doc/doc, query= not search=), HTTP fallback on TLS failure.
  • NWS: never send bbox (400s); clip locally; nws_ok on the FeatureCollection when upstream fails.
  • FIRMS: one ON CONFLICT batch, one commit (no 93k NATS/commits).
  • ADS-B: GET /api/aircraft serves last-known only; persist/geofence/WS is a background viewport loop.
  • Health: /api/health stays 200 if DB is up, with status: degraded + checks. /api/ready 503s only when DB is down.
  • Scraper: populate seen_urls, don't log DropItem as error; skip audio enclosures; store feed pubDate.
  • HUD: stop /api/market 404 poll; defer startup overlay fan-out; immutable Cache-Control on /static/vendor/*.
  • New sources: NASA EONET v3 + CISA KEV (no fake map coords).

Tests

pytest -q → 106 passed, 15 skipped.

Ops note

After merge, copy deploy/osint-ws.nginx.conf to /etc/nginx/snippets/osint-ws.conf and include it in the osint.rpi.local server block, then nginx -t && systemctl reload nginx. Until then live AIS/ADS-B WS still 404s through nginx (backoff only reduces the storm).

## Summary Fixes the confirmed live bugs and the ranked load-time issues. - **Summarizer image** copies `intel.py` + `nous_client.py` (was crashing `ModuleNotFoundError: intel` on boot). - **WebSocket**: `deploy/osint-ws.nginx.conf` with `proxy_http_version 1.1` + Upgrade/Connection. HUD reconnect uses exponential backoff instead of 4s. **Nginx snippet is not applied by CI** — include it in the osint vhost and reload. - **Event idempotency**: `event_dedup` table (Timescale cannot unique-index `url` alone). Alembic `007` deletes URL dupes, then compression/retention on events/fires/tracks. - **GDELT** uses official DOC API (`/api/v2/doc/doc`, `query=` not `search=`), HTTP fallback on TLS failure. - **NWS**: never send `bbox` (400s); clip locally; `nws_ok` on the FeatureCollection when upstream fails. - **FIRMS**: one `ON CONFLICT` batch, one commit (no 93k NATS/commits). - **ADS-B**: `GET /api/aircraft` serves last-known only; persist/geofence/WS is a background viewport loop. - **Health**: `/api/health` stays 200 if DB is up, with `status: degraded` + `checks`. `/api/ready` 503s only when DB is down. - Scraper: populate `seen_urls`, don't log `DropItem` as error; skip audio enclosures; store feed `pubDate`. - HUD: stop `/api/market` 404 poll; defer startup overlay fan-out; immutable Cache-Control on `/static/vendor/*`. - New sources: NASA EONET v3 + CISA KEV (no fake map coords). ## Tests `pytest -q` → 106 passed, 15 skipped. ## Ops note After merge, copy `deploy/osint-ws.nginx.conf` to `/etc/nginx/snippets/osint-ws.conf` and `include` it in the osint.rpi.local server block, then `nginx -t && systemctl reload nginx`. Until then live AIS/ADS-B WS still 404s through nginx (backoff only reduces the storm).
sirius added 1 commit 2026-08-28 21:47:28 -04:00 
Stop the live HUD reconnect storm (nginx WS snippet + backoff), copy
intel/nous_client into the summarizer image, and make event ingest
idempotent on URL. GDELT uses the DOC API; NWS no longer sends bbox;
FIRMS is one ON CONFLICT batch; GET /api/aircraft serves last-known.
Health reports freshness without 503ing docker. EONET + CISA KEV added.
sirius merged commit 19fa0db32c into master 2026-08-28 21:49:57 -04:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: sirius/osint-dashboard#13
No description provided.
Delete branch "fix/pipeline-reliability"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?