Commit graph

117 commits

Author SHA1 Message Date
Sirius DevOps
5212aaa5cd feat(sentinel1): Sentinel-1 SAR STAC -> self-hosted TiTiler tile template
Add GET /api/map/sentinel1?bbox=... which queries Planetary Computer STAC
(sentinel-1-grd, last 7d, most recent), signs the vv/hh COG with a SAS token,
and returns a same-origin /titiler/... XYZ tile template. Cache keyed on
quantized bbox + UTC day (20 min TTL). 429 -> 429 (Retry-After), no imagery
-> 404, other upstream errors -> 502.

Self-host TiTiler on the Pi (ghcr.io/developmentseed/titiler, arm64, 1G cap,
host loopback 8001 -> container 8000) instead of titiler.xyz. nginx
/titiler/ proxy snippet routes browser tiles to it; TITILER_PUBLIC_BASE and
TITILER_INTERNAL_URL are env-driven (no hardcoded hostnames).
2026-08-29 01:01:20 -04:00
30a9c9e9b1 Merge pull request 'fix(aircraft): show planespotters photo in overlay popup' (#18) from fix/planespotters-popup-photo into master
All checks were successful
build-and-deploy / build-push-deploy (push) Successful in 2m55s
Reviewed-on: #18
2026-08-28 23:27:23 -04:00
Sirius DevOps
20a266563c fix(aircraft): show planespotters photo in overlay popup
Planespotters 403s server clients whose User-Agent has no contact
email; compose default UA lacked one. Leaflet popup.update() also
re-ran the bindPopup factory and wiped the thumbnail after load.

- Send a contact UA on planespotters lookups; align compose default.
- Resize the popup without re-rendering content.
- Keep hex/src on live firefighter upserts so they get a photo too.
2026-08-28 23:28:28 -04:00
bb849515fb Merge pull request 'fix: stop summarizer from bailing to a canned empty brief' (#17) from fix/summarizer-empty-brief into master
All checks were successful
build-and-deploy / build-push-deploy (push) Successful in 3m20s
Reviewed-on: #17
2026-08-28 23:19:14 -04:00
Sirius DevOps
7822d45f37 fix: stop summarizer from bailing to a canned empty brief
The reduce/recap prompts told the model to AND STOP with "No qualifying…"
whenever nothing looked breaking enough, so full article batches became
one sentence. Always write summary_en from the provided stories; ticker
and map may still be empty. Futures/market tape stay ignored.
2026-08-28 23:20:40 -04:00
680aeea29f Merge pull request 'feat(aircraft): planespotters.net photo in aircraft popup' (#16) from feat/planespotters-photos into master
All checks were successful
build-and-deploy / build-push-deploy (push) Successful in 3m53s
Reviewed-on: #16
2026-08-28 22:58:29 -04:00
Sirius DevOps
665aaec22f feat(aircraft): planespotters.net photo in aircraft popup
Add planespotters.net latest-photo lookup for ADS-B aircraft.

- app/live_layers.py: fetch_planespotters_photo() (hex preferred, reg
  fallback) + _normalize_planespotter_photo(); 24h TTL cache (their ToS cap).
- app/main.py: GET /api/aircraft/photo?hex=...|reg=... (422/404/502).
- app/static/index.html: .ps-photo block in ADS-B popup; lazy load on
  popupopen; thumbnail links to photo page + photographer credit.

Server-side proxy, not browser fetch(): planespotters 403s any request
carrying an Origin header (every browser fetch() sends one). The thumbnail
binary is loaded by the browser straight from their CDN, never re-hosted.

Tests: 4 unit + 4 API contract (38 pass in the two files).
2026-08-28 22:58:04 -04:00
e568d013ca Merge pull request 'feat: 23:00 daily news recap; prompts drop futures' (#15) from feat/news-daily-recap into master
All checks were successful
build-and-deploy / build-push-deploy (push) Successful in 4m5s
Reviewed-on: #15
2026-08-28 22:53:59 -04:00
Sirius DevOps
49ee5fe7a3 feat: 23:00 daily news recap; prompts drop futures
Summarizer still runs the 15-min analyst, plus a 24h breaking-news recap
at 23:00 America/New_York. HUD pins kind=daily_recap. Prompts ignore
futures/market tape.
2026-08-28 22:53:31 -04:00
d25ca8ed9c Merge pull request 'fix: chunk FIRMS bulk inserts under asyncpg 32767 bind cap' (#14) from fix/firms-bind-limit into master
All checks were successful
build-and-deploy / build-push-deploy (push) Successful in 3m35s
Reviewed-on: #14
2026-08-28 22:20:48 -04:00
Sirius DevOps
5302e261d7 fix: chunk FIRMS bulk inserts under asyncpg 32767 bind cap
A ~90k-row INSERT ... ON CONFLICT dies with InterfaceError, so fire_loop
fails every poll and /api/health stays degraded (fires age > 30 min).
Chunk 2000 rows per statement, still one commit per poll.
2026-08-28 22:22:12 -04:00
19fa0db32c Merge pull request 'fix: pipeline reliability — WS, summarizer, ingest, health' (#13) from fix/pipeline-reliability into master
All checks were successful
build-and-deploy / build-push-deploy (push) Successful in 4m11s
Reviewed-on: #13
2026-08-28 21:49:56 -04:00
Sirius DevOps
d81ea68dec fix: pipeline reliability — WS, summarizer, ingest, health
Stop the live HUD reconnect storm (nginx WS snippet + backoff), copy
intel/nous_client into the summarizer image, and make event ingest
idempotent on URL. GDELT uses the DOC API; NWS no longer sends bbox;
FIRMS is one ON CONFLICT batch; GET /api/aircraft serves last-known.
Health reports freshness without 503ing docker. EONET + CISA KEV added.
2026-08-28 21:49:05 -04:00
c6bfc50705 Merge pull request 'feat: classify aircraft/vessels as military vs civilian in popups' (#12) from feat/ac-vessel-intel into master
All checks were successful
build-and-deploy / build-push-deploy (push) Successful in 4m11s
2026-08-28 20:58:51 -04:00
Sirius DevOps
ca15885e7f feat: classify aircraft/vessels as military vs civilian in popups
Aircraft: readsb dbFlags bit0, then ICAO combat types / US-DoD hex /
callsign. Popup shows type, operator, alt, vs, squawk, emitter class.
Vessels: ITU ship type 35 = military, 50/51/55 = government; cargo/
tanker/passenger/etc otherwise. Popup shows kind, flag, dest, nav,
IMO, size. Magenta glyphs = military, gold = law/gov.
2026-08-28 21:00:20 -04:00
dd8d665af0 Merge pull request 'feat: continuous news scrape + 15-min analyst, expand urls.txt' (#11) from feat/news-cadence-feeds into master
Some checks failed
build-and-deploy / build-push-deploy (push) Has been cancelled
Reviewed-on: #11
2026-08-28 20:56:02 -04:00
0d1c095296 Merge branch 'master' into feat/news-cadence-feeds 2026-08-28 20:55:43 -04:00
Sirius DevOps
8643153954 feat: continuous news scrape + 15-min analyst, expand urls.txt
Scraper loops with NEWS_SCRAPE_INTERVAL_S (default 10s after each
crawl). Summarizer runs every NEWS_SUMMARIZE_INTERVAL_S (default 900)
over the last 15 minutes of articles. Feed list replaced from the
k8s scrapy-urls configmap (334 sources).
2026-08-28 20:53:32 -04:00
dcb90b3a8b Merge pull request 'fix(ci): recreate named compose containers without collisions' (#10) from fix/ci-compose-reup into master
All checks were successful
build-and-deploy / build-push-deploy (push) Successful in 3m35s
Reviewed-on: #10
2026-08-28 20:34:59 -04:00
Sirius DevOps
b057ff48f9 fix(ci): recreate named compose containers without collisions
CI ran `compose up --force-recreate` without the ingest profile, so
compose tried to *create* osint-dashboard while the old name-pinned
container was still running. down + rm -f the known names, then up
--profile ingest. Volumes stay.
2026-08-28 20:36:03 -04:00
eb883f0801 Merge pull request 'fix: hydrate camera popup from GET /api/cameras/{id}' (#9) from fix/camera-popup-hydrate into master
Some checks failed
build-and-deploy / build-push-deploy (push) Failing after 3m14s
Reviewed-on: #9
2026-08-28 20:09:30 -04:00
Sirius DevOps
700f56ef51 fix: hydrate camera popup from GET /api/cameras/{id}
List payload is slim (no URLs). YouTube thumbs need source_url from the
detail endpoint, so the popup fetches the full row before rendering.
2026-08-28 20:04:53 -04:00
Sirius DevOps
1b1f2d3b3a fix: merge alembic heads so the dashboard stops crash-looping
Some checks failed
build-and-deploy / build-push-deploy (push) Failing after 2m59s
005_phase2 and 005_news_items both revised 004_camera_enum. Entrypoint
`alembic upgrade head` then failed and the app never bound :8000 (502
over Tailscale). Merge revision + `upgrade heads`.
2026-08-28 18:29:06 -04:00
f1d70d3f4c Merge pull request 'feat: Nous Portal news summarizer, ticker flags, map pins' (#8) from feat/nous-news-summarizer into master
All checks were successful
build-and-deploy / build-push-deploy (push) Successful in 3m27s
Reviewed-on: #8
2026-08-28 17:43:40 -04:00
Sirius DevOps
d2bf55183c fix: let Settings model reach summarizer; cap map pins at 20 2026-08-28 17:06:55 -04:00
Sirius DevOps
450de45bfc docs: nous portal news summarizer 2026-08-28 17:06:55 -04:00
Sirius DevOps
496d13b8b0 feat: critical news pins on the osint map 2026-08-28 17:06:55 -04:00
Sirius DevOps
3b4dd49253 feat: news ticker shows flagged hourly items 2026-08-28 17:06:55 -04:00
Sirius DevOps
610666ef6a feat: settings model selector for news summarizer 2026-08-28 17:06:55 -04:00
Sirius DevOps
d93ad88956 chore: point news-summarizer compose at Nous portal 2026-08-28 17:06:55 -04:00
Sirius DevOps
8f3699f133 feat: summarize hourly news via Nous portal into brief/ticker/map 2026-08-28 17:06:55 -04:00
Sirius DevOps
1616413e65 feat: persist summarizer model in app_settings 2026-08-28 17:06:55 -04:00
Sirius DevOps
f50268ef81 feat: register NOUS_API_KEY in dashboard keystore 2026-08-28 17:06:55 -04:00
Sirius DevOps
5b23578302 feat: news ticker and map API contracts 2026-08-28 17:06:55 -04:00
Sirius DevOps
cbc573b830 feat: add news_items table and summary model column 2026-08-28 17:06:55 -04:00
Sirius DevOps
4b00eed566 feat: nous portal chat client for news summarizer 2026-08-28 17:06:55 -04:00
Sirius DevOps
96590bc16d test: add news summarizer intel parser tests + impl 2026-08-28 17:06:55 -04:00
104fb2072a Merge pull request 'feat: lean-Pi WS fan-out, geofences, DVR playback, fire/aircraft correlation' (#7) from feat/osint-lean-pi-geofence-dvr into master
All checks were successful
build-and-deploy / build-push-deploy (push) Successful in 3m34s
Reviewed-on: #7
2026-08-28 09:40:33 -04:00
Sirius DevOps
5ea9a4e879 feat: lean-Pi WS fan-out, geofences, DVR playback, fire/aircraft correlation
Phase 1: in-memory ConnectionManager viewport fan-out, 500ms map debounce,
cachetools TTLCache, background masscan/ffmpeg, compose memory caps.

Phase 2: PostGIS geofences + ST_Intersects alerts, Timescale 1-min CAGGs
and timestamp playback, FIRMS/WFIGS x firefighting ADS-B within 20 miles.

No Redis/Kafka/Celery.
2026-08-28 09:35:18 -04:00
8978324d33 ci: push images to Forgejo over loopback, not Cloudflare
All checks were successful
build-and-deploy / build-push-deploy (push) Successful in 4m42s
GITHUB_TOKEN cannot write packages (401). PAT works, but layers
over https://forgejo.siriusdevops.com 413 at Cloudflare's ~100MB
POST cap. Push 127.0.0.1:3000; pull names stay public.
2026-08-28 00:17:52 -04:00
6e4209e745 ci: login to Forgejo registry with PAT, not GITHUB_TOKEN
Some checks failed
build-and-deploy / build-push-deploy (push) Failing after 9s
Actions GITHUB_TOKEN reports Login Succeeded then 401s on
POST /v2/.../blobs/uploads/. User PAT can write packages.
2026-08-28 00:15:19 -04:00
1e49062b78 Merge pull request 'ci: publish all OSINT images to Forgejo registry' (#6) from feat/forgejo-registry-images into master
Some checks failed
build-and-deploy / build-push-deploy (push) Failing after 8s
Reviewed-on: #6
2026-08-28 00:08:04 -04:00
Sirius Devops
9a9c740b9a ci: publish all OSINT images to Forgejo registry
Push api/pg/news-scraper/news-summarizer to
forgejo.siriusdevops.com/sirius/* on every master build, then deploy
on the Pi runner. No GHCR. Helper script retags pulls to localhost/*
for compose.
2026-08-28 03:55:00 +00:00
Sirius DevOps
32320635dc feat: AIS stream follows the map viewport
All checks were successful
build-and-deploy / build (push) Successful in 2m32s
The Vessels layer now retunes the server-side AISStream subscription to the
client viewport instead of a static AISSTREAM_BBOX. The frontend POSTs its
quantized viewport box to /api/vessels/subscribe on moveend; the ais_stream
worker coalesces and applies it at the service's 1 subscription/s cap, then
last-known positions for the new area arrive within a couple of seconds (the
frontend does one follow-up fetch after retuning). Bounds the in-memory
vessel store across regions. Key stays server-side.
2026-08-27 21:50:38 -04:00
c58f66770c Merge pull request 'fix: restore news ticker on the map HUD' (#5) from fix/news-ticker-hud into master
Some checks failed
build-and-deploy / build (push) Has been cancelled
Reviewed-on: #5
2026-08-27 21:45:42 -04:00
Sirius DevOps
0a7a9236d6 fix: restore news ticker on the map HUD
Boot deferred news to the News view for overlay speed. Bring the footer
ticker back after map init starts; article bodies stay off the wire.
2026-08-27 21:47:22 -04:00
61feeb6c5a Merge pull request 'perf: faster map overlay load' (#4) from perf/map-layer-load into master
All checks were successful
build-and-deploy / build (push) Successful in 2m47s
Reviewed-on: #4
2026-08-27 21:24:24 -04:00
Sirius DevOps
734c310d2c perf: abort stale overlay fetches, skip same-cell rebuilds, update markers in place
moveend aborts in-flight overlay JSON and skips work when the 0.01° cell
is unchanged. Live aircraft/trains/vessels diff-update by id. Camera popup
HTML is built on open. Alert/perimeter polygons paint on canvas.
2026-08-27 21:23:25 -04:00
Sirius DevOps
435f63e473 perf: shared httpx client, gzip, overlay cache-control
Reuse one TLS pool for overlay upstreams (8s/3s timeouts). Gzip JSON
over 1 KB. Aircraft/vessels Cache-Control max-age=5, alerts/perimeters 30.
Lifespan replaces deprecated on_event startup.
2026-08-27 21:21:19 -04:00
Sirius DevOps
8f89994201 perf: slim fires/cameras/events/news map payloads
Heatmap uses {lat,lon,i,c}. Camera list drops URLs (detail via GET
/api/cameras/{id}). Event blips skip body. News omits content unless
include_content=true.
2026-08-27 21:19:42 -04:00