Commit graph

6 commits

Author SHA1 Message Date
Sirius DevOps
a9ca98e791 feat(cameras): UDOT IBI 511 parser -> cameras table
Add parse_udot_ibi_page + scrape_udot_ibi for the UDOT IBI 511 traffic
camera feed (prod-ut.ibi511.com, no key). DataTables endpoint is POST
form-encoded and caps at 100 rows/page regardless of `length`; page walk
uses recordsTotal with a UDOT_IBI_MAX_PAGES (default 40) runaway cap.

Skip images[0].blocked/disabled, parse WKT POINT(lng lat) from
latLng.geography.wellKnownText, drop outside the Utah bbox
(lat 36.9-42.1, lon -114.2--108.9). discovery_source=udot, stable
source_url == snapshot_url == /map/Cctv/{id} (never scrape frames),
url_hash dedupe, OSINT_USER_AGENT + X-Requested-With header.

Unit tests: WKT lng/lat order, blocked/disabled skip, bbox drop,
malformed payload, missing WKT.
2026-08-31 21:35:07 -04:00
Sirius DevOps
f779b1f225 feat: toggleable live map feeds (ADS-B, trains, AIS, radar, WFIGS, NWS)
Wire the free data streams from docs/free-data-streams.md into the
dashboard as layer-panel toggles. Third-party APIs are proxied/cached
in FastAPI; raster tiles (IEM, RainViewer, GIBS) stay in the browser.

- Aircraft via ADSB.lol viewport poll (bbox required, radius ≤ 150 nm)
- Amtraker trains, NHC storms, WFIGS incidents/perimeters
- NWS + IEM SBW as /api/weather-alerts (does not collide with /api/alerts)
- AISStream worker is server-side only and idles without AISSTREAM_API_KEY
- Caltrans CWWP2 D1–D12 camera parser; FIRMS dual-write NOAA-20/21
2026-08-27 19:08:30 -04:00
Sirius DevOps
0c7f80655e cameras: Live-Environment-Streams + HLS.js player
All checks were successful
build-and-deploy / build (push) Successful in 3m8s
Ingest ~4.2k direct HLS/YouTube feeds (VDOT, MDSHA, DelDOT, Iowa DOT,
OpenCCTV, etc.) from the public GeoJSON catalog. Popup plays HLS via a
CORS-safe playlist proxy + vendored hls.js, YouTube via embed, with
ffmpeg-MJPEG fallback. ALERTWest rows now use a stable camera id so
hourly scrapes do not duplicate.
2026-08-27 15:54:54 -04:00
Sirius DevOps
f3c35c2230 cameras: ingest ALERTCalifornia/ALERTWest official public JPEGs
All checks were successful
build-and-deploy / build (push) Successful in 3m36s
Adds the documented getCameraDataByLoc API (~10k public wildfire, DOT,
and FAA stills with lat/lon). Masscan will not produce viewable feeds;
this will. Map bbox fetch raised to 2000 markers.
2026-08-27 15:46:49 -04:00
Sirius DevOps
347ee3a642 Camera scraper: live public source (fury999io/public-ip-cams), markdown-list parser, batch IP geolocation via ip-api; fix empty-env fallback so default source applies
All checks were successful
build-and-deploy / build (push) Successful in 1m38s
2026-08-24 21:03:33 -04:00
Sirius DevOps
69a30115cc Add open IP camera discovery scraper (OSINT map)
Some checks failed
build-and-deploy / build (push) Failing after 5s
- camera_scraper: public-directory-only discovery (Insecam-style HTML +
  plain-text lists), hard private-range guard (fail closed), per-host
  rate limiting, Nominatim geocoding at <=1 req/s, TTL'd local snapshot
  cache, sha256 url_hash dedupe with Postgres upsert
- cameras table (migration 002) + /api/cameras?bbox= + snapshot endpoint
  with cache passthrough in main.py
- run_camera_service: long-running cycle worker following NATS->ingester
  pattern; publishes events.camera for shared ingester
- docker-compose camera-service profile, .env.example knobs

Verified E2E against TimescaleDB+PostGIS: private-range entries dropped,
dedupe across cycles holds, bbox query returns expected rows.
2026-08-24 14:54:51 -04:00