feat(cameras): UDOT IBI 511 traffic cameras -> cameras table #38
Loading…
Add table
Reference in a new issue
No description provided.
Delete branch "osint-dashboard/t_5bfc52e8-osint-udot-ibi-511-cameras-cameras-table"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Adds the Utah DOT traffic camera feed from the public IBI 511 platform (prod-ut.ibi511.com, no API key, discovery_source=udot) to the existing
camerastable pipeline.What
parse_udot_ibi_page— pure parser for one DataTables response: skipsimages[0].blocked/disabledand coord-less rows, parses WKTPOINT(lng lat)fromlatLng.geography.wellKnownText, drops anything outside the Utah bbox (lat 36.9-42.1, lon -114.2--108.9).scrape_udot_ibi— paginated fetcher: POSTsstart/lengthform fields (server caps at 100 rows/page), walks pages viarecordsTotal, capped atUDOT_IBI_MAX_PAGES(default 40) so a runaway total cannot fan out.https://prod-ut.ibi511.com/map/Cctv/{id}; stored once, frames are never scraped. Dedupe viaurl_hash. IdentifyingOSINT_USER_AGENT+X-Requested-With: XMLHttpRequest.run_cyclealongside the existing sources.Tests
Pure parser tests (no live IBI in CI): WKT lng/lat order, blocked/disabled skip, bbox drop, malformed payload, missing-WKT drop — in
tests/test_live_layers.py.Verified live: 2075 records, 100 rows/page, 21 pages; parser yields 100/100 in-bbox unblocked rows on page 1, clean dedupe across pages.
Add parse_udot_ibi_page + scrape_udot_ibi for the UDOT IBI 511 traffic camera feed (prod-ut.ibi511.com, no key). DataTables endpoint is POST form-encoded and caps at 100 rows/page regardless of `length`; page walk uses recordsTotal with a UDOT_IBI_MAX_PAGES (default 40) runaway cap. Skip images[0].blocked/disabled, parse WKT POINT(lng lat) from latLng.geography.wellKnownText, drop outside the Utah bbox (lat 36.9-42.1, lon -114.2--108.9). discovery_source=udot, stable source_url == snapshot_url == /map/Cctv/{id} (never scrape frames), url_hash dedupe, OSINT_USER_AGENT + X-Requested-With header. Unit tests: WKT lng/lat order, blocked/disabled skip, bbox drop, malformed payload, missing WKT.