sirius/osint-dashboard
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(cameras): MDOT MiDrive parser -> cameras table #34

Merged
sirius merged 1 commit from osint-dashboard/t_8be8d860-osint-mdot-midrive-cameras-cameras-table into master 2026-08-31 21:50:46 -04:00
Conversation 0 Commits 1 Files changed 3 +141 -1
sirius commented 2026-08-31 21:29:29 -04:00
Owner
Copy link

Summary

Add parse_mdot_json for the official MDOT MiDrive GET /MiDrive/camera/list JSON, whose fields carry rendered HTML: coordinates and the stable id live in the county field's map link (lat=/lon=/id=), and the JPEG still lives in the image field's <img src>.

  • discovery_source=mdot, JPEG/snapshot_url only (RTSP rejected).
  • Stable source_url keyed on camera id → dedupes via existing url_hash.
  • Michigan bbox filter (lat 41.6–48.3, lon -90.5–-82.1); coord-less rows dropped.
  • Wired into scrape_source dispatch and CAMERA_SOURCE_URLS defaults.

Tests

  • test_parse_mdot_extracts_html_fields_and_bbox_filters — HTML field extract, bbox drop, missing-coords/image drop, RTSP drop.
  • test_parse_mdot_handles_malformed_payload — non-JSON / non-list payloads return [].

Verified against the live endpoint: 806 cameras parsed, all in-bbox, 806 unique url_hash keys.

## Summary Add `parse_mdot_json` for the official MDOT MiDrive `GET /MiDrive/camera/list` JSON, whose fields carry rendered HTML: coordinates and the stable id live in the `county` field's map link (`lat=`/`lon=`/`id=`), and the JPEG still lives in the `image` field's `<img src>`. - `discovery_source=mdot`, JPEG/`snapshot_url` only (RTSP rejected). - Stable `source_url` keyed on camera id → dedupes via existing `url_hash`. - Michigan bbox filter (lat 41.6–48.3, lon -90.5–-82.1); coord-less rows dropped. - Wired into `scrape_source` dispatch and `CAMERA_SOURCE_URLS` defaults. ## Tests - `test_parse_mdot_extracts_html_fields_and_bbox_filters` — HTML field extract, bbox drop, missing-coords/image drop, RTSP drop. - `test_parse_mdot_handles_malformed_payload` — non-JSON / non-list payloads return []. Verified against the live endpoint: 806 cameras parsed, all in-bbox, 806 unique url_hash keys.
sirius added 1 commit 2026-08-31 21:29:29 -04:00 
Add parse_mdot_json for the MDOT MiDrive camera/list JSON where
coordinates/id live in the county field's map link (lat=/lon=/id=) and
the JPEG still lives in the image field's <img src>. Michigan bbox
filter, missing-coords drop, RTSP reject. discovery_source=mdot,
stable source_url keyed on camera id, url_hash dedupe. Wired into
scrape_source dispatch + CAMERA_SOURCE_URLS defaults.

Unit tests: HTML field extract, bbox drop, missing-coords/image drop,
malformed payload.
sirius merged commit 9712ad03a7 into master 2026-08-31 21:50:46 -04:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: sirius/osint-dashboard#34
No description provided.
Delete branch "osint-dashboard/t_8be8d860-osint-mdot-midrive-cameras-cameras-table"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?