085061492e
All checks were successful
build-and-deploy / build (push) Successful in 2m0s
Continuous whole-IPv4 rolling sweep for open TCP 554, feeding the same cameras table as the passive scraper (discovery_source=masscan). - masscan_config.py: env-driven knobs (range, ports, rate, retries, excludes) - masscan_scanner.py: JSON-lines parser, rtsp://IP/ URL + url_hash dedupe, ip-api geolocation, insert/refresh, NATS publish for new finds - run_masscan_service.py: long-lived rolling-sweep runner (streams results in, restarts on pass completion); fails closed without an excludefile - deploy/: systemd unit + README + excludes file for the Pi host - .env.example: masscan section Verified end-to-end against a local Postgres: parse, insert, and dedupe (0 new on re-ingest) all pass.
64 lines
2.9 KiB
Python
64 lines
2.9 KiB
Python
"""Active camera-discovery configuration (masscan-based, env-driven).
|
|
|
|
All knobs read from the environment with safe defaults. The scanner targets
|
|
open TCP port 554 (RTSP — the typical IP-camera port) across a configured
|
|
range and feeds results into the same `cameras` table as the passive scraper
|
|
(discovery_source='masscan'), deduped by URL hash.
|
|
|
|
ETHICS / SCOPE (mirrors camera_scraper.py):
|
|
* Detection only — a SYN port scan for OPEN hosts. No credential guessing,
|
|
no login attempts, no banner grabbing, and no access to camera feeds.
|
|
* Private / reserved ranges are excluded via MASSCAN_EXCLUDEFILE so the
|
|
scanner never probes RFC1918, loopback, link-local, multicast, or the
|
|
bogons. Fail closed if the excludefile is missing.
|
|
|
|
TIMING REALITY: at the default conservative rate of 10,000 pps a full IPv4
|
|
sweep (0.0.0.0/0, ~4.29B addresses) takes ~119 hours (~5 days). This is
|
|
therefore a CONTINUOUS ROLLING SWEEP, not a "finish in a day" job: masscan
|
|
streams open hosts to stdout and the runner ingests them incrementally, then
|
|
restarts the sweep when a pass completes. New cameras are detected as they
|
|
appear on each pass.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
|
|
# Path to the masscan binary (installed on the Pi host).
|
|
MASSCAN_BIN = os.getenv("MASSCAN_BIN", "masscan")
|
|
|
|
# CIDR(s) to sweep. Default = the whole public IPv4 space.
|
|
MASSCAN_RANGE = os.getenv("MASSCAN_RANGE", "0.0.0.0/0")
|
|
|
|
# Port(s) to probe. Default 554 = RTSP, the typical IP-camera port.
|
|
MASSCAN_PORTS = os.getenv("MASSCAN_PORTS", "554")
|
|
|
|
# Packets/sec. 10,000 = conservative, polite, residential-IP friendly
|
|
# (~5 days for a full sweep). Raise carefully on a capable host/VPS.
|
|
MASSCAN_RATE = int(os.getenv("MASSCAN_RATE", "10000"))
|
|
|
|
# Retransmission count. 1 maximizes unique-host coverage at low rate; the
|
|
# default (10) spends most of the budget re-probing the same hosts.
|
|
MASSCAN_RETRIES = int(os.getenv("MASSCAN_RETRIES", "1"))
|
|
|
|
# Seconds to keep listening for straggler responses after the last probe.
|
|
# 0 avoids a 10s tail per pass; tiny loss of the very last hosts is fine
|
|
# since the sweep repeats.
|
|
MASSCAN_WAIT = int(os.getenv("MASSCAN_WAIT", "0"))
|
|
|
|
# Excludefile path on the Pi host. Must contain RFC1918/loopback/link-local/
|
|
# multicast/bogons so the scanner never probes private ranges. Fail closed if
|
|
# the file is absent (the runner refuses to start rather than scan wide).
|
|
MASSCAN_EXCLUDEFILE = os.getenv(
|
|
"MASSCAN_EXCLUDEFILE", "/etc/osint-dashboard/masscan-excludes.txt"
|
|
)
|
|
|
|
# Ingest batch size — flush this many newly-seen hosts to the DB per round.
|
|
MASSCAN_FLUSH_EVERY = int(os.getenv("MASSCAN_FLUSH_EVERY", "250"))
|
|
|
|
# NATS subject newly-found cameras are published on (same feed as the
|
|
# passive scraper so the shared ingester persists them).
|
|
MASSCAN_NATS_SUBJECT = os.getenv("MASSCAN_NATS_SUBJECT", "events.camera")
|
|
|
|
# discovery_source tag written into the cameras table.
|
|
MASSCAN_DISCOVERY_SOURCE = os.getenv("MASSCAN_DISCOVERY_SOURCE", "masscan")
|