sirius/osint-dashboard
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ingest): skip known KEV/FIRMS rows; CI skips unchanged images #31

Merged
sirius merged 1 commit from feat/lean-ingest-ci-skip-unchanged into master 2026-08-29 19:37:39 -04:00
Conversation 0 Commits 1 Files changed 9 +438 -94
sirius commented 2026-08-29 19:30:33 -04:00
Owner
Copy link

Summary

  • Stop the ingester from republishing the full CISA KEV catalog (1685 NIST CVE URLs) every 5 minutes when they are already in event_dedup.
  • Stop FIRMS from re-inserting ~325k global hotspots every 15 minutes: skip byte-identical CSVs, persist only new rows.
  • CI rebuilds only images whose paths changed. Timescale/osint-db is not pulled, rebuilt, or bounced unless Dockerfile.pg changes (or workflow_dispatch rebuild_pg).

Why

Pi CPU was pegged: KEV skip-duplicate loop + FIRMS 2-day global ON CONFLICT, and every merge rebuilt/pulled the DB image then compose down'd Postgres.

Test plan

  • PYTHONPATH=app pytest tests/test_fire_sources.py tests/test_new_sources.py tests/test_upstream_cache.py tests/test_event_ingest.py tests/test_fire_ingest.py — 31 passed, 2 skipped
  • After merge: first FIRMS poll may still replay once (empty in-process cache); later polls should be no-op/delta. KEV should log 0 published / 1685 already known.
  • CI log should show pg=0 unless Dockerfile.pg changed; osint-db should stay up.

Does not deploy until merged to master.

## Summary - Stop the ingester from republishing the full CISA KEV catalog (1685 NIST CVE URLs) every 5 minutes when they are already in `event_dedup`. - Stop FIRMS from re-inserting ~325k global hotspots every 15 minutes: skip byte-identical CSVs, persist only new rows. - CI rebuilds only images whose paths changed. Timescale/`osint-db` is not pulled, rebuilt, or bounced unless `Dockerfile.pg` changes (or workflow_dispatch `rebuild_pg`). ## Why Pi CPU was pegged: KEV skip-duplicate loop + FIRMS 2-day global ON CONFLICT, and every merge rebuilt/pulled the DB image then `compose down`'d Postgres. ## Test plan - [x] `PYTHONPATH=app pytest tests/test_fire_sources.py tests/test_new_sources.py tests/test_upstream_cache.py tests/test_event_ingest.py tests/test_fire_ingest.py` — 31 passed, 2 skipped - [ ] After merge: first FIRMS poll may still replay once (empty in-process cache); later polls should be no-op/delta. KEV should log `0` published / `1685 already known`. - [ ] CI log should show `pg=0` unless `Dockerfile.pg` changed; `osint-db` should stay up. Does **not** deploy until merged to `master`.
sirius added 1 commit 2026-08-29 19:30:34 -04:00 
CISA KEV republished 1685 NIST URLs every 5 min; FIRMS re-inserted
~325k global hotspots every 15 min. Producer now skips URLs already
in event_dedup and FIRMS CSVs that are unchanged (delta-only persist).

CI rebuilds only images whose paths changed and never pulls/rebuilds
Timescale or bounces osint-db unless Dockerfile.pg changes.
sirius merged commit fdf5969e27 into master 2026-08-29 19:37:39 -04:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: sirius/osint-dashboard#31
No description provided.
Delete branch "feat/lean-ingest-ci-skip-unchanged"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?