sirius/osint-dashboard
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: continuous news scrape + 15-min analyst, expand urls.txt #11

Merged
sirius merged 2 commits from feat/news-cadence-feeds into master 2026-08-28 20:56:02 -04:00
Conversation 0 Commits 2 Files changed 11 +434 -376
sirius commented 2026-08-28 20:51:42 -04:00
Owner
Copy link

Summary

  • Scraper runs back-to-back (NEWS_SCRAPE_INTERVAL_S, default 10s after each crawl) instead of hourly :00.
  • Analyst runs every 15 minutes (NEWS_SUMMARIZE_INTERVAL_S=900). Look-back is SUMMARY_WINDOW_MINUTES=15. Idempotency is interval-based (not UTC hour).
  • urls.txt replaced from the k8s scrapy-urls configmap — 334 sources (RSS + homepages).

After merge: rebuild osint-news-scraper and osint-news-summarizer (compose --profile ingest).

## Summary - **Scraper** runs back-to-back (`NEWS_SCRAPE_INTERVAL_S`, default 10s after each crawl) instead of hourly :00. - **Analyst** runs every 15 minutes (`NEWS_SUMMARIZE_INTERVAL_S=900`). Look-back is `SUMMARY_WINDOW_MINUTES=15`. Idempotency is interval-based (not UTC hour). - **urls.txt** replaced from the k8s `scrapy-urls` configmap — 334 sources (RSS + homepages). After merge: rebuild `osint-news-scraper` and `osint-news-summarizer` (`compose --profile ingest`).
sirius added 1 commit 2026-08-28 20:51:42 -04:00 
Scraper loops with NEWS_SCRAPE_INTERVAL_S (default 10s after each
crawl). Summarizer runs every NEWS_SUMMARIZE_INTERVAL_S (default 900)
over the last 15 minutes of articles. Feed list replaced from the
k8s scrapy-urls configmap (334 sources).
sirius added 1 commit 2026-08-28 20:55:45 -04:00
sirius merged commit dd8d665af0 into master 2026-08-28 20:56:02 -04:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: sirius/osint-dashboard#11
No description provided.
Delete branch "feat/news-cadence-feeds"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?