osint-dashboard/docs/news.md

News pipeline — scraper + summarizer

The OSINT dashboard ingests ~257 global news RSS sources hourly and produces LLM master summaries. Both services were vendored from the upstream ~/Projects/newsPipeline project and re-integrated here to replace the old k8s CronJob choreography with in-compose scheduling against the EXISTING osint-db — no second Postgres.

Architecture

257 RSS feeds (news/scraper/urls.txt)
        │
        ▼
news-scraper   (Scrapy, hourly :00) ──► articles table (osint-db)
        │                                      │
        │                                      ▼
news-summarizer (Gemini map-reduce, hourly :05) ──► article_summaries table
                                                      │
                                                      ▼
                              GET /api/news  ·  GET /api/news/summaries

Component	Image	Container	Scheduling
Scraper	localhost/osint-news-scraper	osint-news-scraper	wall-clock loop, minute NEWS_SCRAPE_MINUTE (default :00)
Summarizer	localhost/osint-news-summarizer	osint-news-summarizer	wall-clock loop, minute NEWS_SUMMARIZE_MINUTE (default :05)

Both services live under the ingest compose profile (same as the ingester and camera-scraper): docker compose --profile ingest up -d.

Data flow

1. Scraper — news/scraper/run_news_scraper.py runs scrapy crawl articles (spider news/scraper/newsScraper/spiders/news_spider.py) at the top of each hour. The spider reads the RSS feed URLs from urls.txt, follows each <item> link, extracts the main article body, and the PostgresPipeline writes to articles with URL-based dedup (ON CONFLICT (url) DO NOTHING).
2. Summarizer — news/summerizer/run_news_summarizer.py runs summarizer.py at :05 past each hour. It reads articles from the last SUMMARY_WINDOW_HOURS, map-reduces them through Gemini (SUMMARY_MODEL, default gemini-2.0-flash), and inserts one master summary into article_summaries.

Scheduling is done with small in-compose wall-clock loops (not host cron): each loop runs once on boot (*_RUN_ON_START=1, seeds data fast) then sleeps until the next scheduled minute. The loop is serial, so a run that overruns its slot simply shifts to the next boundary — two crawls/summaries never overlap.

The articles and article_summaries tables are created by the idempotent alembic migration 003_news (also created by the scraper's own CREATE TABLE IF NOT EXISTS, so container startup order doesn't matter).

Endpoints

GET /api/news — recent articles

Query param	Meaning	Default
domain	filter by source domain (e.g. www.reuters.com)	none
since	only articles captured at/after this UTC instant (ISO-8601)	none
limit	max rows	50 (max 500)
offset	pagination offset	0

[
  {
    "id": 1,
    "title": "…",
    "url": "https://…",
    "content": "full extracted article text…",
    "domain": "www.reuters.com",
    "timestamp": "2026-08-24T18:10:00Z"
  }
]

GET /api/news/summaries — master LLM summaries

Query param	Meaning	Default
since	only summaries generated at/after this UTC instant	none
limit	max rows	20 (max 100)
offset	pagination offset	0

[
  {
    "id": 1,
    "summary_text": "master LLM summary (markdown)…",
    "batch_timestamp": "2026-08-24T18:10:00Z"
  }
]

Configuration (all via env / .env)

Var	Default	Notes
GEMINI_API_KEY	(blank)	Required for summaries. Unset = summarizer logs and idles (never crashes).
SUMMARY_MODEL	gemini-2.0-flash	Gemini model id.
NEWS_BATCH_SIZE	50	Articles per map-phase batch.
SUMMARY_WINDOW_HOURS	1	How far back the summarizer looks for new articles.
INCLUDE_FUTURES	0	Legacy futures-prices coupling (upstream pipeline). OFF for OSINT; set 1 + install yfinance to enable.
NEWS_SCRAPE_MINUTE	0	Wall-clock minute the scraper fires.
NEWS_SUMMARIZE_MINUTE	5	Wall-clock minute the summarizer fires.
NEWS_SCRAPE_RUN_ON_START	1	Run one scrape immediately on container start.
NEWS_SUMMARIZE_RUN_ON_START	1	Run one summarize immediately on container start.
NEWS_LOG_LEVEL	INFO	Scrapy log level.
TELEGRAM_TOKEN / TELEGRAM_CHAT_ID	(blank)	Reserved for the (out-of-scope) Telegram delivery bot.

DB_* for both services is mapped to the shared osint-db credentials (DB_HOST=db, same DB_USER/DB_PASSWORD/DB_NAME as the rest of the stack).

Prompts

Both prompts are env-overridable — the default MAP_PROMPT is OSINT-neutral (facts, locations, entities, category, OSINT signal per article) and the default SUMMARY_PROMPT produces a concise executive summary of the most impactful items (with a "no qualifying events" escape hatch). Upstream's futures/markets prompt language is gated behind INCLUDE_FUTURES=1.

Tests

tests/test_api_news.py — DB-backed API contract tests (auto-skip without a reachable test database, same as the FIRMS tests):

DB_HOST=... DB_PORT=... DB_USER=osint DB_PASSWORD=... DB_NAME=osint_data \
  pytest tests/test_api_news.py -v

Live verification

End-to-end (real crawl → DB → API) is verified after deploy on the Pi: check docker compose --profile ingest logs -f news-scraper news-summarizer, then curl -s localhost:8000/api/news | head. Summaries additionally require GEMINI_API_KEY to be set in .env on the Pi.