osint-dashboard/docs/news.md

News pipeline — scraper + Nous Portal summarizer

The OSINT dashboard ingests ~257 global news RSS sources hourly and produces an English LLM brief plus flagged ticker/map rows. Both services were vendored from the upstream ~/Projects/newsPipeline project and re-integrated here to replace the old k8s CronJob choreography with in-compose scheduling against the EXISTING osint-db — no second Postgres. The LLM is Nous Portal (inference-api.nousresearch.com) — not Gemini.

Architecture

257 RSS feeds (news/scraper/urls.txt)
        │
        ▼
news-scraper   (Scrapy, hourly :00) ──► articles table (osint-db)
        │                                      │
        │                                      ▼
news-summarizer (Nous Portal map-reduce, :05) ──► article_summaries + news_items
                                                      │
                                                      ▼
     GET /api/news · /api/news/summaries · /api/news/ticker · /api/news/map
     GET /api/news/models · GET/PUT /api/settings

Component	Image	Container	Scheduling
Scraper	localhost/osint-news-scraper	osint-news-scraper	wall-clock loop, minute NEWS_SCRAPE_MINUTE (default :00)
Summarizer	localhost/osint-news-summarizer	osint-news-summarizer	wall-clock loop, minute NEWS_SUMMARIZE_MINUTE (default :05)

Both services live under the ingest compose profile (same as the ingester and camera-scraper): docker compose --profile ingest up -d.

The summarizer is a batch sidecar, not a live overlay. Do not reuse GET /api/alerts (dashboard entity/keyword alerts). Do not stuff news into overlay_catalog() — /api/map/layers overlays stays live upstream feeds (GET /api/news exact key set is unchanged on purpose).

Data flow

1. Scraper — news/scraper/run_news_scraper.py runs scrapy crawl articles (spider news/scraper/newsScraper/spiders/news_spider.py) at the top of each hour. The spider reads the RSS feed URLs from urls.txt, follows each <item> link, extracts the main article body, and the PostgresPipeline writes to articles with URL-based dedup (ON CONFLICT (url) DO NOTHING).
2. Summarizer — news/summerizer/run_news_summarizer.py runs summarizer.py at :05 past each hour. It reads articles from the last SUMMARY_WINDOW_HOURS, map-reduces them through Nous Portal (SUMMARY_MODEL / Settings, default Hermes-4.3-36B), writes the English brief to article_summaries (column model is the LLM id), and flagged ticker/map rows to news_items.

Scheduling is done with small in-compose wall-clock loops (not host cron): each loop runs once on boot (*_RUN_ON_START=1, seeds data fast) then sleeps until the next scheduled minute. The loop is serial, so a run that overruns its slot simply shifts to the next boundary — two crawls/summaries never overlap.

Hour-truncation idempotency: if article_summaries already has a row for the current UTC hour, the summarizer skips (prevents double-pins on RUN_ON_START recreate). Set NEWS_SUMMARIZE_FORCE=1 to ignore that skip.

The articles and article_summaries tables are created by the idempotent alembic migration 003_news (also created by the scraper's own CREATE TABLE IF NOT EXISTS). news_items is alembic 005_news_items. Container startup order doesn't matter.

Keys and Settings

• NOUS_API_KEY — paste in the dashboard Keys UI (api_keys / keystore.KEY_REGISTRY). Env / .env is an override (env wins, same as FIRMS). Never returned by any API; never emitted into index.html; never proxied from the browser.
• Idle without a key — if env is unset and the keystore row is empty, the summarizer logs and idles (never crashes). News intel APIs return [].
• Model — non-secret. Settings UI model selector PUT /api/settings { "summary_model": "…" } stores SUMMARY_MODEL in app_settings (1–128 chars). GET /api/settings echoes {summary_model, nous_base_url}. nous_base_url is read-only. Default Hermes-4.3-36B. Live catalog is best-effort GET /api/news/models.

Endpoints

GET /api/news — recent articles

Key set unchanged (no lat/lon on articles; geo lives on /api/news/map).

Query param	Meaning	Default
domain	filter by source domain (e.g. www.reuters.com)	none
since	only articles captured at/after this UTC instant (ISO-8601)	none
limit	max rows	50 (max 500)
offset	pagination offset	0
include_content	include full article body	false

[
  {
    "id": 1,
    "title": "…",
    "url": "https://…",
    "content": null,
    "domain": "www.reuters.com",
    "timestamp": "2026-08-24T18:10:00Z"
  }
]

GET /api/news/summaries — master LLM briefs

Query param	Meaning	Default
since	only summaries generated at/after this UTC instant	none
limit	max rows	20 (max 100)
offset	pagination offset	0

[
  {
    "id": 1,
    "summary_text": "English markdown brief…",
    "batch_timestamp": "2026-08-24T18:10:00Z",
    "model": "Hermes-4.3-36B"
  }
]

model is additive. Empty DB → [] (no crash).

GET /api/news/ticker — flagged HUD headlines

Critical/high news_items with kind=ticker only. Do not reuse GET /api/alerts. Bottom HUD #nt-track scrolls these rows, not a dump of the whole brief.

Query param	Meaning	Default
since	only items created at/after this UTC instant	none
limit	max rows	20 (max 50)

[
  {
    "id": 1,
    "headline": "…",
    "importance": "critical",
    "location_name": "Kyiv",
    "url": "https://…",
    "created_at": "2026-08-24T18:10:00Z"
  }
]

GET /api/news/map — geolocated critical/high pins

Only rows with valid lat/lon. Optional bbox. No zoom skip — world view is the point. Layer-panel toggle uses this dedicated path (same as event blips), not overlay_catalog.

Query param	Meaning	Default
bbox	minlon,minlat,maxlon,maxlat	all flagged pins
since	only items created at/after this UTC instant	last 24 hours
limit	max rows	200 (max 500)

Malformed bbox → 422.

[
  {
    "id": 1,
    "headline": "…",
    "importance": "high",
    "location_name": "Kyiv",
    "lat": 50.45,
    "lon": 30.52,
    "location_confidence": "city",
    "category": "military/conflict",
    "url": "https://…",
    "created_at": "2026-08-24T18:10:00Z"
  }
]

Pins are LLM-estimated and clamped (lat∈[-90,90], lon∈[-180,180]). No Nominatim. No writes into events.

GET /api/news/models — Settings dropdown catalog

Never 502s. { "source": "live"|"fallback", "models": [{"id": "…"}] }.

GET /api/settings · PUT /api/settings

{ "summary_model": "Hermes-4.3-36B", "nous_base_url": "https://inference-api.nousresearch.com/v1" }

PUT body is { "summary_model": "<1–128 char id>" }. nous_base_url is ignored even if sent.

Reduce JSON contract

Reduce phase (response_format: json_object, English only) must be a single object. Parser (intel.parse_reduce_json) strips <think>…</think> and markdown json fences, then brace-slices:

{
  "summary_en": "English markdown brief or the no-qualifying-events sentence",
  "ticker": [
    {"headline": "", "importance": "critical", "url": "", "location_name": ""}
  ],
  "map_items": [
    {
      "headline": "",
      "importance": "critical",
      "location_name": "",
      "lat": 0,
      "lon": 0,
      "location_confidence": "city",
      "category": "military/conflict",
      "url": ""
    }
  ]
}

Persist ticker/map only for importance in critical/high. Map rows also need valid coords; Unknown / invented places are dropped. Caps: 12 ticker (≤140 chars, no markdown), 20 map. Empty ticker is allowed. summary_en lands in article_summaries.summary_text.

Configuration (all via env / .env)

Var	Default	Notes
NOUS_API_KEY	(blank)	Required for summaries. Prefer Keys UI; env overrides. Unset in both env and api_keys = summarizer logs and idles (never crashes); APIs return [].
NOUS_BASE_URL	https://inference-api.nousresearch.com/v1	Read-only in Settings.
SUMMARY_MODEL	Hermes-4.3-36B	Compose default. Operator-facing choice is Settings → app_settings.SUMMARY_MODEL.
NEWS_BATCH_SIZE	50	Articles per map-phase batch (compose maps to container BATCH_SIZE).
SUMMARY_WINDOW_HOURS	1	How far back the summarizer looks for new articles.
INCLUDE_FUTURES	0	Legacy futures-prices coupling (upstream pipeline). OFF for OSINT; set 1 + install yfinance to enable.
NEWS_SCRAPE_MINUTE	0	Wall-clock minute the scraper fires.
NEWS_SUMMARIZE_MINUTE	5	Wall-clock minute the summarizer fires.
NEWS_SCRAPE_RUN_ON_START	1	Run one scrape immediately on container start.
NEWS_SUMMARIZE_RUN_ON_START	1	Run one summarize immediately on container start.
NEWS_SUMMARIZE_FORCE	0	1 ignores the current-UTC-hour idempotency skip (double-pins on recreate).
NEWS_LOG_LEVEL	INFO	Scrapy log level.
OSINT_USER_AGENT	osint-dashboard-news-summarizer	Sent on every outbound Nous call.
TELEGRAM_TOKEN / TELEGRAM_CHAT_ID	(blank)	Reserved for the (out-of-scope) Telegram delivery bot.

DB_* for both services is mapped to the shared osint-db credentials (DB_HOST=db, same DB_USER/DB_PASSWORD/DB_NAME as the rest of the stack).

Nous chat: POST {NOUS_BASE_URL}/chat/completions via news/summerizer/nous_client.py (httpx, no openai SDK). Auth is a Bearer token from NOUS_API_KEY. No Hermes-4 reasoning system prompt. Reduce uses json_mode=True.

Prompts

Map/reduce prompts are the k8s deepseek-configmap text in news/summerizer/prompt_files/{map,summary}.txt — not hardcoded in Python. Resolution order: MAP_PROMPT / SUMMARY_PROMPT env (non-blank wins) → MAP_PROMPT_FILE / SUMMARY_PROMPT_FILE → bundled files.

Compose passes ${MAP_PROMPT:-} so a host .env swap takes effect on container recreate (no image rebuild). Blank env is treated as unset.

The reduce JSON is the market-intel schema (market_overview + geopolitical_osint). intel.parse_reduce_json maps it onto the dashboard summary_en / ticker / map_items contract. Additive lat/lon on critical_events and active_conflicts feed Critical News pins.

Upstream's futures/markets tape is gated behind INCLUDE_FUTURES=1.

Tests

PYTHONPATH=news/summerizer pytest news/summerizer/tests -v
# intel + nous_client tests PASS (no network)

PYTHONPATH=app pytest tests/test_api_news.py \
  tests/test_api_settings.py tests/test_api_live_layers.py -v
# DB-marked tests skip without Postgres; live_layers must still PASS
# /api/map/layers overlays key set UNCHANGED

Live verification

After deploy / compose rebuild of news-summarizer on the Pi:

1. Keys UI: save NOUS_API_KEY → status ****last4.
2. Settings: pick a model → Save → GET /api/settings echoes it.
3. docker compose --profile ingest logs -f news-summarizer — next run (or NEWS_SUMMARIZE_RUN_ON_START=1 recreate) logs Processing N articles with <model>.
4. curl -s localhost:8000/api/news/summaries?limit=1 — English summary_text, model set.
5. curl -s localhost:8000/api/news/ticker — flagged headlines only.
6. curl -s localhost:8000/api/news/map — only rows with lat/lon.
7. HUD: NEWS ticker scrolls flagged items; map overlay pins popup with location.
8. Unset key + empty keystore → summarizer logs idle, APIs return [], no crash.

Operator action after merge: paste a Nous Portal API key in API Keys; pick a model in Settings if the default Hermes-4.3-36B is not wanted; rebuild osint-news-summarizer on the Pi (pi-app-deploy / compose).