osint-dashboard/docs/news.md

News pipeline — scraper + Nous Portal summarizer

The OSINT dashboard ingests a large curated feed list (news/scraper/urls.txt) continuously and produces an English LLM brief plus flagged ticker/map rows every 15 minutes. Both services were vendored from the upstream ~/Projects/newsPipeline project and re-integrated here against the EXISTING osint-db — no second Postgres. The LLM is Nous Portal (inference-api.nousresearch.com) — not Gemini.

Architecture

urls.txt (RSS + homepages)
        │
        ▼
news-scraper   (Scrapy, continuous) ──► articles table (osint-db)
        │                                      │
        │                                      ▼
news-summarizer (Nous Portal, every 15m + 23:00 recap) ──► article_summaries + news_items
                                                      │
                                                      ▼
     GET /api/news · /api/news/summaries · /api/news/ticker · /api/news/map
     GET /api/news/models · GET/PUT /api/settings

Component	Image	Container	Scheduling
Scraper	localhost/osint-news-scraper	osint-news-scraper	loop, NEWS_SCRAPE_INTERVAL_S (default 10s after each crawl)
Summarizer	localhost/osint-news-summarizer	osint-news-summarizer	loop, NEWS_SUMMARIZE_INTERVAL_S (default 900s)

Both services live under the ingest compose profile (same as the ingester and camera-scraper): docker compose --profile ingest up -d.

The summarizer is a batch sidecar, not a live overlay. Do not reuse GET /api/alerts (dashboard entity/keyword alerts). Do not stuff news into overlay_catalog() — /api/map/layers overlays stays live upstream feeds (GET /api/news exact key set is unchanged on purpose).

Data flow

1. Scraper — news/scraper/run_news_scraper.py runs scrapy crawl articles back-to-back (default 10s pause). The spider reads URLs from urls.txt (homepages autodiscover RSS; feed URLs are parsed directly), follows each <item> link, extracts the main article body, and the PostgresPipeline writes to articles with URL-based dedup (ON CONFLICT (url) DO NOTHING).
2. Summarizer — news/summerizer/run_news_summarizer.py runs summarizer.py every NEWS_SUMMARIZE_INTERVAL_S (default 900) over the last SUMMARY_WINDOW_MINUTES (default 15), and again at 23:00 America/New_York (TZ) over the last 24 hours as a daily recap (kind=daily_recap). Both map-reduce through Nous Portal (SUMMARY_MODEL / Settings, default Hermes-4.3-36B), write the English brief to article_summaries (column model is the LLM id; kind is interval or daily_recap), and flagged ticker/map rows to news_items.

Loops are serial (two crawls/summaries never overlap). Interval idempotency: if article_summaries already has a row in the last interval, the summarizer skips (prevents double-pins on RUN_ON_START recreate). Set NEWS_SUMMARIZE_FORCE=1 to ignore that skip.

The articles and article_summaries tables are created by the idempotent alembic migration 003_news (also created by the scraper's own CREATE TABLE IF NOT EXISTS). news_items is alembic 005_news_items. Container startup order doesn't matter.

Keys and Settings

• NOUS_API_KEY — paste in the dashboard Keys UI (api_keys / keystore.KEY_REGISTRY). Env / .env is an override (env wins, same as FIRMS). Never returned by any API; never emitted into index.html; never proxied from the browser.
• Idle without a key — if env is unset and the keystore row is empty, the summarizer logs and idles (never crashes). News intel APIs return [].
• Model — non-secret. Settings UI model selector PUT /api/settings { "summary_model": "…" } stores SUMMARY_MODEL in app_settings (1–128 chars). GET /api/settings echoes {summary_model, nous_base_url}. nous_base_url is read-only. Default Hermes-4.3-36B. Live catalog is best-effort GET /api/news/models.

Endpoints

GET /api/news — recent articles

Key set unchanged (no lat/lon on articles; geo lives on /api/news/map).

Query param	Meaning	Default
domain	filter by source domain (e.g. www.reuters.com)	none
since	only articles captured at/after this UTC instant (ISO-8601)	none
limit	max rows	50 (max 500)
offset	pagination offset	0
include_content	include full article body	false

[
  {
    "id": 1,
    "title": "…",
    "url": "https://…",
    "content": null,
    "domain": "www.reuters.com",
    "timestamp": "2026-08-24T18:10:00Z"
  }
]

GET /api/news/summaries — master LLM briefs

Query param	Meaning	Default
since	only summaries generated at/after this UTC instant	none
limit	max rows	20 (max 100)
offset	pagination offset	0

[
  {
    "id": 1,
    "summary_text": "English markdown brief…",
    "batch_timestamp": "2026-08-24T18:10:00Z",
    "model": "Hermes-4.3-36B",
    "kind": "daily_recap"
  }
]

model and kind are additive (interval | daily_recap | null for old rows). ?kind=daily_recap pins the nightly 24h recap. Empty DB → [] (no crash). Malformed kind → 422.

GET /api/news/ticker — HUD headlines

Critical/high news_items with kind=ticker first. If none are flagged, medium/low ticker rows fill the tape so the dock is not blank. Do not reuse GET /api/alerts. Bottom HUD #nt-track scrolls these rows, not a dump of the whole brief.

Query param	Meaning	Default
since	only items created at/after this UTC instant	none
limit	max rows	20 (max 50)

[
  {
    "id": 1,
    "headline": "…",
    "importance": "critical",
    "location_name": "Kyiv",
    "url": "https://…",
    "created_at": "2026-08-24T18:10:00Z"
  }
]

GET /api/news/map — geolocated critical/high pins

Only rows with valid lat/lon. Optional bbox. No zoom skip — world view is the point. Layer-panel toggle uses this dedicated path (same as event blips), not overlay_catalog.

Query param	Meaning	Default
bbox	minlon,minlat,maxlon,maxlat	all flagged pins
since	only items created at/after this UTC instant	last 24 hours
limit	max rows	200 (max 500)

Malformed bbox → 422.

[
  {
    "id": 1,
    "headline": "…",
    "importance": "high",
    "location_name": "Kyiv",
    "lat": 50.45,
    "lon": 30.52,
    "location_confidence": "city",
    "category": "military/conflict",
    "url": "https://…",
    "created_at": "2026-08-24T18:10:00Z"
  }
]

Pins are LLM-estimated and clamped (lat∈[-90,90], lon∈[-180,180]). No Nominatim. No writes into events.

GET /api/news/models — Settings dropdown catalog

Never 502s. { "source": "live"|"fallback", "models": [{"id": "…"}] }.

GET /api/settings · PUT /api/settings

{ "summary_model": "Hermes-4.3-36B", "nous_base_url": "https://inference-api.nousresearch.com/v1" }

PUT body is { "summary_model": "<1–128 char id>" }. nous_base_url is ignored even if sent.

Reduce JSON contract

Reduce phase (response_format: json_object, English only) must be a single object. Parser (intel.parse_reduce_json) strips <think>…</think> and markdown json fences, then brace-slices:

{
  "summary_en": "English markdown brief or the no-qualifying-events sentence",
  "ticker": [
    {"headline": "", "importance": "critical", "url": "", "location_name": ""}
  ],
  "map_items": [
    {
      "headline": "",
      "importance": "critical",
      "location_name": "",
      "lat": 0,
      "lon": 0,
      "location_confidence": "city",
      "category": "military/conflict",
      "url": ""
    }
  ]
}

Persist ticker for critical/high first; if none, persist medium/low so the tape is not empty. Map rows stay critical/high with valid coords; Unknown / invented places are dropped. Caps: 12 ticker (≤140 chars, no markdown), 20 map. summary_en lands in article_summaries.summary_text.

Configuration (all via env / .env)

Var	Default	Notes
NOUS_API_KEY	(blank)	Required for summaries. Prefer Keys UI; env overrides. Unset in both env and api_keys = summarizer logs and idles (never crashes); APIs return [].
NOUS_BASE_URL	https://inference-api.nousresearch.com/v1	Read-only in Settings.
SUMMARY_MODEL	Hermes-4.3-36B	Compose default. Operator-facing choice is Settings → app_settings.SUMMARY_MODEL.
NEWS_BATCH_SIZE	50	Articles per map-phase batch (compose maps to container BATCH_SIZE).
SUMMARY_WINDOW_MINUTES	15	How far back the summarizer looks for new articles.
NEWS_SCRAPE_INTERVAL_S	10	Pause after each crawl before the next (scraper is otherwise continuous).
NEWS_SUMMARIZE_INTERVAL_S	900	Seconds between analyst runs (default 15 min).
TZ	America/New_York	Timezone for the 23:00 daily recap.
NEWS_RECAP_HOUR	23	Local hour of the daily 24h recap.
NEWS_RECAP_MINUTE	0	Local minute of the daily recap.
NEWS_SCRAPE_RUN_ON_START	1	Run one scrape immediately on container start.
NEWS_SUMMARIZE_RUN_ON_START	1	Run one summarize immediately on container start.
NEWS_SUMMARIZE_FORCE	0	1 ignores the interval/recap idempotency skip (double-pins on recreate).
INCLUDE_FUTURES	0	Legacy. Ignored — prompts never inject futures/market tape.
NEWS_LOG_LEVEL	INFO	Scrapy log level.
OSINT_USER_AGENT	osint-dashboard-news-summarizer	Sent on every outbound Nous call.
TELEGRAM_TOKEN / TELEGRAM_CHAT_ID	(blank)	Reserved for the (out-of-scope) Telegram delivery bot.

DB_* for both services is mapped to the shared osint-db credentials (DB_HOST=db, same DB_USER/DB_PASSWORD/DB_NAME as the rest of the stack).

Nous chat: POST {NOUS_BASE_URL}/chat/completions via news/summerizer/nous_client.py (httpx, no openai SDK). Auth is a Bearer token from NOUS_API_KEY. No Hermes-4 reasoning system prompt. Reduce uses json_mode=True.

Prompts

Both prompts are env-overridable. Defaults recap the articles actually provided, ranked by breaking important news, and ignore futures / commodity tape. ticker/map may be empty; summary_en must still be a real brief. RECAP_PROMPT (23:00, 24h window) is the daily recap; SUMMARY_PROMPT is the 15-min analyst. INCLUDE_FUTURES is ignored.

Tests

PYTHONPATH=news/summerizer pytest news/summerizer/tests -v
# intel + nous_client tests PASS (no network)

PYTHONPATH=app pytest tests/test_api_news.py \
  tests/test_api_settings.py tests/test_api_live_layers.py -v
# DB-marked tests skip without Postgres; live_layers must still PASS
# /api/map/layers overlays key set UNCHANGED

Live verification

After deploy / compose rebuild of news-summarizer on the Pi:

1. Keys UI: save NOUS_API_KEY → status ****last4.
2. Settings: pick a model → Save → GET /api/settings echoes it.
3. docker compose --profile ingest logs -f news-summarizer — next run (or NEWS_SUMMARIZE_RUN_ON_START=1 recreate) logs Processing N articles with <model>.
4. curl -s localhost:8000/api/news/summaries?limit=1 — English summary_text, model set.
5. curl -s localhost:8000/api/news/ticker — flagged headlines only.
6. curl -s localhost:8000/api/news/map — only rows with lat/lon.
7. HUD: NEWS ticker scrolls flagged items; map overlay pins popup with location.
8. Unset key + empty keystore → summarizer logs idle, APIs return [], no crash.

Operator action after merge: paste a Nous Portal API key in API Keys; pick a model in Settings if the default Hermes-4.3-36B is not wanted; rebuild osint-news-summarizer on the Pi (pi-app-deploy / compose).