diff --git a/docs/news.md b/docs/news.md index 75afc25..93084d0 100644 --- a/docs/news.md +++ b/docs/news.md @@ -1,10 +1,11 @@ -# News pipeline — scraper + summarizer +# News pipeline — scraper + Nous Portal summarizer The OSINT dashboard ingests ~257 global news RSS sources hourly and produces -LLM master summaries. Both services were vendored from the upstream -`~/Projects/newsPipeline` project and re-integrated here to replace the old -k8s CronJob choreography with in-compose scheduling against the EXISTING -osint-db — **no second Postgres**. +an English LLM brief plus flagged ticker/map rows. Both services were vendored +from the upstream `~/Projects/newsPipeline` project and re-integrated here to +replace the old k8s CronJob choreography with in-compose scheduling against +the EXISTING osint-db — **no second Postgres**. The LLM is **Nous Portal** +(`inference-api.nousresearch.com`) — not Gemini. ## Architecture @@ -15,10 +16,11 @@ osint-db — **no second Postgres**. news-scraper (Scrapy, hourly :00) ──► articles table (osint-db) │ │ │ ▼ -news-summarizer (Gemini map-reduce, hourly :05) ──► article_summaries table +news-summarizer (Nous Portal map-reduce, :05) ──► article_summaries + news_items │ ▼ - GET /api/news · GET /api/news/summaries + GET /api/news · /api/news/summaries · /api/news/ticker · /api/news/map + GET /api/news/models · GET/PUT /api/settings ``` | Component | Image | Container | Scheduling | @@ -29,6 +31,11 @@ news-summarizer (Gemini map-reduce, hourly :05) ──► article_summaries tabl Both services live under the `ingest` compose profile (same as the ingester and camera-scraper): `docker compose --profile ingest up -d`. +The summarizer is a batch sidecar, **not** a live overlay. Do **not** reuse +`GET /api/alerts` (dashboard entity/keyword alerts). Do **not** stuff news +into `overlay_catalog()` — `/api/map/layers` `overlays` stays live upstream +feeds (`GET /api/news` exact key set is unchanged on purpose). + ## Data flow 1. **Scraper** — `news/scraper/run_news_scraper.py` runs @@ -39,29 +46,52 @@ and camera-scraper): `docker compose --profile ingest up -d`. (`ON CONFLICT (url) DO NOTHING`). 2. **Summarizer** — `news/summerizer/run_news_summarizer.py` runs `summarizer.py` at :05 past each hour. It reads articles from the last - `SUMMARY_WINDOW_HOURS`, map-reduces them through Gemini - (`SUMMARY_MODEL`, default `gemini-2.0-flash`), and inserts one master - summary into `article_summaries`. + `SUMMARY_WINDOW_HOURS`, map-reduces them through Nous Portal + (`SUMMARY_MODEL` / Settings, default `Hermes-4.3-36B`), writes the English + brief to `article_summaries` (column `model` is the LLM id), and flagged + ticker/map rows to `news_items`. Scheduling is done with small in-compose wall-clock loops (not host cron): each loop runs once on boot (`*_RUN_ON_START=1`, seeds data fast) then sleeps until the next scheduled minute. The loop is serial, so a run that overruns its slot simply shifts to the next boundary — two crawls/summaries never overlap. +Hour-truncation idempotency: if `article_summaries` already has a row for the +current UTC hour, the summarizer **skips** (prevents double-pins on +`RUN_ON_START` recreate). Set `NEWS_SUMMARIZE_FORCE=1` to ignore that skip. + The `articles` and `article_summaries` tables are created by the idempotent alembic migration `003_news` (also created by the scraper's own -`CREATE TABLE IF NOT EXISTS`, so container startup order doesn't matter). +`CREATE TABLE IF NOT EXISTS`). `news_items` is alembic `005_news_items`. +Container startup order doesn't matter. + +## Keys and Settings + +- **`NOUS_API_KEY`** — paste in the dashboard **Keys** UI (`api_keys` / + `keystore.KEY_REGISTRY`). Env / `.env` is an **override** (env wins, same + as FIRMS). Never returned by any API; never emitted into `index.html`; + never proxied from the browser. +- **Idle without a key** — if env is unset **and** the keystore row is empty, + the summarizer logs and idles (never crashes). News intel APIs return `[]`. +- **Model** — non-secret. Settings UI model selector `PUT /api/settings` + `{ "summary_model": "…" }` stores `SUMMARY_MODEL` in `app_settings` (1–128 + chars). `GET /api/settings` echoes `{summary_model, nous_base_url}`. + `nous_base_url` is read-only. Default `Hermes-4.3-36B`. Live catalog is + best-effort `GET /api/news/models`. ## Endpoints ### GET /api/news — recent articles +Key set **unchanged** (no `lat`/`lon` on articles; geo lives on `/api/news/map`). + | Query param | Meaning | Default | |---|---|---| | `domain` | filter by source domain (e.g. `www.reuters.com`) | none | | `since` | only articles captured at/after this UTC instant (ISO-8601) | none | | `limit` | max rows | `50` (max `500`) | | `offset` | pagination offset | `0` | +| `include_content` | include full article body | `false` | ```json [ @@ -69,14 +99,14 @@ alembic migration `003_news` (also created by the scraper's own "id": 1, "title": "…", "url": "https://…", - "content": "full extracted article text…", + "content": null, "domain": "www.reuters.com", "timestamp": "2026-08-24T18:10:00Z" } ] ``` -### GET /api/news/summaries — master LLM summaries +### GET /api/news/summaries — master LLM briefs | Query param | Meaning | Default | |---|---|---| @@ -88,52 +118,178 @@ alembic migration `003_news` (also created by the scraper's own [ { "id": 1, - "summary_text": "master LLM summary (markdown)…", - "batch_timestamp": "2026-08-24T18:10:00Z" + "summary_text": "English markdown brief…", + "batch_timestamp": "2026-08-24T18:10:00Z", + "model": "Hermes-4.3-36B" } ] ``` +`model` is additive. Empty DB → `[]` (no crash). + +### GET /api/news/ticker — flagged HUD headlines + +Critical/high `news_items` with `kind=ticker` only. Do **not** reuse +`GET /api/alerts`. Bottom HUD `#nt-track` scrolls these rows, not a dump of +the whole brief. + +| Query param | Meaning | Default | +|---|---|---| +| `since` | only items created at/after this UTC instant | none | +| `limit` | max rows | `20` (max `50`) | + +```json +[ + { + "id": 1, + "headline": "…", + "importance": "critical", + "location_name": "Kyiv", + "url": "https://…", + "created_at": "2026-08-24T18:10:00Z" + } +] +``` + +### GET /api/news/map — geolocated critical/high pins + +Only rows with valid `lat`/`lon`. Optional bbox. **No zoom skip** — world +view is the point. Layer-panel toggle uses this dedicated path (same as +event blips), not `overlay_catalog`. + +| Query param | Meaning | Default | +|---|---|---| +| `bbox` | `minlon,minlat,maxlon,maxlat` | all flagged pins | +| `since` | only items created at/after this UTC instant | last 24 hours | +| `limit` | max rows | `200` (max `500`) | + +Malformed bbox → `422`. + +```json +[ + { + "id": 1, + "headline": "…", + "importance": "high", + "location_name": "Kyiv", + "lat": 50.45, + "lon": 30.52, + "location_confidence": "city", + "category": "military/conflict", + "url": "https://…", + "created_at": "2026-08-24T18:10:00Z" + } +] +``` + +Pins are LLM-estimated and clamped (`lat∈[-90,90]`, `lon∈[-180,180]`). No +Nominatim. No writes into `events`. + +### GET /api/news/models — Settings dropdown catalog + +Never 502s. `{ "source": "live"|"fallback", "models": [{"id": "…"}] }`. + +### GET /api/settings · PUT /api/settings + +```json +{ "summary_model": "Hermes-4.3-36B", "nous_base_url": "https://inference-api.nousresearch.com/v1" } +``` + +PUT body is `{ "summary_model": "<1–128 char id>" }`. `nous_base_url` is +ignored even if sent. + +## Reduce JSON contract + +Reduce phase (`response_format: json_object`, English only) must be a single +object. Parser (`intel.parse_reduce_json`) strips `` and +markdown json fences, then brace-slices: + +```json +{ + "summary_en": "English markdown brief or the no-qualifying-events sentence", + "ticker": [ + {"headline": "", "importance": "critical", "url": "", "location_name": ""} + ], + "map_items": [ + { + "headline": "", + "importance": "critical", + "location_name": "", + "lat": 0, + "lon": 0, + "location_confidence": "city", + "category": "military/conflict", + "url": "" + } + ] +} +``` + +Persist ticker/map only for `importance` in `critical`/`high`. Map rows also +need valid coords; Unknown / invented places are dropped. Caps: 12 ticker +(≤140 chars, no markdown), 20 map. Empty ticker is allowed. `summary_en` +lands in `article_summaries.summary_text`. + ## Configuration (all via env / `.env`) | Var | Default | Notes | |---|---|---| -| `GEMINI_API_KEY` | *(blank)* | **Required for summaries.** Unset = summarizer logs and idles (never crashes). | -| `SUMMARY_MODEL` | `gemini-2.0-flash` | Gemini model id. | -| `NEWS_BATCH_SIZE` | `50` | Articles per map-phase batch. | +| `NOUS_API_KEY` | *(blank)* | **Required for summaries.** Prefer Keys UI; env overrides. Unset in **both** env and `api_keys` = summarizer logs and idles (never crashes); APIs return `[]`. | +| `NOUS_BASE_URL` | `https://inference-api.nousresearch.com/v1` | Read-only in Settings. | +| `SUMMARY_MODEL` | `Hermes-4.3-36B` | Compose default. Operator-facing choice is Settings → `app_settings.SUMMARY_MODEL`. | +| `NEWS_BATCH_SIZE` | `50` | Articles per map-phase batch (compose maps to container `BATCH_SIZE`). | | `SUMMARY_WINDOW_HOURS` | `1` | How far back the summarizer looks for new articles. | | `INCLUDE_FUTURES` | `0` | Legacy futures-prices coupling (upstream pipeline). OFF for OSINT; set `1` + install `yfinance` to enable. | | `NEWS_SCRAPE_MINUTE` | `0` | Wall-clock minute the scraper fires. | | `NEWS_SUMMARIZE_MINUTE` | `5` | Wall-clock minute the summarizer fires. | | `NEWS_SCRAPE_RUN_ON_START` | `1` | Run one scrape immediately on container start. | | `NEWS_SUMMARIZE_RUN_ON_START` | `1` | Run one summarize immediately on container start. | +| `NEWS_SUMMARIZE_FORCE` | `0` | `1` ignores the current-UTC-hour idempotency skip (double-pins on recreate). | | `NEWS_LOG_LEVEL` | `INFO` | Scrapy log level. | +| `OSINT_USER_AGENT` | `osint-dashboard-news-summarizer` | Sent on every outbound Nous call. | | `TELEGRAM_TOKEN` / `TELEGRAM_CHAT_ID` | *(blank)* | Reserved for the (out-of-scope) Telegram delivery bot. | DB_* for both services is mapped to the shared osint-db credentials (`DB_HOST=db`, same `DB_USER/DB_PASSWORD/DB_NAME` as the rest of the stack). +Nous chat: `POST {NOUS_BASE_URL}/chat/completions` via `news/summerizer/nous_client.py` +(`httpx`, no `openai` SDK). Auth is a Bearer token from `NOUS_API_KEY`. +No Hermes-4 reasoning system prompt. Reduce uses `json_mode=True`. + ## Prompts Both prompts are env-overridable — the default `MAP_PROMPT` is OSINT-neutral -(facts, locations, entities, category, OSINT signal per article) and the default -`SUMMARY_PROMPT` produces a concise executive summary of the most impactful -items (with a "no qualifying events" escape hatch). Upstream's futures/markets -prompt language is gated behind `INCLUDE_FUTURES=1`. +(facts, locations, entities, category, OSINT signal per article; English) and +the default `SUMMARY_PROMPT` demands the reduce JSON above (with a +"no qualifying events" escape hatch). Upstream's futures/markets prompt +language is gated behind `INCLUDE_FUTURES=1`. ## Tests -`tests/test_api_news.py` — DB-backed API contract tests (auto-skip without a -reachable test database, same as the FIRMS tests): - ```bash -DB_HOST=... DB_PORT=... DB_USER=osint DB_PASSWORD=... DB_NAME=osint_data \ - pytest tests/test_api_news.py -v +PYTHONPATH=news/summerizer pytest news/summerizer/tests -v +# intel + nous_client tests PASS (no network) + +PYTHONPATH=app pytest tests/test_api_news.py tests/test_api_news_intel.py \ + tests/test_api_settings.py tests/test_api_live_layers.py -v +# DB-marked tests skip without Postgres; live_layers must still PASS +# /api/map/layers overlays key set UNCHANGED ``` ## Live verification -End-to-end (real crawl → DB → API) is verified after deploy on the Pi: check -`docker compose --profile ingest logs -f news-scraper news-summarizer`, then -`curl -s localhost:8000/api/news | head`. Summaries additionally require -`GEMINI_API_KEY` to be set in `.env` on the Pi. +After deploy / compose rebuild of `news-summarizer` on the Pi: + +1. Keys UI: save `NOUS_API_KEY` → status `****last4`. +2. Settings: pick a model → Save → `GET /api/settings` echoes it. +3. `docker compose --profile ingest logs -f news-summarizer` — next run (or + `NEWS_SUMMARIZE_RUN_ON_START=1` recreate) logs `Processing N articles with `. +4. `curl -s localhost:8000/api/news/summaries?limit=1` — English `summary_text`, `model` set. +5. `curl -s localhost:8000/api/news/ticker` — flagged headlines only. +6. `curl -s localhost:8000/api/news/map` — only rows with lat/lon. +7. HUD: NEWS ticker scrolls flagged items; map overlay pins popup with location. +8. Unset key + empty keystore → summarizer logs idle, APIs return `[]`, no crash. + +**Operator action after merge:** paste a Nous Portal API key in API Keys; pick +a model in Settings if the default `Hermes-4.3-36B` is not wanted; rebuild +`osint-news-summarizer` on the Pi (`pi-app-deploy` / compose).