295 lines
12 KiB
Markdown
295 lines
12 KiB
Markdown
# News pipeline — scraper + Nous Portal summarizer
|
||
|
||
The OSINT dashboard ingests ~257 global news RSS sources hourly and produces
|
||
an English LLM brief plus flagged ticker/map rows. Both services were vendored
|
||
from the upstream `~/Projects/newsPipeline` project and re-integrated here to
|
||
replace the old k8s CronJob choreography with in-compose scheduling against
|
||
the EXISTING osint-db — **no second Postgres**. The LLM is **Nous Portal**
|
||
(`inference-api.nousresearch.com`) — not Gemini.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
257 RSS feeds (news/scraper/urls.txt)
|
||
│
|
||
▼
|
||
news-scraper (Scrapy, hourly :00) ──► articles table (osint-db)
|
||
│ │
|
||
│ ▼
|
||
news-summarizer (Nous Portal map-reduce, :05) ──► article_summaries + news_items
|
||
│
|
||
▼
|
||
GET /api/news · /api/news/summaries · /api/news/ticker · /api/news/map
|
||
GET /api/news/models · GET/PUT /api/settings
|
||
```
|
||
|
||
| Component | Image | Container | Scheduling |
|
||
|---|---|---|---|
|
||
| Scraper | `localhost/osint-news-scraper` | `osint-news-scraper` | wall-clock loop, minute `NEWS_SCRAPE_MINUTE` (default :00) |
|
||
| Summarizer | `localhost/osint-news-summarizer` | `osint-news-summarizer` | wall-clock loop, minute `NEWS_SUMMARIZE_MINUTE` (default :05) |
|
||
|
||
Both services live under the `ingest` compose profile (same as the ingester
|
||
and camera-scraper): `docker compose --profile ingest up -d`.
|
||
|
||
The summarizer is a batch sidecar, **not** a live overlay. Do **not** reuse
|
||
`GET /api/alerts` (dashboard entity/keyword alerts). Do **not** stuff news
|
||
into `overlay_catalog()` — `/api/map/layers` `overlays` stays live upstream
|
||
feeds (`GET /api/news` exact key set is unchanged on purpose).
|
||
|
||
## Data flow
|
||
|
||
1. **Scraper** — `news/scraper/run_news_scraper.py` runs
|
||
`scrapy crawl articles` (spider `news/scraper/newsScraper/spiders/news_spider.py`)
|
||
at the top of each hour. The spider reads the RSS feed URLs from `urls.txt`,
|
||
follows each `<item>` link, extracts the main article body, and the
|
||
`PostgresPipeline` writes to `articles` with URL-based dedup
|
||
(`ON CONFLICT (url) DO NOTHING`).
|
||
2. **Summarizer** — `news/summerizer/run_news_summarizer.py` runs
|
||
`summarizer.py` at :05 past each hour. It reads articles from the last
|
||
`SUMMARY_WINDOW_HOURS`, map-reduces them through Nous Portal
|
||
(`SUMMARY_MODEL` / Settings, default `Hermes-4.3-36B`), writes the English
|
||
brief to `article_summaries` (column `model` is the LLM id), and flagged
|
||
ticker/map rows to `news_items`.
|
||
|
||
Scheduling is done with small in-compose wall-clock loops (not host cron): each
|
||
loop runs once on boot (`*_RUN_ON_START=1`, seeds data fast) then sleeps until
|
||
the next scheduled minute. The loop is serial, so a run that overruns its slot
|
||
simply shifts to the next boundary — two crawls/summaries never overlap.
|
||
|
||
Hour-truncation idempotency: if `article_summaries` already has a row for the
|
||
current UTC hour, the summarizer **skips** (prevents double-pins on
|
||
`RUN_ON_START` recreate). Set `NEWS_SUMMARIZE_FORCE=1` to ignore that skip.
|
||
|
||
The `articles` and `article_summaries` tables are created by the idempotent
|
||
alembic migration `003_news` (also created by the scraper's own
|
||
`CREATE TABLE IF NOT EXISTS`). `news_items` is alembic `005_news_items`.
|
||
Container startup order doesn't matter.
|
||
|
||
## Keys and Settings
|
||
|
||
- **`NOUS_API_KEY`** — paste in the dashboard **Keys** UI (`api_keys` /
|
||
`keystore.KEY_REGISTRY`). Env / `.env` is an **override** (env wins, same
|
||
as FIRMS). Never returned by any API; never emitted into `index.html`;
|
||
never proxied from the browser.
|
||
- **Idle without a key** — if env is unset **and** the keystore row is empty,
|
||
the summarizer logs and idles (never crashes). News intel APIs return `[]`.
|
||
- **Model** — non-secret. Settings UI model selector `PUT /api/settings`
|
||
`{ "summary_model": "…" }` stores `SUMMARY_MODEL` in `app_settings` (1–128
|
||
chars). `GET /api/settings` echoes `{summary_model, nous_base_url}`.
|
||
`nous_base_url` is read-only. Default `Hermes-4.3-36B`. Live catalog is
|
||
best-effort `GET /api/news/models`.
|
||
|
||
## Endpoints
|
||
|
||
### GET /api/news — recent articles
|
||
|
||
Key set **unchanged** (no `lat`/`lon` on articles; geo lives on `/api/news/map`).
|
||
|
||
| Query param | Meaning | Default |
|
||
|---|---|---|
|
||
| `domain` | filter by source domain (e.g. `www.reuters.com`) | none |
|
||
| `since` | only articles captured at/after this UTC instant (ISO-8601) | none |
|
||
| `limit` | max rows | `50` (max `500`) |
|
||
| `offset` | pagination offset | `0` |
|
||
| `include_content` | include full article body | `false` |
|
||
|
||
```json
|
||
[
|
||
{
|
||
"id": 1,
|
||
"title": "…",
|
||
"url": "https://…",
|
||
"content": null,
|
||
"domain": "www.reuters.com",
|
||
"timestamp": "2026-08-24T18:10:00Z"
|
||
}
|
||
]
|
||
```
|
||
|
||
### GET /api/news/summaries — master LLM briefs
|
||
|
||
| Query param | Meaning | Default |
|
||
|---|---|---|
|
||
| `since` | only summaries generated at/after this UTC instant | none |
|
||
| `limit` | max rows | `20` (max `100`) |
|
||
| `offset` | pagination offset | `0` |
|
||
|
||
```json
|
||
[
|
||
{
|
||
"id": 1,
|
||
"summary_text": "English markdown brief…",
|
||
"batch_timestamp": "2026-08-24T18:10:00Z",
|
||
"model": "Hermes-4.3-36B"
|
||
}
|
||
]
|
||
```
|
||
|
||
`model` is additive. Empty DB → `[]` (no crash).
|
||
|
||
### GET /api/news/ticker — flagged HUD headlines
|
||
|
||
Critical/high `news_items` with `kind=ticker` only. Do **not** reuse
|
||
`GET /api/alerts`. Bottom HUD `#nt-track` scrolls these rows, not a dump of
|
||
the whole brief.
|
||
|
||
| Query param | Meaning | Default |
|
||
|---|---|---|
|
||
| `since` | only items created at/after this UTC instant | none |
|
||
| `limit` | max rows | `20` (max `50`) |
|
||
|
||
```json
|
||
[
|
||
{
|
||
"id": 1,
|
||
"headline": "…",
|
||
"importance": "critical",
|
||
"location_name": "Kyiv",
|
||
"url": "https://…",
|
||
"created_at": "2026-08-24T18:10:00Z"
|
||
}
|
||
]
|
||
```
|
||
|
||
### GET /api/news/map — geolocated critical/high pins
|
||
|
||
Only rows with valid `lat`/`lon`. Optional bbox. **No zoom skip** — world
|
||
view is the point. Layer-panel toggle uses this dedicated path (same as
|
||
event blips), not `overlay_catalog`.
|
||
|
||
| Query param | Meaning | Default |
|
||
|---|---|---|
|
||
| `bbox` | `minlon,minlat,maxlon,maxlat` | all flagged pins |
|
||
| `since` | only items created at/after this UTC instant | last 24 hours |
|
||
| `limit` | max rows | `200` (max `500`) |
|
||
|
||
Malformed bbox → `422`.
|
||
|
||
```json
|
||
[
|
||
{
|
||
"id": 1,
|
||
"headline": "…",
|
||
"importance": "high",
|
||
"location_name": "Kyiv",
|
||
"lat": 50.45,
|
||
"lon": 30.52,
|
||
"location_confidence": "city",
|
||
"category": "military/conflict",
|
||
"url": "https://…",
|
||
"created_at": "2026-08-24T18:10:00Z"
|
||
}
|
||
]
|
||
```
|
||
|
||
Pins are LLM-estimated and clamped (`lat∈[-90,90]`, `lon∈[-180,180]`). No
|
||
Nominatim. No writes into `events`.
|
||
|
||
### GET /api/news/models — Settings dropdown catalog
|
||
|
||
Never 502s. `{ "source": "live"|"fallback", "models": [{"id": "…"}] }`.
|
||
|
||
### GET /api/settings · PUT /api/settings
|
||
|
||
```json
|
||
{ "summary_model": "Hermes-4.3-36B", "nous_base_url": "https://inference-api.nousresearch.com/v1" }
|
||
```
|
||
|
||
PUT body is `{ "summary_model": "<1–128 char id>" }`. `nous_base_url` is
|
||
ignored even if sent.
|
||
|
||
## Reduce JSON contract
|
||
|
||
Reduce phase (`response_format: json_object`, English only) must be a single
|
||
object. Parser (`intel.parse_reduce_json`) strips `<think>…</think>` and
|
||
markdown json fences, then brace-slices:
|
||
|
||
```json
|
||
{
|
||
"summary_en": "English markdown brief or the no-qualifying-events sentence",
|
||
"ticker": [
|
||
{"headline": "", "importance": "critical", "url": "", "location_name": ""}
|
||
],
|
||
"map_items": [
|
||
{
|
||
"headline": "",
|
||
"importance": "critical",
|
||
"location_name": "",
|
||
"lat": 0,
|
||
"lon": 0,
|
||
"location_confidence": "city",
|
||
"category": "military/conflict",
|
||
"url": ""
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
Persist ticker/map only for `importance` in `critical`/`high`. Map rows also
|
||
need valid coords; Unknown / invented places are dropped. Caps: 12 ticker
|
||
(≤140 chars, no markdown), 20 map. Empty ticker is allowed. `summary_en`
|
||
lands in `article_summaries.summary_text`.
|
||
|
||
## Configuration (all via env / `.env`)
|
||
|
||
| Var | Default | Notes |
|
||
|---|---|---|
|
||
| `NOUS_API_KEY` | *(blank)* | **Required for summaries.** Prefer Keys UI; env overrides. Unset in **both** env and `api_keys` = summarizer logs and idles (never crashes); APIs return `[]`. |
|
||
| `NOUS_BASE_URL` | `https://inference-api.nousresearch.com/v1` | Read-only in Settings. |
|
||
| `SUMMARY_MODEL` | `Hermes-4.3-36B` | Compose default. Operator-facing choice is Settings → `app_settings.SUMMARY_MODEL`. |
|
||
| `NEWS_BATCH_SIZE` | `50` | Articles per map-phase batch (compose maps to container `BATCH_SIZE`). |
|
||
| `SUMMARY_WINDOW_HOURS` | `1` | How far back the summarizer looks for new articles. |
|
||
| `INCLUDE_FUTURES` | `0` | Legacy futures-prices coupling (upstream pipeline). OFF for OSINT; set `1` + install `yfinance` to enable. |
|
||
| `NEWS_SCRAPE_MINUTE` | `0` | Wall-clock minute the scraper fires. |
|
||
| `NEWS_SUMMARIZE_MINUTE` | `5` | Wall-clock minute the summarizer fires. |
|
||
| `NEWS_SCRAPE_RUN_ON_START` | `1` | Run one scrape immediately on container start. |
|
||
| `NEWS_SUMMARIZE_RUN_ON_START` | `1` | Run one summarize immediately on container start. |
|
||
| `NEWS_SUMMARIZE_FORCE` | `0` | `1` ignores the current-UTC-hour idempotency skip (double-pins on recreate). |
|
||
| `NEWS_LOG_LEVEL` | `INFO` | Scrapy log level. |
|
||
| `OSINT_USER_AGENT` | `osint-dashboard-news-summarizer` | Sent on every outbound Nous call. |
|
||
| `TELEGRAM_TOKEN` / `TELEGRAM_CHAT_ID` | *(blank)* | Reserved for the (out-of-scope) Telegram delivery bot. |
|
||
|
||
DB_* for both services is mapped to the shared osint-db credentials
|
||
(`DB_HOST=db`, same `DB_USER/DB_PASSWORD/DB_NAME` as the rest of the stack).
|
||
|
||
Nous chat: `POST {NOUS_BASE_URL}/chat/completions` via `news/summerizer/nous_client.py`
|
||
(`httpx`, no `openai` SDK). Auth is a Bearer token from `NOUS_API_KEY`.
|
||
No Hermes-4 reasoning system prompt. Reduce uses `json_mode=True`.
|
||
|
||
## Prompts
|
||
|
||
Both prompts are env-overridable — the default `MAP_PROMPT` is OSINT-neutral
|
||
(facts, locations, entities, category, OSINT signal per article; English) and
|
||
the default `SUMMARY_PROMPT` demands the reduce JSON above (with a
|
||
"no qualifying events" escape hatch). Upstream's futures/markets prompt
|
||
language is gated behind `INCLUDE_FUTURES=1`.
|
||
|
||
## Tests
|
||
|
||
```bash
|
||
PYTHONPATH=news/summerizer pytest news/summerizer/tests -v
|
||
# intel + nous_client tests PASS (no network)
|
||
|
||
PYTHONPATH=app pytest tests/test_api_news.py \
|
||
tests/test_api_settings.py tests/test_api_live_layers.py -v
|
||
# DB-marked tests skip without Postgres; live_layers must still PASS
|
||
# /api/map/layers overlays key set UNCHANGED
|
||
```
|
||
|
||
## Live verification
|
||
|
||
After deploy / compose rebuild of `news-summarizer` on the Pi:
|
||
|
||
1. Keys UI: save `NOUS_API_KEY` → status `****last4`.
|
||
2. Settings: pick a model → Save → `GET /api/settings` echoes it.
|
||
3. `docker compose --profile ingest logs -f news-summarizer` — next run (or
|
||
`NEWS_SUMMARIZE_RUN_ON_START=1` recreate) logs `Processing N articles with <model>`.
|
||
4. `curl -s localhost:8000/api/news/summaries?limit=1` — English `summary_text`, `model` set.
|
||
5. `curl -s localhost:8000/api/news/ticker` — flagged headlines only.
|
||
6. `curl -s localhost:8000/api/news/map` — only rows with lat/lon.
|
||
7. HUD: NEWS ticker scrolls flagged items; map overlay pins popup with location.
|
||
8. Unset key + empty keystore → summarizer logs idle, APIs return `[]`, no crash.
|
||
|
||
**Operator action after merge:** paste a Nous Portal API key in API Keys; pick
|
||
a model in Settings if the default `Hermes-4.3-36B` is not wanted; rebuild
|
||
`osint-news-summarizer` on the Pi (`pi-app-deploy` / compose).
|