91f436390b
Some checks failed
build-and-deploy / build (push) Failing after 4s
Vendor the newsPipeline scraper + summarizer into the repo and wire them into
docker-compose against the EXISTING osint-db (no second Postgres), replacing
the upstream k8s CronJobs with in-compose wall-clock loops (:00 scrape, :05
summarize).
- news/scraper: vendored Scrapy project (257 RSS feeds) + hourly loop
scheduler (run_news_scraper.py)
- news/summerizer: vendored Gemini map-reduce summarizer, cleaned:
* fix broken google-genai response handling (_extract_text, defensive)
* fix malformed INSERT/GRANT query in save_summary_to_db
* OSINT-neutral default MAP_PROMPT; futures/markets language gated behind
INCLUDE_FUTURES=0 (yfinance lazy-imported)
* env-configurable model, batch size, lookback window
+ hourly loop scheduler (run_news_summarizer.py, :05)
- alembic 003_news: idempotent articles + article_summaries tables
- API: GET /api/news and GET /api/news/summaries (+ models, schemas)
- tests/test_api_news.py: 5 DB-backed contract tests (all pass vs real PG)
- docs/news.md + .env.example updates
Both services run under the `ingest` compose profile (matching the
ingester/camera-scraper pattern) and build arm64 on the Pi via the existing
Forgejo CI workflow. telebot left out of scope (reserved env only).
139 lines
5.6 KiB
Markdown
139 lines
5.6 KiB
Markdown
# News pipeline — scraper + summarizer
|
|
|
|
The OSINT dashboard ingests ~257 global news RSS sources hourly and produces
|
|
LLM master summaries. Both services were vendored from the upstream
|
|
`~/Projects/newsPipeline` project and re-integrated here to replace the old
|
|
k8s CronJob choreography with in-compose scheduling against the EXISTING
|
|
osint-db — **no second Postgres**.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
257 RSS feeds (news/scraper/urls.txt)
|
|
│
|
|
▼
|
|
news-scraper (Scrapy, hourly :00) ──► articles table (osint-db)
|
|
│ │
|
|
│ ▼
|
|
news-summarizer (Gemini map-reduce, hourly :05) ──► article_summaries table
|
|
│
|
|
▼
|
|
GET /api/news · GET /api/news/summaries
|
|
```
|
|
|
|
| Component | Image | Container | Scheduling |
|
|
|---|---|---|---|
|
|
| Scraper | `localhost/osint-news-scraper` | `osint-news-scraper` | wall-clock loop, minute `NEWS_SCRAPE_MINUTE` (default :00) |
|
|
| Summarizer | `localhost/osint-news-summarizer` | `osint-news-summarizer` | wall-clock loop, minute `NEWS_SUMMARIZE_MINUTE` (default :05) |
|
|
|
|
Both services live under the `ingest` compose profile (same as the ingester
|
|
and camera-scraper): `docker compose --profile ingest up -d`.
|
|
|
|
## Data flow
|
|
|
|
1. **Scraper** — `news/scraper/run_news_scraper.py` runs
|
|
`scrapy crawl articles` (spider `news/scraper/newsScraper/spiders/news_spider.py`)
|
|
at the top of each hour. The spider reads the RSS feed URLs from `urls.txt`,
|
|
follows each `<item>` link, extracts the main article body, and the
|
|
`PostgresPipeline` writes to `articles` with URL-based dedup
|
|
(`ON CONFLICT (url) DO NOTHING`).
|
|
2. **Summarizer** — `news/summerizer/run_news_summarizer.py` runs
|
|
`summarizer.py` at :05 past each hour. It reads articles from the last
|
|
`SUMMARY_WINDOW_HOURS`, map-reduces them through Gemini
|
|
(`SUMMARY_MODEL`, default `gemini-2.0-flash`), and inserts one master
|
|
summary into `article_summaries`.
|
|
|
|
Scheduling is done with small in-compose wall-clock loops (not host cron): each
|
|
loop runs once on boot (`*_RUN_ON_START=1`, seeds data fast) then sleeps until
|
|
the next scheduled minute. The loop is serial, so a run that overruns its slot
|
|
simply shifts to the next boundary — two crawls/summaries never overlap.
|
|
|
|
The `articles` and `article_summaries` tables are created by the idempotent
|
|
alembic migration `003_news` (also created by the scraper's own
|
|
`CREATE TABLE IF NOT EXISTS`, so container startup order doesn't matter).
|
|
|
|
## Endpoints
|
|
|
|
### GET /api/news — recent articles
|
|
|
|
| Query param | Meaning | Default |
|
|
|---|---|---|
|
|
| `domain` | filter by source domain (e.g. `www.reuters.com`) | none |
|
|
| `since` | only articles captured at/after this UTC instant (ISO-8601) | none |
|
|
| `limit` | max rows | `50` (max `500`) |
|
|
| `offset` | pagination offset | `0` |
|
|
|
|
```json
|
|
[
|
|
{
|
|
"id": 1,
|
|
"title": "…",
|
|
"url": "https://…",
|
|
"content": "full extracted article text…",
|
|
"domain": "www.reuters.com",
|
|
"timestamp": "2026-08-24T18:10:00Z"
|
|
}
|
|
]
|
|
```
|
|
|
|
### GET /api/news/summaries — master LLM summaries
|
|
|
|
| Query param | Meaning | Default |
|
|
|---|---|---|
|
|
| `since` | only summaries generated at/after this UTC instant | none |
|
|
| `limit` | max rows | `20` (max `100`) |
|
|
| `offset` | pagination offset | `0` |
|
|
|
|
```json
|
|
[
|
|
{
|
|
"id": 1,
|
|
"summary_text": "master LLM summary (markdown)…",
|
|
"batch_timestamp": "2026-08-24T18:10:00Z"
|
|
}
|
|
]
|
|
```
|
|
|
|
## Configuration (all via env / `.env`)
|
|
|
|
| Var | Default | Notes |
|
|
|---|---|---|
|
|
| `GEMINI_API_KEY` | *(blank)* | **Required for summaries.** Unset = summarizer logs and idles (never crashes). |
|
|
| `SUMMARY_MODEL` | `gemini-2.0-flash` | Gemini model id. |
|
|
| `NEWS_BATCH_SIZE` | `50` | Articles per map-phase batch. |
|
|
| `SUMMARY_WINDOW_HOURS` | `1` | How far back the summarizer looks for new articles. |
|
|
| `INCLUDE_FUTURES` | `0` | Legacy futures-prices coupling (upstream pipeline). OFF for OSINT; set `1` + install `yfinance` to enable. |
|
|
| `NEWS_SCRAPE_MINUTE` | `0` | Wall-clock minute the scraper fires. |
|
|
| `NEWS_SUMMARIZE_MINUTE` | `5` | Wall-clock minute the summarizer fires. |
|
|
| `NEWS_SCRAPE_RUN_ON_START` | `1` | Run one scrape immediately on container start. |
|
|
| `NEWS_SUMMARIZE_RUN_ON_START` | `1` | Run one summarize immediately on container start. |
|
|
| `NEWS_LOG_LEVEL` | `INFO` | Scrapy log level. |
|
|
| `TELEGRAM_TOKEN` / `TELEGRAM_CHAT_ID` | *(blank)* | Reserved for the (out-of-scope) Telegram delivery bot. |
|
|
|
|
DB_* for both services is mapped to the shared osint-db credentials
|
|
(`DB_HOST=db`, same `DB_USER/DB_PASSWORD/DB_NAME` as the rest of the stack).
|
|
|
|
## Prompts
|
|
|
|
Both prompts are env-overridable — the default `MAP_PROMPT` is OSINT-neutral
|
|
(facts, locations, entities, category, OSINT signal per article) and the default
|
|
`SUMMARY_PROMPT` produces a concise executive summary of the most impactful
|
|
items (with a "no qualifying events" escape hatch). Upstream's futures/markets
|
|
prompt language is gated behind `INCLUDE_FUTURES=1`.
|
|
|
|
## Tests
|
|
|
|
`tests/test_api_news.py` — DB-backed API contract tests (auto-skip without a
|
|
reachable test database, same as the FIRMS tests):
|
|
|
|
```bash
|
|
DB_HOST=... DB_PORT=... DB_USER=osint DB_PASSWORD=... DB_NAME=osint_data \
|
|
pytest tests/test_api_news.py -v
|
|
```
|
|
|
|
## Live verification
|
|
|
|
End-to-end (real crawl → DB → API) is verified after deploy on the Pi: check
|
|
`docker compose --profile ingest logs -f news-scraper news-summarizer`, then
|
|
`curl -s localhost:8000/api/news | head`. Summaries additionally require
|
|
`GEMINI_API_KEY` to be set in `.env` on the Pi.
|