The vendored spider only started on urls.txt lines containing '/rss' or
'/feed' — but the curated urls.txt holds 200 homepages, so the crawler
matched ZERO feeds and silently did nothing (2ms, 0 items). Rewrite the
spider with feed autodiscovery: fetch each homepage, find its
<link rel="alternate" type="application/rss+xml"> (or /feed|/rss link),
parse the feed, then follow each item to extract the article. Bump
DEPTH_LIMIT 1->3 (homepage -> feed -> article).
Also make the summarizer resilient to booting before the app container has
run alembic (docker-compose only guarantees `db` is up): add idempotent
ensure_tables() mirroring the scraper's CREATE TABLE IF NOT EXISTS.
Vendor the newsPipeline scraper + summarizer into the repo and wire them into
docker-compose against the EXISTING osint-db (no second Postgres), replacing
the upstream k8s CronJobs with in-compose wall-clock loops (:00 scrape, :05
summarize).
- news/scraper: vendored Scrapy project (257 RSS feeds) + hourly loop
scheduler (run_news_scraper.py)
- news/summerizer: vendored Gemini map-reduce summarizer, cleaned:
* fix broken google-genai response handling (_extract_text, defensive)
* fix malformed INSERT/GRANT query in save_summary_to_db
* OSINT-neutral default MAP_PROMPT; futures/markets language gated behind
INCLUDE_FUTURES=0 (yfinance lazy-imported)
* env-configurable model, batch size, lookback window
+ hourly loop scheduler (run_news_summarizer.py, :05)
- alembic 003_news: idempotent articles + article_summaries tables
- API: GET /api/news and GET /api/news/summaries (+ models, schemas)
- tests/test_api_news.py: 5 DB-backed contract tests (all pass vs real PG)
- docs/news.md + .env.example updates
Both services run under the `ingest` compose profile (matching the
ingester/camera-scraper pattern) and build arm64 on the Pi via the existing
Forgejo CI workflow. telebot left out of scope (reserved env only).
