sirius/osint-dashboard
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
osint-dashboard/news/summerizer/requirements.txt

3 lines
38 B
Text
Raw Normal View History

Add news pipeline: hourly scraper + Gemini summarizer in compose Vendor the newsPipeline scraper + summarizer into the repo and wire them into docker-compose against the EXISTING osint-db (no second Postgres), replacing the upstream k8s CronJobs with in-compose wall-clock loops (:00 scrape, :05 summarize). - news/scraper: vendored Scrapy project (257 RSS feeds) + hourly loop scheduler (run_news_scraper.py) - news/summerizer: vendored Gemini map-reduce summarizer, cleaned: * fix broken google-genai response handling (_extract_text, defensive) * fix malformed INSERT/GRANT query in save_summary_to_db * OSINT-neutral default MAP_PROMPT; futures/markets language gated behind INCLUDE_FUTURES=0 (yfinance lazy-imported) * env-configurable model, batch size, lookback window + hourly loop scheduler (run_news_summarizer.py, :05) - alembic 003_news: idempotent articles + article_summaries tables - API: GET /api/news and GET /api/news/summaries (+ models, schemas) - tests/test_api_news.py: 5 DB-backed contract tests (all pass vs real PG) - docs/news.md + .env.example updates Both services run under the `ingest` compose profile (matching the ingester/camera-scraper pattern) and build arm64 on the Pi via the existing Forgejo CI workflow. telebot left out of scope (reserved env only).
2026-08-24 17:28:46 -04:00
psycopg2-binary==2.9.11
chore: point news-summarizer compose at Nous portal
2026-08-27 23:24:45 -04:00
httpx==0.28.1
Reference in a new issue Copy permalink