91f436390b
Some checks failed
build-and-deploy / build (push) Failing after 4s
Vendor the newsPipeline scraper + summarizer into the repo and wire them into
docker-compose against the EXISTING osint-db (no second Postgres), replacing
the upstream k8s CronJobs with in-compose wall-clock loops (:00 scrape, :05
summarize).
- news/scraper: vendored Scrapy project (257 RSS feeds) + hourly loop
scheduler (run_news_scraper.py)
- news/summerizer: vendored Gemini map-reduce summarizer, cleaned:
* fix broken google-genai response handling (_extract_text, defensive)
* fix malformed INSERT/GRANT query in save_summary_to_db
* OSINT-neutral default MAP_PROMPT; futures/markets language gated behind
INCLUDE_FUTURES=0 (yfinance lazy-imported)
* env-configurable model, batch size, lookback window
+ hourly loop scheduler (run_news_summarizer.py, :05)
- alembic 003_news: idempotent articles + article_summaries tables
- API: GET /api/news and GET /api/news/summaries (+ models, schemas)
- tests/test_api_news.py: 5 DB-backed contract tests (all pass vs real PG)
- docs/news.md + .env.example updates
Both services run under the `ingest` compose profile (matching the
ingester/camera-scraper pattern) and build arm64 on the Pi via the existing
Forgejo CI workflow. telebot left out of scope (reserved env only).
44 lines
812 B
Text
44 lines
812 B
Text
attrs==25.4.0
|
|
Automat==25.4.16
|
|
brotli==1.2.0
|
|
certifi==2026.1.4
|
|
cffi==2.0.0
|
|
charset-normalizer==3.4.4
|
|
constantly==23.10.4
|
|
cryptography==46.0.3
|
|
cssselect==1.3.0
|
|
defusedxml==0.7.1
|
|
filelock==3.20.2
|
|
hyperlink==21.0.0
|
|
idna==3.11
|
|
Incremental==24.11.0
|
|
itemadapter==0.13.1
|
|
itemloaders==1.3.2
|
|
jmespath==1.0.1
|
|
lxml==6.0.2
|
|
packaging==25.0
|
|
parsel==1.10.0
|
|
Protego==0.5.0
|
|
pyasn1==0.6.1
|
|
pyasn1_modules==0.4.2
|
|
pycparser==2.23
|
|
PyDispatcher==2.0.7
|
|
pyOpenSSL==25.3.0
|
|
queuelib==1.8.0
|
|
requests==2.32.5
|
|
requests-file==3.0.1
|
|
Scrapy==2.14.0
|
|
scrapy-user-agents==0.1.1
|
|
service-identity==24.2.0
|
|
tldextract==5.3.1
|
|
Twisted==25.5.0
|
|
typing_extensions==4.15.0
|
|
ua-parser==1.0.1
|
|
ua-parser-builtins==202601
|
|
urllib3==2.6.3
|
|
user-agents==2.2.0
|
|
w3lib==2.3.1
|
|
zope.interface==8.1.1
|
|
psycopg2-binary==2.9.11
|
|
python-dotenv==0.20.0
|
|
scrapy-playwright==0.0.45
|