91f436390b
Some checks failed
build-and-deploy / build (push) Failing after 4s
Vendor the newsPipeline scraper + summarizer into the repo and wire them into
docker-compose against the EXISTING osint-db (no second Postgres), replacing
the upstream k8s CronJobs with in-compose wall-clock loops (:00 scrape, :05
summarize).
- news/scraper: vendored Scrapy project (257 RSS feeds) + hourly loop
scheduler (run_news_scraper.py)
- news/summerizer: vendored Gemini map-reduce summarizer, cleaned:
* fix broken google-genai response handling (_extract_text, defensive)
* fix malformed INSERT/GRANT query in save_summary_to_db
* OSINT-neutral default MAP_PROMPT; futures/markets language gated behind
INCLUDE_FUTURES=0 (yfinance lazy-imported)
* env-configurable model, batch size, lookback window
+ hourly loop scheduler (run_news_summarizer.py, :05)
- alembic 003_news: idempotent articles + article_summaries tables
- API: GET /api/news and GET /api/news/summaries (+ models, schemas)
- tests/test_api_news.py: 5 DB-backed contract tests (all pass vs real PG)
- docs/news.md + .env.example updates
Both services run under the `ingest` compose profile (matching the
ingester/camera-scraper pattern) and build arm64 on the Pi via the existing
Forgejo CI workflow. telebot left out of scope (reserved env only).
24 lines
674 B
Docker
24 lines
674 B
Docker
FROM python:3.11-slim
|
|
|
|
ENV PYTHONDONTWRITEBYTECODE=1
|
|
ENV PYTHONUNBUFFERED=1
|
|
|
|
WORKDIR /app
|
|
|
|
# libpq-dev + gcc for psycopg2 build/adapters; keep the image lean.
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
libpq-dev gcc \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
COPY requirements.txt .
|
|
RUN pip install --no-cache-dir -r requirements.txt
|
|
|
|
COPY summarizer.py run_news_summarizer.py ./
|
|
|
|
# Security: run as a non-privileged user.
|
|
RUN useradd -m summarizer_user
|
|
USER summarizer_user
|
|
|
|
# Default: single pass (as the old k8s CronJob ran). The compose service
|
|
# overrides `command` to the scheduler loop (run_news_summarizer.py).
|
|
CMD ["python", "summarizer.py"]
|