91f436390b
Some checks failed
build-and-deploy / build (push) Failing after 4s
Vendor the newsPipeline scraper + summarizer into the repo and wire them into
docker-compose against the EXISTING osint-db (no second Postgres), replacing
the upstream k8s CronJobs with in-compose wall-clock loops (:00 scrape, :05
summarize).
- news/scraper: vendored Scrapy project (257 RSS feeds) + hourly loop
scheduler (run_news_scraper.py)
- news/summerizer: vendored Gemini map-reduce summarizer, cleaned:
* fix broken google-genai response handling (_extract_text, defensive)
* fix malformed INSERT/GRANT query in save_summary_to_db
* OSINT-neutral default MAP_PROMPT; futures/markets language gated behind
INCLUDE_FUTURES=0 (yfinance lazy-imported)
* env-configurable model, batch size, lookback window
+ hourly loop scheduler (run_news_summarizer.py, :05)
- alembic 003_news: idempotent articles + article_summaries tables
- API: GET /api/news and GET /api/news/summaries (+ models, schemas)
- tests/test_api_news.py: 5 DB-backed contract tests (all pass vs real PG)
- docs/news.md + .env.example updates
Both services run under the `ingest` compose profile (matching the
ingester/camera-scraper pattern) and build arm64 on the Pi via the existing
Forgejo CI workflow. telebot left out of scope (reserved env only).
76 lines
3.5 KiB
Text
76 lines
3.5 KiB
Text
# OSINT Dashboard — environment configuration
|
|
# Copy to `.env` and adjust. All values have safe defaults for local compose.
|
|
|
|
# ── PostgreSQL / TimescaleDB ───────────────────────────────────────────────
|
|
DB_USER=osint
|
|
DB_PASSWORD=osint
|
|
DB_HOST=db
|
|
DB_PORT=5432
|
|
DB_NAME=osint_data
|
|
|
|
# ── NATS JetStream (optional — only needed when profile `ingest` is used) ──
|
|
NATS_URL=nats://nats:4222
|
|
|
|
# ── MinIO (optional — document storage; endpoint used if wired later) ──────
|
|
MINIO_ENDPOINT=minio:9000
|
|
MINIO_ACCESS_KEY=
|
|
MINIO_SECRET_KEY=
|
|
# "true" for TLS endpoints (e.g. S3-compatible prod); "false" for local HTTP.
|
|
MINIO_SECURE=false
|
|
|
|
# ── Camera discovery scraper ───────────────────────────────────────────────
|
|
# Comma-separated public directory/list URLs (Insecam-style pages or
|
|
# plain-text lists: url[|lat,lon|vendor|location]).
|
|
CAMERA_SOURCE_URLS=
|
|
CAMERA_SCRAPE_INTERVAL=3600
|
|
CAMERA_REQUEST_DELAY=2.0
|
|
NOMINATIM_URL=https://nominatim.openstreetmap.org
|
|
NOMINATIM_MIN_INTERVAL=1.1
|
|
SNAPSHOT_TTL_SECONDS=300
|
|
|
|
# ── NASA FIRMS (active fire / hotspot ingest) ──────────────────────────────
|
|
# MAP_KEY is FREE — get one at https://firms.modaps.eosdis.nasa.gov/api/map_key_info/
|
|
# (1-minute signup, no payment). Leave blank to keep fire ingest idle.
|
|
FIRMS_MAP_KEY=
|
|
# NRT VIIRS S-NPP 375m active fire detection (near-real-time).
|
|
FIRMS_DATASET=VIIRS_SNPP_NRT
|
|
# Area to poll as "minlon,minlat,maxlon,maxlat". Narrow it to reduce payload
|
|
# (e.g. CONUS "-125,24,-66,50"). Default covers most of the inhabited globe.
|
|
FIRMS_BBOX=-180,-60,180,75
|
|
# Poll cadence in seconds (~15 min). FIRMS NRT updates every ~5-10 min.
|
|
FIRMS_INTERVAL=900
|
|
# Set to 0 to disable the fire loop entirely.
|
|
INGEST_FIRES=1
|
|
|
|
# ── API keys (managed from the dashboard UI) ──────────────────────────────
|
|
# Keys such as GEMINI_API_KEY and TELEGRAM_TOKEN are stored in the Postgres
|
|
# `api_keys` table and managed from the dashboard's "Keys" tab
|
|
# (GET/POST/DELETE /api/keys/{name}) — see app/keystore.py. The FIRMS ingestor
|
|
# currently reads FIRMS_MAP_KEY from .env (above); wiring the Keys-UI store as
|
|
# its lookup/fallback is a planned follow-up.
|
|
|
|
# ── News pipeline (scraper + summarizer, profile `ingest`) ────────────────
|
|
# Hourly: the scraper crawls 257 RSS sources at minute :00 and the summarizer
|
|
# runs the Gemini map-reduce at minute :05, both writing to the shared osint-db
|
|
# (tables `articles` + `article_summaries`, created by alembic 003_news).
|
|
# Consume via GET /api/news and GET /api/news/summaries.
|
|
# GEMINI_API_KEY is REQUIRED for summarization; unset = summarizer idles.
|
|
GEMINI_API_KEY=
|
|
# Optional LLM knobs
|
|
SUMMARY_MODEL=gemini-2.0-flash
|
|
NEWS_BATCH_SIZE=50
|
|
SUMMARY_WINDOW_HOURS=1
|
|
# Futures/markets coupling from the upstream pipeline is OFF by default
|
|
# (irrelevant to OSINT). Set INCLUDE_FUTURES=1 + install yfinance to enable.
|
|
INCLUDE_FUTURES=0
|
|
# Wall-clock scheduling (k8s CronJob replacement): scrape minute, summarize minute
|
|
NEWS_SCRAPE_MINUTE=0
|
|
NEWS_SUMMARIZE_MINUTE=5
|
|
# Run once immediately on container start (seeds data fast), then align to the
|
|
# scheduled minute.
|
|
NEWS_SCRAPE_RUN_ON_START=1
|
|
NEWS_SUMMARIZE_RUN_ON_START=1
|
|
NEWS_LOG_LEVEL=INFO
|
|
# Reserved for the (out-of-scope) Telegram delivery bot.
|
|
TELEGRAM_TOKEN=
|
|
TELEGRAM_CHAT_ID=
|