91f436390b
Some checks failed
build-and-deploy / build (push) Failing after 4s
Vendor the newsPipeline scraper + summarizer into the repo and wire them into
docker-compose against the EXISTING osint-db (no second Postgres), replacing
the upstream k8s CronJobs with in-compose wall-clock loops (:00 scrape, :05
summarize).
- news/scraper: vendored Scrapy project (257 RSS feeds) + hourly loop
scheduler (run_news_scraper.py)
- news/summerizer: vendored Gemini map-reduce summarizer, cleaned:
* fix broken google-genai response handling (_extract_text, defensive)
* fix malformed INSERT/GRANT query in save_summary_to_db
* OSINT-neutral default MAP_PROMPT; futures/markets language gated behind
INCLUDE_FUTURES=0 (yfinance lazy-imported)
* env-configurable model, batch size, lookback window
+ hourly loop scheduler (run_news_summarizer.py, :05)
- alembic 003_news: idempotent articles + article_summaries tables
- API: GET /api/news and GET /api/news/summaries (+ models, schemas)
- tests/test_api_news.py: 5 DB-backed contract tests (all pass vs real PG)
- docs/news.md + .env.example updates
Both services run under the `ingest` compose profile (matching the
ingester/camera-scraper pattern) and build arm64 on the Pi via the existing
Forgejo CI workflow. telebot left out of scope (reserved env only).
113 lines
3.9 KiB
Python
113 lines
3.9 KiB
Python
# Define here the models for your spider middleware
|
|
#
|
|
# See documentation in:
|
|
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
|
|
|
|
from scrapy import signals
|
|
|
|
# useful for handling different item types with a single interface
|
|
from itemadapter import ItemAdapter
|
|
|
|
|
|
class NewsscraperSpiderMiddleware:
|
|
# Not all methods need to be defined. If a method is not defined,
|
|
# scrapy acts as if the spider middleware does not modify the
|
|
# passed objects.
|
|
|
|
@classmethod
|
|
def from_crawler(cls, crawler):
|
|
# This method is used by Scrapy to create your spiders.
|
|
s = cls()
|
|
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
|
|
return s
|
|
|
|
def process_spider_input(self, response, spider):
|
|
# Called for each response that goes through the spider
|
|
# middleware and into the spider.
|
|
|
|
# Should return None or raise an exception.
|
|
return None
|
|
|
|
def process_spider_output(self, response, result, spider):
|
|
# Called with the results returned from the Spider, after
|
|
# it has processed the response.
|
|
|
|
# Must return an iterable of Request, or item objects.
|
|
for i in result:
|
|
yield i
|
|
|
|
def process_spider_exception(self, response, exception, spider):
|
|
# Called when a spider or process_spider_input() method
|
|
# (from other spider middleware) raises an exception.
|
|
|
|
# Should return either None or an iterable of Request or item objects.
|
|
pass
|
|
|
|
async def process_start(self, start):
|
|
# Called with an async iterator over the spider start() method or the
|
|
# matching method of an earlier spider middleware.
|
|
async for item_or_request in start:
|
|
yield item_or_request
|
|
|
|
def spider_opened(self, spider):
|
|
spider.logger.info("Spider opened: %s" % spider.name)
|
|
|
|
|
|
class NewsscraperDownloaderMiddleware:
|
|
# Not all methods need to be defined. If a method is not defined,
|
|
# scrapy acts as if the downloader middleware does not modify the
|
|
# passed objects.
|
|
|
|
@classmethod
|
|
def from_crawler(cls, crawler):
|
|
# This method is used by Scrapy to create your spiders.
|
|
s = cls()
|
|
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
|
|
return s
|
|
|
|
def process_request(self, request, spider):
|
|
# Called for each request that goes through the downloader
|
|
# middleware.
|
|
|
|
# Must either:
|
|
# - return None: continue processing this request
|
|
# - or return a Response object
|
|
# - or return a Request object
|
|
# - or raise IgnoreRequest: process_exception() methods of
|
|
# installed downloader middleware will be called
|
|
return None
|
|
|
|
def process_response(self, request, response, spider):
|
|
# Called with the response returned from the downloader.
|
|
|
|
# Must either;
|
|
# - return a Response object
|
|
# - return a Request object
|
|
# - or raise IgnoreRequest
|
|
return response
|
|
|
|
def process_exception(self, request, exception, spider):
|
|
# Called when a download handler or a process_request()
|
|
# (from other downloader middleware) raises an exception.
|
|
|
|
# Must either:
|
|
# - return None: continue processing this exception
|
|
# - return a Response object: stops process_exception() chain
|
|
# - return a Request object: stops process_exception() chain
|
|
pass
|
|
|
|
def spider_opened(self, spider):
|
|
spider.logger.info("Spider opened: %s" % spider.name)
|
|
|
|
class ProxyMiddleware:
|
|
def __init__(self, proxy_url):
|
|
self.proxy_url = proxy_url
|
|
|
|
@classmethod
|
|
def from_crawler(cls, crawler):
|
|
return cls(proxy_url=crawler.settings.get('PROXY_URL'))
|
|
|
|
def process_request(self, request, spider):
|
|
# Only attach the proxy if PROXY_URL was successfully built
|
|
if self.proxy_url:
|
|
request.meta['proxy'] = self.proxy_url
|