5849d72299
Some checks failed
build-and-deploy / build (push) Failing after 4s
The vendored spider only started on urls.txt lines containing '/rss' or '/feed' — but the curated urls.txt holds 200 homepages, so the crawler matched ZERO feeds and silently did nothing (2ms, 0 items). Rewrite the spider with feed autodiscovery: fetch each homepage, find its <link rel="alternate" type="application/rss+xml"> (or /feed|/rss link), parse the feed, then follow each item to extract the article. Bump DEPTH_LIMIT 1->3 (homepage -> feed -> article). Also make the summarizer resilient to booting before the app container has run alembic (docker-compose only guarantees `db` is up): add idempotent ensure_tables() mirroring the scraper's CREATE TABLE IF NOT EXISTS. |
||
|---|---|---|
| .. | ||
| spiders | ||
| __init__.py | ||
| items.py | ||
| middlewares.py | ||
| pipelines.py | ||
| settings.py | ||