clean up
This commit is contained in:
sirius0xdev
parent
4f80f1eede
commit
99cf482ba7
5 changed files with 336 additions and 16 deletions
|
|
@ -1,7 +0,0 @@
|
|||
{
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"Bash(tree:*)"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
@ -11,9 +11,27 @@ data:
|
|||
LLM_URL: "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent"
|
||||
MODEL_NAME: "gemini-2.0-flash"
|
||||
SUMMARY_PROMPT: |
|
||||
You are a professional news editor. Combine these findings into
|
||||
a Telegram-friendly update with emojis and bold headers. Translate everything to english
|
||||
and give the update in english as well.
|
||||
You are an advanced LLM news analyst tasked with processing hourly feeds from 300 diverse news sources worldwide, including major outlets like BBC, CNN, Al Jazeera, Reuters, Xinhua, The New York Times, Le Monde, and regional sources from Asia, Europe, Africa, Latin America, and the Middle East. Your goal is to identify and summarize the 5-10 most important news stories of the past hour, prioritizing those with the highest global significance, urgency, or potential impact.Step-by-Step Analysis Process:Aggregate and Deduplicate: Review all incoming articles, headlines, and summaries from the 300 sources. Group similar stories by topic (e.g., using semantic similarity clustering) to avoid redundancy. Identify clusters based on key entities, events, locations, and themes.
|
||||
Evaluate Importance: Score each unique story or cluster on a scale of 1-10 based on these criteria:Global Impact: Does it affect multiple countries, economies, or populations? (e.g., geopolitical conflicts, pandemics, market crashes).
|
||||
Urgency and Timeliness: Is it breaking news with immediate consequences? (e.g., natural disasters, political announcements, security threats).
|
||||
Novelty: Is it a new development or update on an ongoing major story, not just routine reporting?
|
||||
Cross-Source Coverage: How many sources are reporting it? Higher coverage indicates broader relevance (e.g., 50+ sources = high score).
|
||||
Diversity of Perspectives: Include stories that represent balanced viewpoints from different regions or ideologies to avoid bias.
|
||||
Human Interest or Long-Term Significance: Stories involving humanitarian crises, technological breakthroughs, environmental issues, or cultural shifts.
|
||||
Penalize sensationalism, celebrity gossip, or hyper-local stories unless they have wider implications.
|
||||
|
||||
Filter and Rank: Select the top 5-10 stories with the highest scores. Ensure diversity: no more than 2-3 from the same category (e.g., politics, economy, science). If ties occur, prioritize underrepresented regions or sources.
|
||||
Summarize Objectively: For each selected story, provide:A neutral, concise headline (1 sentence).
|
||||
A 2-4 sentence summary capturing key facts: who, what, when, where, why, and how.
|
||||
Key sources (e.g., "Reported by BBC, Reuters, and Al Jazeera").
|
||||
Potential implications or context (1 sentence).
|
||||
Avoid opinions, speculation, or bias—stick to verified facts from the sources.
|
||||
|
||||
Output Format:Output in a clean, readable markdown format:Top News of the Hour: [Current Hour, e.g., 14:00-15:00 UTC]
|
||||
Numbered list of stories, starting with the most important.
|
||||
At the end, include a brief "Trends Overview" section noting any emerging patterns across stories (e.g., "Rising tensions in geopolitics dominate this hour").
|
||||
|
||||
If no major news emerges, state: "This hour's feeds show no standout global events; monitoring for developments.
|
||||
|
||||
DATA TO SUMMARIZE:
|
||||
{data}
|
||||
|
|
|
|||
|
|
@ -12,8 +12,14 @@ spec:
|
|||
spec:
|
||||
containers:
|
||||
- name: news-scraper
|
||||
image: siriussec/newsscraper:2.1
|
||||
|
||||
image: siriussec/newsscraper:0.5.1
|
||||
resources:
|
||||
requests:
|
||||
memory: "256Mi"
|
||||
cpu: "100m"
|
||||
limits:
|
||||
memory: "512Mi"
|
||||
cpu: "500m"
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: scrapy-config
|
||||
|
|
@ -27,9 +33,15 @@ spec:
|
|||
- name: url-storage
|
||||
mountPath: /app/urls.txt
|
||||
subPath: urls.txt
|
||||
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
volumes:
|
||||
- name: url-storage
|
||||
configMap:
|
||||
name: scrapy-urls
|
||||
- name: dshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
restartPolicy: OnFailure
|
||||
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ data:
|
|||
DB_PORT: "5432"
|
||||
|
||||
# Scrapy Settings
|
||||
LOG_LEVEL: "INFO"
|
||||
LOG_LEVEL: "WARNING"
|
||||
USER_AGENT: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) NewsScraper/1.0"
|
||||
|
||||
# Proxy Endpoint (The URL part only)
|
||||
|
|
|
|||
|
|
@ -182,3 +182,300 @@ data:
|
|||
https://www.wvgazettemail.com
|
||||
https://www.jsonline.com
|
||||
https://www.trib.com
|
||||
http://worldunitednews.blogspot.com/
|
||||
https://quicktrendnews.wordpress.com/
|
||||
https://www.newsdesk.io/
|
||||
https://3-mob.com/
|
||||
https://smallwarsjournal.com/recent/
|
||||
https://globalpressjournal.com/
|
||||
https://www.thetimesinplainenglish.com/
|
||||
https://newswarden.com/
|
||||
https://www.buzzfeednews.com/section/world
|
||||
https://www.spiegel.de/
|
||||
https://www.thestar.com/news/world/
|
||||
https://ifpnews.com/
|
||||
https://cbn.com/cwn
|
||||
https://worldnews24you.blogspot.com/?m=1
|
||||
https://soulwriteempire.com/
|
||||
https://newszetu.com/
|
||||
https://www.carihargaterupdate.com/
|
||||
https://statehooddc.com/
|
||||
https://www.suresource24.com/
|
||||
https://www.usnn.news/
|
||||
https://www.worldstopexports.com/
|
||||
https://www.headlinesoftoday.com/
|
||||
https://articleify.com/
|
||||
https://www.thedailycanvas.com/
|
||||
https://wng.org/topics/international
|
||||
https://insidexpress.com/
|
||||
https://www.reuters.com/
|
||||
https://www.yahoo.com/news/world/
|
||||
https://wowplus.net/category/world-news/niger..
|
||||
https://www.scmp.com/
|
||||
https://worldweeklynews.com/
|
||||
https://timesofindia.indiatimes.com/world
|
||||
https://radarr.africa/
|
||||
# --- GLOBAL / WORLD NEWS (Text-Only & Lite) ---
|
||||
https://text.npr.org
|
||||
https://lite.cnn.com
|
||||
https://www.cbc.ca/lite/news
|
||||
https://www.csmonitor.com/text_only
|
||||
https://feeds.bbci.co.uk/news/world/rss.xml
|
||||
https://www.reutersagency.com/feed/
|
||||
|
||||
# --- NORTH CAROLINA (RSS & Local) ---
|
||||
https://www.wral.com/news/rss/48
|
||||
https://www.newsobserver.com/?service=rss
|
||||
https://abc11.com/feed/
|
||||
https://www.wcnc.com/rss
|
||||
https://www.wbtv.com/arc/outboundfeeds/rss/
|
||||
https://www.northcarolinahealthnews.org/feed/
|
||||
https://indyweek.com/feed/
|
||||
|
||||
# --- NEW YORK (RSS & Local) ---
|
||||
https://nypost.com/feed/
|
||||
https://gothamist.com/feed/
|
||||
https://www.cityandstateny.com/rss
|
||||
https://feeds.nytimes.com/nyt/rss/HomePage
|
||||
https://www.thecity.nyc/rss/index.xml
|
||||
https://brooklyneagle.com/feed/
|
||||
https://www.reutersagency.com/feed/,Hard
|
||||
https://newsatme.com/api/v1/rss/ap/world,Global
|
||||
https://feeds.bbci.co.uk/news/world/rss.xml,International
|
||||
https://www.aljazeera.com/xml/rss/all.xml,Global
|
||||
https://rss.dw.com/rdf/rss-en-all,European
|
||||
https://www.france24.com/en/rss,Global
|
||||
https://www3.nhk.or.jp/rss/news/shakaitokushu.xml,Asia-Pacific
|
||||
https://www.cbc.ca/cctoc/rss/topstories.north
|
||||
https://www.defensenews.com/arc/outboundfeeds/rss/
|
||||
https://therecord.media/feed
|
||||
https://www.cfr.org/rss/newsletters/daily-news-brief
|
||||
https://warontherocks.com/feed/
|
||||
https://www.reutersagency.com/feed/
|
||||
https://www.thecipherbrief.com/feed
|
||||
https://www.foreignaffairs.com/rss.xml
|
||||
https://geopoliticalfutures.com/feed
|
||||
|
||||
# --- TACTICAL CYBER & VULNERABILITIES ---
|
||||
https://www.bleepingcomputer.com/feed/
|
||||
https://www.cisa.gov/cybersecurity-advisory-feeds
|
||||
https://krebsonsecurity.com/feed/
|
||||
https://thehackernews.com/feeds/posts/default
|
||||
https://www.darkreading.com/rss.xml
|
||||
https://www.mandiant.com/resources/blog/rss.xml
|
||||
https://schneier.com/feed/atom/
|
||||
https://www.securityweek.com/feed/
|
||||
|
||||
# --- REGIONAL THREAT LANDSCAPE ---
|
||||
https://www.aljazeera.com/xml/rss/all.xml
|
||||
https://www.thenationalnews.com/rss/
|
||||
https://www.scmp.com/rss/91/feed
|
||||
https://www.batimes.com.ar/rss
|
||||
https://brazilian.report/feed/
|
||||
|
||||
https://www.wvtm13.com/topstories-rss
|
||||
https://dothaneagle.com/search/?f=rss&t=article&c=news&l=50&s=start_time&sd=desc
|
||||
https://www.ktoo.org/feed/
|
||||
https://www.juneauempire.com/feed/
|
||||
https://www.nomenugget.net/rss.xml
|
||||
https://www.homernews.com/feed/
|
||||
https://azmirror.com/feed/
|
||||
https://azcapitoltimes.com/feed/
|
||||
https://azdailysun.com/search/?f=rss&t=article&c=news/local&l=50&s=start_time&sd=desc
|
||||
http://rssfeeds.12news.com/kpnx/home
|
||||
https://nwahomepage.com/feed
|
||||
https://arktimes.com/feed
|
||||
https://arkansasbusiness.com/feed
|
||||
https://talkbusiness.net/feed
|
||||
https://www.latimes.com/news/rss2.0.xml
|
||||
https://www.nbclosangeles.com/?rss=y
|
||||
https://calmatters.org/feed/
|
||||
https://www.mercurynews.com/feed/
|
||||
https://www.denver7.com/news/front-range/fort-collins.rss
|
||||
https://www.denverpost.com/feed/
|
||||
https://coloradosun.com/feed/
|
||||
https://sentinelcolorado.com/feed/
|
||||
https://www.wtnh.com/feed/
|
||||
https://www.ctpost.com/rss/feed/News-1331.php
|
||||
https://ctmirror.org/feed/
|
||||
https://ctnewsjunkie.com/feed/
|
||||
https://delawarelive.com/feed/
|
||||
https://delawaretoday.com/feed/
|
||||
http://delawarefreenews.org/feed/
|
||||
https://www.wdel.com/
|
||||
https://www.orlandosentinel.com/feed/
|
||||
https://www.wfla.com/news/florida/feed/
|
||||
https://therealdeal.com/miami/feed/
|
||||
https://www.local10.com/arc/outboundfeeds/rss/category/news/?outputType=xml&size=10
|
||||
https://www.gainesvilletimes.com/rss/
|
||||
https://www.covnews.com/rss/
|
||||
https://www.henryherald.com/search/?f=rss&t=article&c=news&l=50&s=start_time&sd=desc
|
||||
https://accesswdun.com/feed
|
||||
https://www.khon2.com/feed/
|
||||
https://www.staradvertiser.com/feed/
|
||||
https://www.westhawaiitoday.com/feed/
|
||||
https://mauinow.com/feed/
|
||||
https://www.idahofallsidaho.gov/RSSFeed.aspx?ModID=1&CID=All-newsflash.xml
|
||||
https://www.eastidahonews.com/feed/
|
||||
https://localnews8.com/feed/
|
||||
https://www.boisestatepublicradio.org/news.rss
|
||||
https://www.illinoistimes.com/springfield/Rss.xml
|
||||
https://www.thecentersquare.com/search/?f=rss&t=article&l=20&s=start_time&fulltext=showtext&sd=desc&c%5B%5D=Illinois
|
||||
https://chicago.suntimes.com/rss/index.xml
|
||||
https://wgntv.com/feed/
|
||||
http://feeds.indiana.statenews.net/rss/7b3aa09cdd5d5eac
|
||||
https://fox59.com/feed/
|
||||
https://www.nwitimes.com/search/?f=rss&t=article&c=news/local&l=50&s=start_time&sd=desc
|
||||
https://www.wishtv.com/feed/
|
||||
https://www.kcci.com/topstories-rss
|
||||
https://www.myiowainfo.com/feed/
|
||||
https://feeds.feedburner.com/radioiowanews
|
||||
https://www.mississippivalleypublishing.com/search/?f=rss&t=article&c=the_hawk_eye&l=50&s=start_time&sd=desc
|
||||
https://www.ksn.com/feed/
|
||||
https://www.ksnt.com/feed/
|
||||
https://www.hdnews.net/feed/
|
||||
https://themercury.com/search/?f=rss&t=article&c=news&l=50&s=start_time&sd=desc
|
||||
https://www.wdrb.com/search/?f=rss&t=article&c=news&l=50&s=start_time&sd=desc
|
||||
https://www.wtvq.com/feed/
|
||||
https://www.wnky.com/feed/
|
||||
https://www.wlky.com/topstories-rss
|
||||
https://thehayride.com/feed/
|
||||
https://wgno.com/feed/
|
||||
https://feeds.feedburner.com/wbrz/news
|
||||
https://thelensnola.org/feed/
|
||||
https://www.pressherald.com/news/feed/
|
||||
https://www.centralmaine.com/feed/
|
||||
https://www.bangordailynews.com/feed/
|
||||
https://www.sunjournal.com/news/feed/
|
||||
https://www.wbaltv.com/topstories-rss
|
||||
https://baltimorefishbowl.com/feed/
|
||||
https://thedailyrecord.com/feed/
|
||||
https://southernmarylandchronicle.com/feed/
|
||||
https://www.boston.com/feed/
|
||||
https://whdh.com/feed/
|
||||
https://www.wwlp.com/feed/
|
||||
https://www.lowellsun.com/feed/
|
||||
https://www.metrotimes.com/detroit/Rss.xml
|
||||
https://www.manisteenews.com/news/feed/Latest-News-Feed-2564.php
|
||||
https://www.theoaklandpress.com/feed/
|
||||
https://www.macombdaily.com/feed/
|
||||
https://www.startribune.com/local/index.rss2
|
||||
https://www.wctrib.com/index.rss
|
||||
https://www.austindailyherald.com/feed/
|
||||
https://www.duluthnewstribune.com/news.rss
|
||||
https://mississippitoday.org/feed/
|
||||
https://www.wapt.com/topstories-rss
|
||||
https://www.wjtv.com/feed/
|
||||
https://www.biloxinewsevents.com/feed/
|
||||
https://www.missourinet.com/feed/
|
||||
https://clear99.com/feed/
|
||||
https://showmeprogress.com/feed/
|
||||
https://suntimesnews.com/feed/
|
||||
https://www.montanarightnow.com/search/?f=rss&t=article&l=50&s=start_time&sd=desc
|
||||
https://helenair.com/search/?f=rss&t=article&l=50&s=start_time&sd=desc
|
||||
https://www.ktvq.com/news.rss
|
||||
https://mtstandard.com/search/?f=rss&t=article&l=50&s=start_time&sd=desc
|
||||
https://www.ketv.com/topstories-rss
|
||||
https://nebraskaexaminer.com/feed/
|
||||
https://kearneyhub.com/rss
|
||||
https://www.wowt.com/rss
|
||||
https://thenevadaindependent.com/feed/
|
||||
https://www.8newsnow.com/feed/
|
||||
https://www.reviewjournal.com/feed/
|
||||
https://thisisreno.com/feed/
|
||||
https://www.conwaydailysun.com/search/?f=rss&t=article&c=berlin_sun/community/news&l=50&s=start_time&sd=desc
|
||||
https://newhampshirebulletin.com/feed/
|
||||
https://www.nhgazette.com/feed/
|
||||
https://www.nhbr.com/feed/
|
||||
https://www.nj.com/arc/outboundfeeds/rss/?outputType=xml
|
||||
https://www.njspotlightnews.org/feed/
|
||||
https://njmonthly.com/feed/
|
||||
https://www.trentonian.com/feed/
|
||||
https://www.krqe.com/feed/
|
||||
https://www.santafenewmexican.com/search/?f=rss&t=article&l=50&s=start_time&sd=desc
|
||||
https://www.easternnewmexiconews.com/rss
|
||||
https://www.koat.com/topstories-rss
|
||||
https://rss.nytimes.com/services/xml/rss/nyt/HomePage.xml
|
||||
https://nypost.com/feed/
|
||||
https://www.thecity.nyc/feed/
|
||||
https://www.nbcnewyork.com/?rss=y
|
||||
https://www.wral.com/news/rss/48/
|
||||
https://www.cbs17.com/news/north-carolina-news/feed/
|
||||
https://abc11.com/feed/
|
||||
https://myfox8.com/news/feed/
|
||||
https://www.kxnet.com/feed/
|
||||
https://www.wday.com/feed/
|
||||
https://www.jamestownsun.com/index.rss
|
||||
https://www.inforum.com/index.rss
|
||||
http://rssfeeds.wkyc.com/wkyc/news
|
||||
https://theohiostar.com/feed/
|
||||
https://feeds.feedblitz.com/wtol/news
|
||||
https://www.wcpo.com/news.rss
|
||||
https://kfor.com/feed/
|
||||
https://oklahomawatch.org/feed/
|
||||
https://freepressokc.com/feed/
|
||||
https://osagenews.org/feed/
|
||||
http://rssfeeds.kgw.com/kgw/local
|
||||
https://www.koin.com/feed/
|
||||
https://www.bendsource.com/bend/Rss.xml/feed
|
||||
https://eugeneweekly.com/feed/
|
||||
https://www.wtae.com/topstories-rss
|
||||
https://www.montgomerycountypa.gov/RSSFeed.aspx?ModID=76&CID=All-0
|
||||
https://www.mainlinemedianews.com/feed/
|
||||
https://www.dailylocal.com/feed/
|
||||
https://www.wpri.com/feed/
|
||||
https://www.abc6.com/feed/
|
||||
https://whdh.com/regional/rhode-island/feed/
|
||||
https://warwickpost.com/feed/
|
||||
https://www.wyff4.com/topstories-rss
|
||||
http://rssfeeds.wltx.com/wltx/topstories
|
||||
https://www.counton2.com/feed/
|
||||
https://www.abccolumbia.com/feed/
|
||||
https://www.keloland.com/feed/
|
||||
https://www.mitchellrepublic.com/index.rss
|
||||
https://drgnews.com/feed/
|
||||
https://www.bhpioneer.com/search/?f=rss&t=article&l=50&s=start_time&sd=desc&c=news/state_news
|
||||
https://tennesseestar.com/news/feed/
|
||||
https://tntribune.com/feed/
|
||||
https://www.wkrn.com/news/tennessee-news/feed/
|
||||
https://wpln.org/feed/
|
||||
https://abc13.com/feed/
|
||||
https://www.mysanantonio.com/default/feed/local-news-176.php
|
||||
https://www.wfaa.com/feeds/syndication/rss/news
|
||||
https://www.kxan.com/feed/
|
||||
https://www.fox13now.com/index.rss
|
||||
https://www.parkrecord.com/feed/
|
||||
https://suindependent.com/feed/
|
||||
https://lehifreepress.com/feed/
|
||||
https://www.sevendaysvt.com/vermont/Rss.xml/feed
|
||||
https://www.addisonindependent.com/feed/
|
||||
https://thevermontstandard.com/feed/
|
||||
https://www.reformer.com/feed/
|
||||
https://www.wric.com/app-feed
|
||||
https://www.styleweekly.com/feed/
|
||||
https://valawyersweekly.com/feed/
|
||||
https://richmond.com/search/?f=rss&t=article&c=news/state-regional&l=50&s=start_time&sd=desc
|
||||
https://mynorthwest.com/feed
|
||||
https://www.columbian.com/feed/
|
||||
https://rssfeeds.king5.com/king5/home
|
||||
https://www.westsideseattle.com/rss.xml
|
||||
https://www.wboy.com/feed/
|
||||
https://www.dominionpost.com/feed/
|
||||
https://www.morganmessenger.com/feed/
|
||||
https://apps.wv.gov/portal/rss/limit/50
|
||||
https://isthmus.com/api/rss/content.rss
|
||||
https://www.wispolitics.com/feed/
|
||||
https://wiseye.org/feed/
|
||||
https://wisconsinexaminer.com/feed/
|
||||
https://trib.com/search/?f=rss&t=article&c=news/state-and-regional&l=50&s=start_time&sd=desc
|
||||
https://wyofile.com/feed/
|
||||
https://www.wyomingnews.com/search/?f=rss&t=article&c=news&l=50&s=start_time&sd=desc
|
||||
https://www.wyodaily.com/rss
|
||||
https://www.cbs17.com/news/north-carolina-news/feed/
|
||||
https://abc11.com/feed/
|
||||
https://www.wnct.com/news/north-carolina/feed/
|
||||
https://www.usnews.com/rss/news/north-carolina
|
||||
https://indyweek.com/feed/
|
||||
https://portcitydaily.com/feed/
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue