我正在尝试将Python库报纸与Wayback Machine的归档文件一起使用,该文件存储了已归档网站的旧版本。从理论上讲,旧新闻文章可以从这些档案库中查询和下载。
例如,以下代码查询存档中CNBC的特定存档日期。
import newspaper
url = 'http://web.archive.org/web/20161201123529/http://www.cnbc.com/'
paper = newspaper.build(url, memoize_articles = False )
尽管已归档的网站本身包含指向2016年12月1日起的实际新闻文章的链接,但报纸模块似乎并没有接收这些文章。相反,你将获得诸如以下的URL:
https://blog.archive.org/2016/10/23/defining-web-pages-web-sites-and-web-captures/
并非来自此CNBC存档版本的实际文章。但是,报纸可以与今天的CNBC版本配合使用。
我想由于url的格式(包含两个http
s)而感到困惑。有人对从Wayback Machine档案中提取文章有任何建议吗?
这是一个有趣的问题,我将其添加到GitHub上的“报纸使用概述”文档中。
我尝试使用报纸.build,但无法使其正常工作,因此我使用了报纸Source。
from time import sleep
from random import randint
from newspaper import Config
from newspaper import Source
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0'
config = Config()
config.browser_user_agent = USER_AGENT
config.request_timeout = 10
wayback_cnbc = Source(url='https://web.archive.org/web/20180301012621/https://www.cnbc.com/', config=config,
memoize_articles=False, language='en', number_threads=20, thread_timeout_seconds=2)
wayback_cnbc.build()
for article_extract in wayback_cnbc.articles:
article_extract.download()
article_extract.parse()
print(article_extract.publish_date)
print(article_extract.title)
print(article_extract.url)
print('')
# this sleep timer is helping with some timeout issues
# that were happening when querying
sleep(randint(1,3))
上面的示例输出以下内容:
None
Media
https://web.archive.org/web/20180301012621/https://www.cnbc.com/media/
None
CNBC Video
https://web.archive.org/web/20180301012621/https://www.cnbc.com/video/
2017-11-08 00:00:00
CNBC Healthy Returns
https://web.archive.org/web/20180301012621/https://www.cnbc.com/2017/11/08/healthy-returns.html
2018-02-28 00:00:00
Markets in Asia decline as dollar steadies; Nikkei falls 307 points
https://web.archive.org/web/20180301012621/https://www.cnbc.com/2018/02/28/asia-markets-stocks-dollar-and-china-caixin-pmi-in-focus.html
2018-02-28 00:00:00
S&P 500 rises, but on track to snap longest monthly win streak since 1959
https://web.archive.org/web/20180301012621/https://www.cnbc.com/2018/02/28/us-stocks-interest-rates-fed-markets.html
希望此答案对你查询WayBack Machine文章的用例有所帮助。如果你有任何疑问,请告诉我。