Warm tip: This article is reproduced from serverfault.com, please click
ajax python scrapy screen-scraping web-scraping

Scraping ajax pages using python

发布于 2013-05-05 23:38:57

I've already seen this question about scraping ajax, but python isn't mentioned there. I considered using scrapy, i believe they have some docs on that subject, but as you can see the website is down. So i don't know what to do. I want to do the following:

I only have one url, example.com you go from page to page by clicking submit, the url doesn't change since they're using ajax to display the content. I want to scrape the content of each page, how to do it?

Lets say that i want to scrape only the numbers, is there anything other than scrapy that would do it? If not, would you give me a snippet on how to do it, just because their website is down so i can't reach the docs.

Questioner
Lynob
Viewed
0
分享
♦ 2017-05-23 19:47:26

First of all, scrapy docs are available at https://scrapy.readthedocs.org/en/latest/.

Speaking about handling ajax while web scraping. Basically, the idea is rather simple:

  • open browser developer tools, network tab
  • go to the target site
  • click submit button and see what XHR request is going to the server
  • simulate this XHR request in your spider

Also see:

Hope that helps.

Lynob 2013-05-06 18:06:50

i was referring to this url blog.scrapy.org/scraping-ajax-sites-with-scrapy‎ which is no longer available, thanks for reminding me of readthedocs.com

alecxe 2013-05-06 18:23:47

Got it. If you have problems with the spider implementation, consider posting another question with what url are you trying to crawl, what button to click etc. Happy scraping!

2upmedia 2015-12-17 20:57:05

@Lynob here's the URL you're talking about: web.archive.org/web/20130525095330/http://blog.scrapy.org/…

热门帖子

1
google doc如何快速插入日期时间？
2
求一个 Spotify 长期车
3
C++新手，求助一个关于怎么使用第三方库的问题
4
西安出个联通千兆宽带
5
关于英语学习的重要性的思考
6
出美版 iPhone 13promax， V 友 4600 吧
7
这里分享一个免费的在线 PDF 总结工具： NoteGPT
8
没想到 Arc 浏览器对网络要求如此严格
9
国内用户 android 手机如何使用 chatgpt 客户端
10
深陷消费主义陷阱的背后，是我空洞的灵魂

热门github

1
A multi-platform library for OpenGL, OpenGL ES, Vulkan, window and input
2
Dev tool that writes scalable apps from scratch while the developer oversees the implementation
3
shadcn/ui, but for Svelte. ✨
4
The Python Risk Identification Tool for generative AI (PyRIT) is an open access automation framework to empower security professionals and machine learning engineers to proactively find risks in their generative AI systems.
5
Performance-portable, length-agnostic SIMD with runtime dispatch
6
ZK Credo
7
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
8
Joplin - the secure note taking and to-do app with synchronisation capabilities for Windows, macOS, Linux, Android and iOS.
9
Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of Transformers. It is based on the line of progress on structured state space models, with an efficient hardware-aware design and implementation in the spirit of FlashAttention.
10
This repository contains System Design resources which are useful while preparing for interviews and learning Distributed Systems
11
Curso para aprender el lenguaje de programación Python desde cero y para principiantes. 75 clases, 37 horas en vídeo, código, proyectos y grupo de chat. Fundamentos, frontend, backend, testing, IA...
12
🎓 Path to a free self-taught education in Computer Science!
13
1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java
14
A collective list of free APIs
15
📚 Freely available programming books