Mikhail Gerasimov
2019-06-11 00:58:01
1) Create a coroutine to make request asynchronously:
import asyncio
import aiohttp
async def get_text(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as resp:
return await resp.text()
2) Replace all synchronious requests with awaiting for this coroutine, making outer functions coroutines also:
async def field_info(field_link): # async - to make outer function coroutine
text = await get_text(field_link) # await - to get result from async funcion
soup = BeautifulSoup(text, 'html.parser')
3) Make outer code to do jobs concurrently using asyncio.gather():
async def main():
links = [link1, link2, link3]
scraped_info = asyncio.gather(*[
field_info(link)
for link
in links
]) # do multiple field_info coroutines concurrently (parallely)
4) Pass top-level coroutine to asyncio.run():
asyncio.run(main())
热门帖子
热门github
6
Thank you for your step by step answer, it really helped me to understand how this stuff is working.
@paskh you're welcome! You may be also interested in reading this answer - it's about how
asyncioworks and how to use it in general.