Warm tip: This article is reproduced from serverfault.com, please click

scraping table with python based on dates

发布于 2020-11-29 03:19:35

since a week ago i have been trying to scrape a table from this site https://www.bi.go.id/id/moneter/informasi-kurs/transaksi-bi/Default.aspx but i dont have an idea what to write,i am very confused. iam trying to scrape table of kurs transaction from 2015-2020(20 nov 2015-20 nov 2020, enter image description here but the problem is the link between the default date and the date that I chose is still the same.please help me in any way,Thank you before !

import requests
from bs4 import BeautifulSoup
import pandas as pd
headers={
"User-Agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36",
"X-Requested-With":"XMLHttpRequest"
}
url = "https://www.bi.go.id/id/moneter/informasi-kurs/transaksi-bi/Default.aspx"
import requests
from lxml import html
response = requests.get(url)
content= response.content
print(content)
Questioner
adinda aulia
Viewed
0
gosalia 2020-11-29 23:29:39

You need to use Selenium. You can install Selenium and then you can install a driver. I use chrome and then once you install it make a note of that path and set your DRIVER_PATH to the location

In the code below what i do is basically request the link you posted and then I enter the dates which you can change. Finally i click on the submit button. That generates the table within the date range. Now you can write follow up code to scrape the information from the table.

Code

import requests
from selenium import webdriver

DRIVER_PATH = 'Yourpath/chromedriver'
driver = webdriver.Chrome(executable_path=DRIVER_PATH)
driver.get('https://www.bi.go.id/id/moneter/informasi-kurs/transaksi-bi/Default.aspx')
start_date = driver.find_element_by_id("ctl00_PlaceHolderMain_biWebKursTransaksiBI_txtFrom")
start_date.send_keys("15-Nov-20")
end_date = driver.find_element_by_id("ctl00_PlaceHolderMain_biWebKursTransaksiBI_txtTo")
end_date.send_keys("20-Nov-20")
submit_button = driver.find_element_by_id("ctl00_PlaceHolderMain_biWebKursTransaksiBI_btnSearch1").click()