Warm tip: This article is reproduced from serverfault.com, please click

python- Scraping Scraping Tableau公共 dashboard 时遇到问题

(python - Trouble scraping Tableau public dashboard)

发布于 2020-12-04 02:00:37

我想 scrape Tableau dashboard ,但是遇到一个问题,我在输出中缺少值。具体来说,似乎我的代码不会 scrape /打印重复的值(显示两次的值只会 scrape /打印一次)。

这是我正在使用的代码:

import requests
from bs4 import BeautifulSoup
import json
import re

r = requests.get("https://public.tableau.com/views/COVID-19HospitalsDashboard/Hospitals?%3Aembed=y&%3AshowVizHome=no", 
    params = {
    ":embed": "y",
    ":showVizHome": "no",
    ":host_url": "https://public.tableau.com/",
    ":embed_code_version": 3,
    ":tabs": "no",
    ":toolbar": "no",
    ":animate_transition": "yes",
    ":display_static_image": "no",
    ":display_spinner": "no",
    ":display_overlay": "yes",
    ":display_count": "yes",
    ":language": "en",
    ":loadOrderID": 0
})
soup = BeautifulSoup(r.text, "html.parser")

tableauData = json.loads(soup.find("textarea",{"id": "tsConfigContainer"}).text)

dataUrl = f'https://public.tableau.com{tableauData["vizql_root"]}/bootstrapSession/sessions/{tableauData["sessionid"]}'

r = requests.post(dataUrl, data= {
    "sheet_id": tableauData["sheetId"],
})
dataReg = re.search('\d+;({.*})\d+;({.*})', r.text, re.MULTILINE)
info = json.loads(dataReg.group(1))
data = json.loads(dataReg.group(2))

print(data["secondaryInfo"]["presModelMap"]["dataDictionary"]["presModelHolder"]["genDataDictionaryPresModel"]["dataSegments"]["0"]["dataColumns"])
Questioner
Henry St
Viewed
0
groseries 2020-12-14 23:26:17

我已经浏览了你的原始请求文本和正则表达式搜索数据,但找不到你正在讨论的内容。来自请求和你自己的正则表达式搜索的原始数据中的两个点都使用以下命令返回1132个匹配项:

x = data["secondaryInfo"]["presModelMap"]["dataDictionary"]["presModelHolder"]["genDataDictionaryPresModel"]["dataSegments"]["0"]["dataColumns"][0]["dataValues"]

duplicates = set()
# loop through elements and find matches
for i in x:
    if i not in duplicates:
        duplicates.add(i)

print(len(duplicates))

由此看来,你的代码可以正常工作。