Warm tip: This article is reproduced from serverfault.com, please click
api python-3.x request

python 3.x-从其URL再现对API的查询

(python 3.x - Reproducing a query to an API from its URL)

发布于 2020-11-30 15:43:26

我想将查询重现到我知道Python的URL的API，以便能够修改ProductID（P20201，P618001 ...），以便可以返回它们各自的信息。

查询是：

https://api.bazaarvoice.com/data/batch.json?passkey=iohrnzjadededr160osgfvimy&apiversion=5.5&displaycode=3232-fr_fr&resource.q0=products&filter.q0=id%3Aeq%3AP618001&stats.q0=questions%2Creviews&filteredstats.q0=questions%2Creviews&filter_questions.q0=contentlocale%3Aeq%3Afr_FR&filter_answers.q0=contentlocale%3Aeq%3Afr_FR&filter_reviews.q0=contentlocale%3Aeq%3Afr_FR&filter_reviewcomments.q0=contentlocale%3Aeq%3Afr_FR&resource.q1=questions&filter.q1=productid%3Aeq%3AP618001&filter.q1=contentlocale%3Aeq%3Afr_FR&sort.q1=lastapprovedanswersubmissiontime%3Adesc&stats.q1=questions&filteredstats.q1=questions&include.q1=authors%2Cproducts%2Canswers&filter_questions.q1=contentlocale%3Aeq%3Afr_FR&filter_answers.q1=contentlocale%3Aeq%3Afr_FR&limit.q1=10&offset.q1=0&limit_answers.q1=10&resource.q2=reviews&filter.q2=isratingsonly%3Aeq%3Afalse&filter.q2=productid%3Aeq%3AP618001&filter.q2=contentlocale%3Aeq%3Afr_FR&sort.q2=submissiontime%3Adesc&stats.q2=reviews&filteredstats.q2=reviews&include.q2=authors%2Cproducts%2Ccomments&filter_reviews.q2=contentlocale%3Aeq%3Afr_FR&filter_reviewcomments.q2=contentlocale%3Aeq%3Afr_FR&filter_comments.q2=contentlocale%3Aeq%3Afr_FR&limit.q2=5&offset.q2=0&limit_comments.q2=3&callback=BV._internal.dataHandler0

因此，我尝试通过以下方式在Scrapy中做到这一点：

C:\Users\antoi>python
Python 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 13:35:33) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> params = {'passkey': ['iohrnzjadededr160osgfvimy'], 'apiversion': ['5.5'], 'displaycode': ['3232-fr_fr'],
...               'resource.q0': ['products'], 'filter.q0': ['id:eq:P618001'], 'stats. q0': ['questions,reviews'],
...               'filteredstats.q0': ['questions,reviews'], 'filter_questions.q0': ['contentlocale:eq:fr_FR'],
...               'filter_answers.q0': ['contentlocale:eq:fr_ FR'], 'filter_reviews.q0': ['contentlocale:eq:fr_FR'],
...               'filter_reviewcomments.q0': ['contentlocale:eq:fr_FR'], 'resource.q1': ['questions'],
...               'filter.q1': ['productid :eq:P618001', 'contentlocale:eq:fr_FR'],
...               'sort.q1': ['lastapprovedanswersubmissiontime:desc'],
...               'stats.q1': ['questions'], 'filteredstats.q1': ['questions'],
...               'include .q1': ['authors,products,answers'], 'filter_questions.q1': ['contentlocale:eq:fr_FR'],
...               'filter_answers.q1': ['contentlocale:eq:fr_FR'], 'limit.q1': ['10'],
...               'offset.q 1': ['0'], 'limit_answers.q1': ['10'], 'resource.q2': ['reviews'],
...               'filter.q2': ['isratingsonly:eq:false', 'productid:eq:P618001',
...                             'contentlocale:eq:fr_FR'], 'sort.q 2': ['submissiontime:desc'],
...               'stats.q2': ['reviews'], 'filteredstats.q2': ['reviews'], 'include.q2': ['authors,products,comments'],
...               'filter_reviews.q2': ['contentloc ale:eq:fr_FR'], 'filter_reviewcomments.q2': ['contentlocale:eq:fr_FR'],
...               'filter_comments.q2': ['contentlocale:eq:fr_FR'], 'limit.q2': ['5'], 'offset.q2': ['0'],
...               'lim it_comments.q2': ['3'], 'callback': ['BV._internal.dataHandler0']}
>>> product_id = 'P618001'
>>> params['filter.q0'] = 'id:eq:' + product_id
>>> params['filter.q1'][0] = 'productid :eq:' + product_id
>>> params['filter.q2'][1] = 'productid :eq:' + product_id
>>> perfume = requests.get("https://api.bazaarvoice.com/data/batch.json", params=self.params)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'self' is not defined
>>> perfume = requests.get("https://api.bazaarvoice.com/data/batch.json", params=params)

但它返回json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)：

>>> print("perfume.json(): ", perfume.json())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\antoi\AppData\Roaming\Python\Python36\site-packages\requests\models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Python36\lib\site-packages\simplejson\__init__.py", line 518, in loads
    return _default_decoder.decode(s)
  File "C:\Python36\lib\site-packages\simplejson\decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "C:\Python36\lib\site-packages\simplejson\decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

那么，如何从URL复制对API的查询呢？

Questioner
Revolucion for Monica
Viewed
11
分享
baduker 2020-12-01 21:52:57

如果请求有效，但为什么要重新创建它，但得到的不是JSON，至少最初不是这样。你需要先将其挖出，然后才能获取数据。

看到这个：

import json
import re

import requests

response = requests.get("https://api.bazaarvoice.com/data/batch.json?passkey=iohrnzjadededr160osgfvimy&apiversion=5.5&displaycode=3232-fr_fr&resource.q0=products&filter.q0=id%3Aeq%3AP618001&stats.q0=questions%2Creviews&filteredstats.q0=questions%2Creviews&filter_questions.q0=contentlocale%3Aeq%3Afr_FR&filter_answers.q0=contentlocale%3Aeq%3Afr_FR&filter_reviews.q0=contentlocale%3Aeq%3Afr_FR&filter_reviewcomments.q0=contentlocale%3Aeq%3Afr_FR&resource.q1=questions&filter.q1=productid%3Aeq%3AP618001&filter.q1=contentlocale%3Aeq%3Afr_FR&sort.q1=lastapprovedanswersubmissiontime%3Adesc&stats.q1=questions&filteredstats.q1=questions&include.q1=authors%2Cproducts%2Canswers&filter_questions.q1=contentlocale%3Aeq%3Afr_FR&filter_answers.q1=contentlocale%3Aeq%3Afr_FR&limit.q1=10&offset.q1=0&limit_answers.q1=10&resource.q2=reviews&filter.q2=isratingsonly%3Aeq%3Afalse&filter.q2=productid%3Aeq%3AP618001&filter.q2=contentlocale%3Aeq%3Afr_FR&sort.q2=submissiontime%3Adesc&stats.q2=reviews&filteredstats.q2=reviews&include.q2=authors%2Cproducts%2Ccomments&filter_reviews.q2=contentlocale%3Aeq%3Afr_FR&filter_reviewcomments.q2=contentlocale%3Aeq%3Afr_FR&filter_comments.q2=contentlocale%3Aeq%3Afr_FR&limit.q2=5&offset.q2=0&limit_comments.q2=3&callback=BV._internal.dataHandler0").text

data = json.loads(re.search(r"\((.*)\)", response).group(1))
print(data["BatchedResults"]["q2"]["Results"][0])

输出：

{'Id': '158638580', 'CID': '9b474509-239b-5f43-9926-6ca76192c563', 'SourceClient': 'sephora-fr', 'Badges': {'loyaltyYes--Im-a-beauty-insider': {'ContentType': 'REVIEW', 'Id': 'loyaltyYes--Im-a-beauty-insider', 'BadgeType': 'Custom'}}, 'BadgesOrder': ['loyaltyYes--Im-a-beauty-insider'], 'LastModeratedTime': '2020-06-24T11:45:09.000+00:00', 'LastModificationTime': '2020-07-13T16:40:52.000+00:00', 'ProductId': 'P618001', 'CampaignId': 'BV_PIE_ONLINE', 'ContextDataValuesOrder': ['Gender', 'Age', 'Eyes', 'Skin'], 'UserLocation': 'Clermont Ferrand', 'AuthorId': '72416110', 'ContentLocale': 'fr_FR', 'IsFeatured': False, 'TotalInappropriateFeedbackCount': 0, 'TotalClientResponseCount': 0, 'TotalCommentCount': 0, 'Rating': 5, 'IsRatingsOnly': False, 'IsRecommended': True, 'Helpfulness': 1.0, 'TotalFeedbackCount': 1, 'TotalNegativeFeedbackCount': 0, 'TotalPositiveFeedbackCount': 1, 'ModerationStatus': 'APPROVED', 'SubmissionId': 'r23232-fr__159299794LEh4bHKjq', 'SubmissionTime': '2020-06-24T11:25:40.000+00:00', 'ReviewText': 'Très contente de mon achat. Je cherchais ce parfum depuis un temps en magasin et je suis heureuse qu’il soit disponible en ligne il sent tellement bon !! En plus en promo, génial ! \r\nLivraison très rapide !', 'Title': 'Satisfaite', 'UserNickname': 'oceaned03', 'ContextDataValues': {'Age': {'Value': '18to24', 'Id': 'Age'}, 'Gender': {'Value': 'Female', 'Id': 'Gender'}, 'Skin': {'Value': 'Seche', 'Id': 'Skin'}, 'Eyes': {'Value': 'Bleus', 'Id': 'Eyes'}}, 'RatingRange': 5, 'AdditionalFieldsOrder': [], 'ProductRecommendationIds': [], 'Cons': None, 'TagDimensionsOrder': [], 'CommentIds': [], 'TagDimensions': {}, 'Videos': [], 'Photos': [], 'ClientResponses': [], 'InappropriateFeedbackList': [], 'SecondaryRatings': {}, 'AdditionalFields': {}, 'SecondaryRatingsOrder': [], 'IsSyndicated': False, 'Pros': None}

如果要“解码” URL，请使用parse_qs：

from urllib.parse import parse_qs

api_url = "https://api.bazaarvoice.com/data/batch.json?passkey=iohrnzjadededr160osgfvimy&apiversion=5.5&displaycode=3232-fr_fr&resource.q0=products&filter.q0=id%3Aeq%3AP618001&stats.q0=questions%2Creviews&filteredstats.q0=questions%2Creviews&filter_questions.q0=contentlocale%3Aeq%3Afr_FR&filter_answers.q0=contentlocale%3Aeq%3Afr_FR&filter_reviews.q0=contentlocale%3Aeq%3Afr_FR&filter_reviewcomments.q0=contentlocale%3Aeq%3Afr_FR&resource.q1=questions&filter.q1=productid%3Aeq%3AP618001&filter.q1=contentlocale%3Aeq%3Afr_FR&sort.q1=lastapprovedanswersubmissiontime%3Adesc&stats.q1=questions&filteredstats.q1=questions&include.q1=authors%2Cproducts%2Canswers&filter_questions.q1=contentlocale%3Aeq%3Afr_FR&filter_answers.q1=contentlocale%3Aeq%3Afr_FR&limit.q1=10&offset.q1=0&limit_answers.q1=10&resource.q2=reviews&filter.q2=isratingsonly%3Aeq%3Afalse&filter.q2=productid%3Aeq%3AP618001&filter.q2=contentlocale%3Aeq%3Afr_FR&sort.q2=submissiontime%3Adesc&stats.q2=reviews&filteredstats.q2=reviews&include.q2=authors%2Cproducts%2Ccomments&filter_reviews.q2=contentlocale%3Aeq%3Afr_FR&filter_reviewcomments.q2=contentlocale%3Aeq%3Afr_FR&filter_comments.q2=contentlocale%3Aeq%3Afr_FR&limit.q2=5&offset.q2=0&limit_comments.q2=3&callback=BV._internal.dataHandler0"
print(parse_qs(api_url))

这给你一个命令：

{'https://api.bazaarvoice.com/data/batch.json?passkey': ['iohrnzjadededr160osgfvimy'], 'apiversion': ['5.5'], 'displaycode': ['3232-fr_fr'], 'resource.q0': ['products'], 'filter.q0': ['id:eq:P618001'], 'stats.q0': ['questions,reviews'], 'filteredstats.q0': ['questions,reviews'], 'filter_questions.q0': ['contentlocale:eq:fr_FR'], 'filter_answers.q0': ['contentlocale:eq:fr_FR'], 'filter_reviews.q0': ['contentlocale:eq:fr_FR'], 'filter_reviewcomments.q0': ['contentlocale:eq:fr_FR'], 'resource.q1': ['questions'], 'filter.q1': ['productid:eq:P618001', 'contentlocale:eq:fr_FR'], 'sort.q1': ['lastapprovedanswersubmissiontime:desc'], 'stats.q1': ['questions'], 'filteredstats.q1': ['questions'], 'include.q1': ['authors,products,answers'], 'filter_questions.q1': ['contentlocale:eq:fr_FR'], 'filter_answers.q1': ['contentlocale:eq:fr_FR'], 'limit.q1': ['10'], 'offset.q1': ['0'], 'limit_answers.q1': ['10'], 'resource.q2': ['reviews'], 'filter.q2': ['isratingsonly:eq:false', 'productid:eq:P618001', 'contentlocale:eq:fr_FR'], 'sort.q2': ['submissiontime:desc'], 'stats.q2': ['reviews'], 'filteredstats.q2': ['reviews'], 'include.q2': ['authors,products,comments'], 'filter_reviews.q2': ['contentlocale:eq:fr_FR'], 'filter_reviewcomments.q2': ['contentlocale:eq:fr_FR'], 'filter_comments.q2': ['contentlocale:eq:fr_FR'], 'limit.q2': ['5'], 'offset.q2': ['0'], 'limit_comments.q2': ['3'], 'callback': ['BV._internal.dataHandler0']}

如果你想尝试更改值，则可以尝试以下操作：

from urllib.parse import parse_qsl, urlencode
import json
import re

import requests

url = "https://api.bazaarvoice.com/data/batch.json?"
payload = "passkey=iohrnzjadededr160osgfvimy&apiversion=5.5&displaycode=3232-fr_fr&resource.q0=products&filter.q0=id%3Aeq%3AP618001&stats.q0=questions%2Creviews&filteredstats.q0=questions%2Creviews&filter_questions.q0=contentlocale%3Aeq%3Afr_FR&filter_answers.q0=contentlocale%3Aeq%3Afr_FR&filter_reviews.q0=contentlocale%3Aeq%3Afr_FR&filter_reviewcomments.q0=contentlocale%3Aeq%3Afr_FR&resource.q1=questions&filter.q1=productid%3Aeq%3AP618001&filter.q1=contentlocale%3Aeq%3Afr_FR&sort.q1=lastapprovedanswersubmissiontime%3Adesc&stats.q1=questions&filteredstats.q1=questions&include.q1=authors%2Cproducts%2Canswers&filter_questions.q1=contentlocale%3Aeq%3Afr_FR&filter_answers.q1=contentlocale%3Aeq%3Afr_FR&limit.q1=10&offset.q1=0&limit_answers.q1=10&resource.q2=reviews&filter.q2=isratingsonly%3Aeq%3Afalse&filter.q2=productid%3Aeq%3AP618001&filter.q2=contentlocale%3Aeq%3Afr_FR&sort.q2=submissiontime%3Adesc&stats.q2=reviews&filteredstats.q2=reviews&include.q2=authors%2Cproducts%2Ccomments&filter_reviews.q2=contentlocale%3Aeq%3Afr_FR&filter_reviewcomments.q2=contentlocale%3Aeq%3Afr_FR&filter_comments.q2=contentlocale%3Aeq%3Afr_FR&limit.q2=5&offset.q2=0&limit_comments.q2=3&callback=BV._internal.dataHandler0"

decoded = parse_qsl(payload)
decoded.pop(4)
decoded.insert(4, ('filter.q0', 'id:eq:P20201'))

response = requests.get(f"{url}{urlencode(decoded)}").text

data = json.loads(re.search(r"\((.*)\)", response).group(1))
print(data["BatchedResults"]["q2"]["Results"][0])
Revolucion for Monica 2020-12-01 12:03:38

感谢您的回答 ！我需要重新创建它才能修改product_id。这里我P618001以示例为例，但我希望能够与其他类似P20201一起进行测试...

baduker 2020-12-01 12:20:21

@RevolucionforMonica我已经更新了答案。

Revolucion for Monica 2020-12-01 13:38:03

感谢那。不幸的是，尽管进行了修改，但它始终会处理相同的答案decoded.insert(4, ('filter.q0', 'id:eq:P2020'))

baduker 2020-12-01 13:52:42

那里有错字。我放了，P2020但你给了P20201。我已经解决了。

Revolucion for Monica 2020-12-01 13:54:00

是的，但我只注意到我不仅需要修改'filter.q0'，但也'filter.q1'[0]和'filter.q2'[1]也

热门帖子

1
推荐一些好玩的/大众的手游
2
求指教后端项目迁移方案
3
迷你洗衣机是不是都是智商税？
4
求助一个排查了半年没解决的 MySQL order by 子句导致索引失效的问题， 500 多万条记录的小表要查快两分钟
5
个人开发了一款 WordPress 主题： iPao，集成了 AI 总结功能
6
偶然发现奇游加速器会在系统里植入根证书
7
国内有蒲公英替代品推荐吗？
8
语音助手这个东西真的会监听谈话并且上传，从而泄漏隐私吗？
9
出一些有意思的域名-明盘
10
jetbrains 全家桶升级 2024 后，在滚动代码时候感觉有点掉帧

热门github

1
Implementation of paper - YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information (翻译：论文实现——YOLOv9: Learning What You Want to Learn using Programmable Gradient Information（利用可编程梯度信息学习您想学的内容）)
2
A Windows and Office activator using HWID / Ohook / KMS38 / Online KMS activation methods, with a focus on open-source code and fewer antivirus detections. (翻译：使用 HWID / KMS38 / Online KMS 激活方法激活 Microsoft 产品的脚本集合，重点是开源代码、较少的防病毒检测和用户友好性。)
3
Get up and running with Llama 2, Mistral, Gemma, and other large language models. (翻译：启动并运行 Llama 2、Mistral、Gemma 和其他大型语言模型。)
4
该项目可以让你通过订阅的方式使用Cloudflare WARP+，自动获取流量。This project enables you to use Cloudflare WARP+ through subscription, automatically acquiring traffic.
5
Multi functional app to find duplicates, empty folders, similar images etc. (翻译：多功能应用程序可查找重复项、空文件夹、相似图像等。)
6
Xray panel supporting multi-protocol multi-user expire day & traffic & ip limit (Vmess & Vless & Trojan & ShadowSocks & Wireguard) (翻译：Xray面板支持多协议多用户到期日&流量&IP限制（Vmess & Vless & Trojan & ShadowSocks & Wireguard)
7
The Free Software Media System (翻译：Jellyfin 是一个免费软件媒体系统，可让您控制管理和流式传输您的媒体。它是专有 Emby 和 Plex 的替代方案，通过多个应用程序从专用服务器向最终用户设备提供媒体。)
8
lightweight, standalone C++ inference engine for Google's Gemma models. (翻译：适用于 Google Gemma 模型的轻量级独立 C++ 推理引擎。)
9
📚 Freely available programming books (翻译：📚 免费提供的编程书籍)
10
A collective list of free APIs (翻译：免费 API 的集合列表)
11
1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java (翻译：十亿行挑战 —— 使用 Java 对文本文件中的 10 亿行数据进行聚合的有趣探索)
12
🎓 Path to a free self-taught education in Computer Science! (翻译：🎓计算机科学免费自学教程！)
13
Curso para aprender el lenguaje de programación Python desde cero y para principiantes. 75 clases, 37 horas en vídeo, código, proyectos y grupo de chat. Fundamentos, frontend, backend, testing, IA... (翻译：从零开始学习 Python 编程语言的课程，适合初学者)
14
This repository contains System Design resources which are useful while preparing for interviews and learning Distributed Systems (翻译：该存储库包含系统设计资源，在准备面试和学习分布式系统时非常有用)
15
Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of Transformers. It is based on the line of progress on structured state space models, with an efficient hardware-aware design and implementation in the spirit of FlashAttention. (翻译：Mamba 是一种新的状态空间模型架构，在信息密集型数据（例如语言建模）上显示出良好的性能，而之前的二次模型在 Transformers 方面存在不足。它基于结构化状态空间模型的进展，并本着FlashAttention的精神进行高效的硬件感知设计和实现。)