No errors in the code, wrong indentation was causing the Urls to be not generated. I fixed the indentation. This is the working solution.
from requests_html import HTMLSession
def get_url(search_text):
session = HTMLSession()
template = 'https://www.amazon.com/s?k={}'
search_term = search_text.replace(' ', '+')
url = template.format(search_term)
url += '&pages={}'
products = []
for x in range(1, 21):
url_inc = url.format(x)
print(url_inc)
r = session.get(url_inc)
r.html.render(sleep=1)
items = r.html.find('h2')
for item in items:
product = item.text
print(product)
products.append(product)
return products
s_list = get_url('Laptop')
Output:-
https://www.amazon.com/s?k=Laptop&pages=1
Laptop 14 Inch, Intel Celeron Processor J3455, Quad-Core, Windows 10, 6GB RAM, 128GB SSD Storage, HD IPS Display, Touch Numeric Keypad, Webcam Baffle, Winbook 140, Space Grey
Acer Aspire 5 Slim Laptop, 15.6 inches Full HD IPS Display, AMD Ryzen 3 3200U, Vega 3 Graphics, 4GB DDR4, 128GB SSD, Backlit Keyboard, Windows 10 in S Mode, A515-43-R19L, Silver
Lenovo IdeaPad 3 14" Laptop, 14.0" FHD 1920 x 1080 Display, AMD Ryzen 5 3500U Processor, 8GB DDR4 RAM, 256GB SSD, AMD Radeon Vega 8 Graphics, Narrow Bezel, Windows 10, 81W0003QUS, Abyss Blue
Acer Nitro 5 Gaming Laptop, 9th Gen Intel Core i5-9300H, NVIDIA GeForce GTX 1650, 15.6" Full HD IPS Display, 8GB DDR4, 256GB NVMe SSD, Wi-Fi 6, Backlit Keyboard, Alexa Built-in, AN515-54-5812
Acer Aspire 5 Slim Laptop, 15.6 inches Full HD IPS Display, AMD Ryzen 3 3200U, Vega 3 Graphics, 4GB DDR4, 128GB SSD, Backlit Keyboard, Windows 10 in S Mode, A515-43-R19L, Silver
HP 15-dy1036nr 10th Gen Intel Core i5-1035G1, 15.6-Inch FHD Laptop, Natural Silver
Acer Chromebook Spin 311 Convertible Laptop, Intel Celeron N4020, 11.6" HD Touch, 4GB LPDDR4, 32GB eMMC, Gigabit Wi-Fi 5, Bluetooth 5.0, Google Chrome, CP311-2H-C679
Acer Predator Helios 300 Gaming Laptop, Intel i7-10750H, NVIDIA GeForce RTX 2060 6GB, 15.6" Full HD 144Hz 3ms IPS Display, 16GB Dual-Channel DDR4, 512GB NVMe SSD, Wi-Fi 6, RGB Keyboard, PH315-53-72XD
ASUS F512JA-AS34 VivoBook 15 Thin and Light Laptop, 15.6” FHD Display, Intel i3-1005G1 CPU, 8GB RAM, 128GB SSD, Backlit Keyboard, Fingerprint, Windows 10 Home in S Mode, Slate Gray
ASUS F512DA-EB51 VivoBook 15 Thin And Light Laptop, 15.6” Full HD, AMD Quad Core R5-3500U CPU, 8GB DDR4 RAM, 256GB PCIe SSD, AMD Radeon Vega 8 Graphics, Windows 10 Home,Slate Gray
ASUS ROG G531GT-BI7N6 15.6" FHD Gaming Laptop Computer, Intel Hexa-Core i7-9750H Up to 4.5GHz, 8GB DDR4, 512GB SSD, NVIDIA GeForce GTX 1650, 802.11ac WiFi, HDMI, USB 3.0, Windows 10
2019 ASUS ROG 15.6" FHD Gaming Laptop Computer, Intel Hexa-Core i7-9750H Up to 4.5GHz, 16GB DDR4, 1TB HDD + 512GB SSD, NVIDIA GeForce GTX 1650, 802.11ac WiFi, HDMI, USB 3.0, Windows 10
Acer Aspire 5 Slim Laptop, 15.6 inches Full HD IPS Display, AMD Ryzen 3 3200U, Vega 3 Graphics, 4GB DDR4, 128GB SSD, Backlit Keyboard, Windows 10 in S Mode, A515-43-R19L, Silver
MSI GL65 Leopard 10SFK-062 15.6" FHD 144Hz 3ms Thin Bezel Gaming Laptop Intel Core i7-10750H RTX2070 16GB 512GB NVMe SSD Win 10
HP 15-dy1036nr 10th Gen Intel Core i5-1035G1, 15.6-Inch FHD Laptop, Natural Silver
iProda Laptop, 14.1 Inch Notebook (Intel Core i3-6157U 2.4GHz, 8GB RAM, 256GB SSD, Windows 10 Professional) with 1080P FHD Display, Lightweight, Best for Work at Home
...
https://www.amazon.com/s?k=Laptop&pages=2 #Now second URL
It prints the URL from page=1 to 21. But it only prints the same xpath result 20 times, not 20 different results of xpath.
We need to use a different selector in that case.
@doghoney Updated answer.
Thanks for the help. But still having a problem with the nested for-loop. The outer loop iterates and executes 20 times and THEN the inner loop executes. How can I make the outer loop execute once -> Inner loop once -> outer loop next -> inner loop next ->... 20times?
Thanks for all the follow-ups! Greatly helped.