Warm tip: This article is reproduced from serverfault.com, please click

javascript-如何在Node.js的每次循环执行中正确添加

(javascript - How to (Properly) add in each execution of a loop in node.js)

发布于 2020-11-29 08:03:02

所以我想写一个谷歌解析器。我的工具的想法是,它接受搜索查询并在google中搜索它们并返回URL。到目前为止,它运行良好,但是现在我正在尝试设置页面配置并遇到麻烦,我的代码是:

const needle = require("needle") //for making get request
const sp = require("serp-parser") //for parsing data from the request
const queryup = "watch movies online free" //my search data
const query = encodeURI(queryup) //my search data so google can read it
var page = 0; //initializing the page counter
let pages = 5; //setting amount of pages to loop through
for (var i = 0; i < pages; i++) { //my loop
  needle.get(`https://www.google.com/search?q=${query}&start=${page}`, function(err, response){ //MY MAIN PROBLEM <<<--- The issue is its adding to the page value but its not effecting it here, why?
    page += 10 //adding to page value (every 10 page value is 1 extra page)
    console.log(`----- Page number: `+ page / 10+" -----") //logging the number of the page to confirm that it is indeed increasing the page value
    let results = response.body; //defining the body of my request
    parser = new sp.GoogleNojsSERP(results); //initializing the parser
    let parsed = parser.serp //parsing the body
    let objarray = parsed.organic; //parsed body (returns as an array of json objects)
    for (var i = 0; i < objarray.length; i++) { //loop the logging of each url
      let url = objarray[i].url //defining url
      console.log(url) //logging each url
    }
  });
}

没有十亿评论:

const needle = require("needle") 
const sp = require("serp-parser") 
const queryup = "watch movies online free"
const query = encodeURI(queryup) 
var page = 0;
let pages = 5; 
for (var i = 0; i < pages; i++) {
  needle.get(`https://www.google.com/search?q=${query}&start=${page}`, function(err, response){ 


    //^^^^^ MY MAIN PROBLEM <<<--- The issue is its adding to the page value but its not effecting it here, why?



    page += 10 
    console.log(`----- Page number: `+ page / 10+" -----")
    let results = response.body;
    parser = new sp.GoogleNojsSERP(results);
    let parsed = parser.serp
    let objarray = parsed.organic;
    for (var i = 0; i < objarray.length; i++) {
      let url = objarray[i].url
      console.log(url)
    }
  });
}
Questioner
djsnoob
Viewed
0
21.1k 2020-11-29 17:22:54

这似乎是异步的问题。我不熟悉needle,但是我知道外部查询基本上从不同步。

你遇到的问题基本上是,实际的Web查询是在循环首次运行后发生的,并且已经page增加到50。然后,构造了5个查询,每个查询的页数为50,因为异步非常复杂且难以管理。

实际上,引擎实际上是在先执行它可能做的所有其他事情,然后再执行你的Web查询。

通过needlenpm docs进行的一次旅行告诉我,你可以使用其他语法来代替needle返回一个promise,然后可以将其包装在一个异步函数中并进行管理await以强制执行同步行为,这是你要执行的操作:

const needle = require('needle');
const sp = require('serp-parser');
const queryup = 'watch movies online free';
const query = encodeURI(queryup);
let page = 0;
const pages = 5;

const googler = async function () {
  for (let i = 0; i < pages; i++) {
    try {
      const response = await needle('get', `https://www.google.com/search?q=${query}&start=${page}`);// MY MAIN PROBLEM <<<--- The issue is its adding to the page value but its not effecting it here, why?
      console.log('----- Page number: ' + page / 10 + ' -----');
      const results = await response.body;
      const parser = new sp.GoogleNojsSERP(results);
      const parsed = parser.serp;
      const objarray = parsed.organic;
      for (let i = 0; i < objarray.length; i++) {
        const url = objarray[i].url;
        console.log(url);
      }
    } catch (err) {
      console.error(err);
    }
    page += 10;
  }
};

googler();

关键区别:

  1. 根据needle文档,不是将request方法作为needle对象上的方法,而是将其直接传递给needle作为函数调用自身的第一个参数
  2. 当你使用来管理诺言时await,被拒绝的诺言会引发应由传统try / catch块捕获的错误。我在这里做了。虽然,如果needle是这样的node-fetch话,基本上可能不会抛出错误,但这是一个好习惯。

我的扩展程序之一自动将你的var声明更改let未分配的let声明更改为const欢迎你将其改回。