Node js how to run axios.get every 2 seconds?

I am kinda of a newbie to node js, Here is what i am trying to do: i am looping through a json file full of links of our website via the map function (around 3000 links), inside the loop i am doing a axios get for each link and getting the response status code(will do other things in the future). But i want to run the axios get only like every 2 seconds or 5 seconds otherwise i am overwhelming the webserver. I am trying to input async await but it’s still too fast and server is taking a hit (i am technically DDos-ing my own website). I put a SetTimeout around the axios but that doesn’t seem like it worked, since in the console results are printing way too fast. so the question is, how do i make each axios.get request wait every 2 seconds before running in the map loop?.

var axios = require('axios');
const fs = require('fs');
var statusCheck = 0;

var main = [];
let rawdata = fs.readFileSync('C:/Users/jay/Documents/crawl/filtered2.json');
let jsonParsed = JSON.parse(rawdata);

jsonParsed.map(async(line) => {

    var encodeLink = encodeURI(line.link);
    const response = await axios.get(encodeLink).catch((err) => {
        var Status_ErrorsCatchaxios = {
            "status Code": err.response.status ? err.response.status : "No status code available",
            "Page title:": $('title').text() ? $('title').text() : 'No title avaialble',
            "Original Link": encodeLink ? encodeLink : "No Original Link Available",
            "errorCode": err
        }
        main.push(Status_ErrorsCatchaxios)
    })
    try {
        console.log(response.status)

        statusCheck = statusCheck + 1;
        console.log("Link: ", statusCheck)
    } catch (error) {
        console.log(error)
    }
})

Answer

The [].map function doesn’t wait for your items to resolve, so your code is currently dispatching all the requests (as you said, around 3000) in parallel.

You can use for...of instead to only run one request at a time. For example:

async function makeRequests (lines) {
  for (const line of lines) {
    const encodedLink = encodeURI(line.link)
    const response = await axios.get(encodedLink)
    // ...your response handling code here...
  }
}

makeRequests(jsonParsed)

If you want to wait for 2s between each request, you can add this line of code inside your for...of loop:

await new Promise(resolve => setTimeout(resolve, 2000))

Better solution

The solution above works, but I assume your webserver can probably take more than one request at a time, so maybe the ideal scenario would be to limit your code to make only up to N requests in parallel at a given time. This way you don’t flood your server but you’re able to get your results faster than just doing one request at a time.

The bluebird NPM module allows you to do that with their Promise.map function.

This function receives your list of items as the first argument, a function that executes something and returns a promise for each item as the second argument, and an object with a concurrency key describing how many items you want to allow to be handled in parallel as the third argument.

Here’s how it could work:

const bluebird = require('bluebird')

async function makeRequests (lines) {
  await bluebird.map(
    lines,
    async (line) => {
      const encodedLink = encodeURI(line.link)
      const response = await axios.get(encodedLink)
      // ...your response handling code here...
    },
    { concurrency: 3 }
  )
}

makeRequests(jsonParsed)