Promise.allSettled stuck forever

I use Promise.allSettled to fetch websites in batch fashion. It works fine if I limit size of website list to 10 but it become stuck as soon as I increase it to 1000.

This is weird and never happened to me. I’ve waited for three days until the script finish but it’s still stuck at the first 1000 items.

const rp = require('request-promise');
const lineReader = require('line-by-line');

const reader = new lineReader("./all.json");

let lines = [];
const limit = 1000;

let successCount = 0;

reader.on('line', async (line) => {
    line = JSON.parse(line);

    lines.push(line);

    if (lines.length === limit) {
        reader.pause();

        let promises = [];

        for (const line of lines) {
            promises.push(rp(line.website))
        }
    
        await Promise.allSettled(promises);

        successCount++
        console.log(`Success Count is ${successCount}`);

        lines = [];

        reader.resume();
    }
});

The data.json file has website list under following format,

{ website: "https://www.google.com" }
{ website: "https://www.bing.com" }
{ website: "https://www.microsoft.com" }

You can reproduce the behavior by duplicating a line like { website: "https://www.google.com" } 2000 times.

Answer

The solution is to wrap up my promise with a promise that timeout after certain period.

Here is the function that provides the ability timeout the promise,

const withTimeout = (millis, promise) => {
    const timeout = new Promise((resolve, reject) =>
        setTimeout(
            () => reject(`Timed out after ${millis} ms.`),
            millis));
    return Promise.race([
        promise,
        timeout
    ]);
};

You can use it like below,

await Promise.allSettled([
   withTimeout(5000, rp("https://www.google.com"),
   withTimeout(5000, rp("https://www.bing.com"),
]));

I know you can specify a timeout in the request-promise options, but for some reason it didn’t work for me, but I noticed certain improvements when I added it.