I came across a problem in a book which I can’t seem to figure out. Unfortunately, I don’t have a live link for it, so if anyone could help me with my approach to this theoretically, I’d really appreciate it.
The process:
- I get from a fetch call an array of string codes (
["abcde", "fghij", "klmno", "pqrst"]). - I want to make a call to a link with each string code. example:
fetch('http://my-url/abcde').then(res => res.json()).then(res => res).catch(error => new Error(`Error: ${error}`)); // result: 12345fetch('http://my-url/fghij').then(res => res.json()).then(res => res).catch(error => new Error(`Error: ${error}`)); // result: 67891etc- Each of the calls is going to give me a number code, as shown.
- I need to get the highest number of the 5 and get its afferent string code and make another call with that.
“abcde” => 1234
“fghij” => 5314
“klmno” => 3465
“pqrst” => 7234 <— winner
fetch('http://my-url/pqrst').then(res => res.json()).then(res => res).catch(error => new Error(`Error: ${error}`));What I tried:
let codesArr = []; // array of string codeslet promiseArr = []; // array of fetch using each string code in `codesArr`, meant to be used in Promise.all()let codesObj = {}; // object with string code and its afferent number code gotten from the Promise.all()fetch('http://my-url/some-code').then(res => res.json()).then(res => codesArr = res) // now `codesArr` is ["abcde", "fghij", "klmno", "pqrst"].catch(error => new Error(`Error: ${error}`);for(let i = 0; i < codesArr.length; i++) { promiseArr.push( fetch(`http://my-url/${codesArr[i]}`) .then(res => res.text()) .then(res => { codesObj[codesArr[i]] = res; // This is to get an object from which I can later get the highest number and its string code. Like this: // codesObj = { // "abcde": 12345, // "fghij": 67891 // } }) .catch(error => new Error(`Error: ${error}`)); // I am trying to make an array with fetch, so that I can use it later in Promise.all()}Promise.all(promiseArray) // I wanted this to go through all the fetches inside the `promiseArr` and return all of the results at once. .then(res => { for(let i = 0; i < res.length; i++) { console.log(res[i]); // this should output the code number for each call (`12345`, `67891`...etc) // this is where I get lost } })- One of the problems with my approach so far seems to be that it makes too many requests and I get 429 error. I sometimes get the number codes all right, but not too often.
Advertisement
Answer
Like you already found out the 429 means that you send too many requests:
429 Too Many Requests
The user has sent too many requests in a given amount of time (“rate limiting”).
The response representations SHOULD include details explaining the condition, and MAY include a Retry-After header indicating how long to wait before making a new request.
For example:
JavaScript115151HTTP/1.1 429 Too Many Requests2Content-Type: text/html3Retry-After: 360045<html>6<head>7<title>Too Many Requests</title>8</head>9<body>10<h1>Too Many Requests</h1>11<p>I only allow 50 requests per hour to this Web site per12logged in user. Try again soon.</p>13</body>14</html>15Note that this specification does not define how the origin server identifies the user, nor how it counts requests. For example, an origin server that is limiting request rates can do so based upon counts of requests on a per-resource basis, across the entire server, or even among a set of servers. Likewise, it might identify the user by its authentication credentials, or a stateful cookie.
Responses with the 429 status code MUST NOT be stored by a cache.
To handle this issue you should reduce the amount of requests made in a set amount of time. You should iterate your codes with a delay, spacing out the request by a few seconds. If not specified in the API documentation or the 429 response, you have to use trial and error approach to find a delay that works. In the example below I’ve spaced them out 2 seconds (2000 milliseconds).
The can be done by using the setTimeout() to execute some piece of code later, combine this with a Promise to create a sleep function. When iterating the initially returned array, make sure to await sleep(2000) to create a 2 second delay between each iteration.
An example could be:
const fetch = createFetchMock({ "/some-code": ["abcde", "fghij", "klmno", "pqrst"], "/abcde": 12345, "/fghij": 67891, "/klmno": 23456, "/pqrst": 78912,});const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));(async function () { try { const url = "https://my-url/some-code"; console.log("fetching url", url); const response = await fetch(url); const codes = await response.json(); console.log("got", codes); const codesObj = {}; for (const code of codes) { await sleep(2000); const url = `https://my-url/${code}`; console.log("fetching url", url); const response = await fetch(url); const value = await response.json(); console.log("got", value); codesObj[code] = value; } console.log("codesObj =", codesObj); } catch (error) { console.error(error); }})();// fetch mocker factoryfunction createFetchMock(dataByPath = {}) { const empty = new Blob([], {type: "text/plain"}); const status = { ok: { status: 200, statusText: "OK" }, notFound: { status: 404, statusText: "Not Found" }, }; const blobByPath = Object.create(null); for (const path in dataByPath) { const json = JSON.stringify(dataByPath[path]); blobByPath[path] = new Blob([json], { type: "application/json" }); } return function (url) { const path = new URL(url).pathname; const response = (path in blobByPath) ? new Response(blobByPath[path], status.ok) : new Response(empty, status.notFound); return Promise.resolve(response); };}