HTTP Request Throttling Not working

I am mapping over a list of 600 URLs and then making Http GET requests. currently the server kicks me out because of making too many requests.

I googled and found this library GitHub - vercel/async-sema: Semaphore using `async` and `await` it allows to throttle the get requests from fetch.

The example which they have on their site is

const { RateLimit } = require('async-sema');

async function f() {
  const lim = RateLimit(5); // rps

  for (let i = 0; i < n; i++) {
    await lim();
    // ... do something async
  }
}

I translated it to rescript like this

open Js.Promise2
open Belt.Array
let urlList = ["a", "b"]
@module("async-sema")
external rateLimit: int => promise<unit> = "RateLimit"
urlList
  -> map(url => 
           rateLimit(1) 
           -> then(_ => url -> Fetch.get -> then(Response.json))
      )

My code compiles and runs (yay!) but it doesn’t throttle anything and all requests are made at the same time. While the issue could be in the library itself and I can log an issue at their GitHub site. I want to know did I translate the javascript code correctly?

I think you have to await the same limiter during the loop. Your code will create a new limiter on each cycle. Here are 2 working examples:

@module("async-sema")
external rateLimit: int => (. unit) => promise<unit> = "RateLimit"

let urlList = ["a", "b"]

let limiter = rateLimit(1)

// for loop
let test = async () => {
  for i in 0 to Array.length(urlList) - 1 {
    await limiter(.)

    Js.log(urlList[i])
  }
}

let _ = test()

// or with map
let _ = Js.Array2.map(urlList, async url => {
  await limiter(.)

  Js.log(url)
})

2 Likes

Thanks @dkirchhof I am not clear about two portions of the code

  1. what does this mean β€œ(. unit)”.
  2. limiter(.) So you are calling the limiter function here. but what is the β€œ.” being passed?

It declares the function as an incurried one. It was my first idea to fix the problem and it worked. Otherwise the compiler unlines the function call.

1 Like