HTTP Request Throttling Not working

1

I am mapping over a list of 600 URLs and then making Http GET requests. currently the server kicks me out because of making too many requests.

I googled and found this library GitHub - vercel/async-sema: Semaphore using `async` and `await` it allows to throttle the get requests from fetch.

The example which they have on their site is

const { RateLimit } = require('async-sema');

async function f() {
  const lim = RateLimit(5); // rps

  for (let i = 0; i < n; i++) {
    await lim();
    // ... do something async
  }
}

I translated it to rescript like this

open Js.Promise2
open Belt.Array
let urlList = ["a", "b"]
@module("async-sema")
external rateLimit: int => promise<unit> = "RateLimit"
urlList
  -> map(url => 
           rateLimit(1) 
           -> then(_ => url -> Fetch.get -> then(Response.json))
      )

My code compiles and runs (yay!) but it doesn’t throttle anything and all requests are made at the same time. While the issue could be in the library itself and I can log an issue at their GitHub site. I want to know did I translate the javascript code correctly?

2

I think you have to await the same limiter during the loop. Your code will create a new limiter on each cycle. Here are 2 working examples:

@module("async-sema")
external rateLimit: int => (. unit) => promise<unit> = "RateLimit"

let urlList = ["a", "b"]

let limiter = rateLimit(1)

// for loop
let test = async () => {
  for i in 0 to Array.length(urlList) - 1 {
    await limiter(.)

    Js.log(urlList[i])
  }
}

let _ = test()

// or with map
let _ = Js.Array2.map(urlList, async url => {
  await limiter(.)

  Js.log(url)
})
2 Likes
3

Thanks @dkirchhof I am not clear about two portions of the code

  1. what does this mean “(. unit)”.
  2. limiter(.) So you are calling the limiter function here. but what is the “.” being passed?
4

It declares the function as an incurried one. It was my first idea to fix the problem and it worked. Otherwise the compiler unlines the function call.

1 Like