RFC: Monorepo support

Development
1

This post is to specify the possible monorepo structures that ReScript should support. As a first step, let’s discuss where the rescript binary should live or how we can assume where it is found by tooling such as rescript-vscode and other editors.

Binary path

The latest rescript-vscode release unfortunately broke resolving the rescript binary path for npm or yarn workspace projects (also called monorepos). While we at least also have a workaround with the new VSCode setting rescript.settings.binaryPath, I think standard workflows should still work out of the box.

To make this functionality more reliable in the future, let’s specify where the binary path(s) should be. Currently, it is installed into node_modules/.bin. Your monorepo setup might be more complex, so please comment, especially if you use some of the more sophisticated monorepo management tools like lerna, turborepo and so on.

Sidenote: The aforementioned bug has been fixed already, at least for the simple case of yarn workspaces.

8 Likes
2

Some things that came to my mind regarding the binary path lookup:

  • What if I have rescript installed globally and not in my project? Then I guess it will not be found and I need to specify the path manually?
  • The current implementation walks up the filesystem tree until it either finds a node_modules/.bin or reaches the filesystem root, right? I think it should not go up any further than the directory that the user opened in VS Code.
1 Like
3

Additionally, if support for multiple rescript versions in the same monorepo is a goal, what if packages have different versions specified?

I’m using yarn v3 workspaces right now. To my knowledge, looking for node_modules/.bin will work for everything but plug n’ play modules?

4

Ping @ostera (IIRC you’ve been talking about monorepo related things before)

5

Previous versions at least worked together with https://github.com/reason-seoul/yarn-plugin-rescript. I did not test it with the newest version though.

6

We are using turborepo with pnpm. rescript is installed separately for every application, and a shared code moved either to public npm modules, or a package, that’s connected as "files": [../../packages/rescript-shared]. It’s done this way to enable watch mode without starting a separate rescript compiler for the package.

2 Likes
7

do you use rescript-shared as pinned dependencies in the apps?

do you run all dev at once ?

do you have the issue in VSCode when you close a terminal pane it doesn’t kill the process running by turborepo?

do you have to sacrifice the Dev experience of the terminal logs?

8

No, because the watch mode doesn’t work with pinned dependencies.

We run rescript compiler for every application separately.

Turborepo is mostly used for production builds, or building application dependencies before starting nextjs in dev mode. We run rescript compiler separately.

That’s the reason why we run rescript compiler separately, so we have nextjs, rescript and relay logs in different terminals.

1 Like
9

interesting.

I was running a similar setup but with pinned-dependencies, until one day I had to do a hot fix and I couldn’t update the pinned dependencies. So out of frustration I removed turborepo and stayed with pnpm workspaces. Still have issues from time to time with the updating of dependencies but not that bad after running pnpm i and rescript clean.

I might give turborepo another chance although I’m not sure how much of a benefit it is to add that dependency to my stack considering that I develop without a team.

10

I’m a big user of monorepos with Yarn. I’m not actually aware of any things currently not working in such a set-up.

A public example is my Twitch overlay (currently on an unmerged branch, hopefully Soon :tm: updated on main).

Beyond that I have a private repository for our product front-end and we recently converted the project I’m working on with @zth to a monorepo as well (code for that is not yet public).

1 Like
11

What if I have rescript installed globally and not in my project? Then I guess it will not be found and I need to specify the path manually?

I think the best way for the VSCode extension to locate the rescript binary is to ask the package manager that’s in use?

For yarn:

$ yarn --version
3.2.1
$ yarn bin --json 
{"name":"npm-run-all","source":"npm-run-all","path":"/Users/alexander/Projects/front-end/node_modules/npm-run-all/bin/npm-run-all/index.js"}
{"name":"run-p","source":"npm-run-all","path":"/Users/alexander/Projects/front-end/node_modules/npm-run-all/bin/run-p/index.js"}
{"name":"run-s","source":"npm-run-all","path":"/Users/alexander/Projects/front-end/node_modules/npm-run-all/bin/run-s/index.js"}
{"name":"bsc","source":"rescript","path":"/Users/alexander/Projects/front-end/node_modules/rescript/bsc"}
{"name":"bsrefmt","source":"rescript","path":"/Users/alexander/Projects/front-end/node_modules/rescript/bsrefmt"}
{"name":"bstracing","source":"rescript","path":"/Users/alexander/Projects/front-end/node_modules/rescript/lib/bstracing"}
{"name":"rescript","source":"rescript","path":"/Users/alexander/Projects/front-end/node_modules/rescript/rescript"}

For npm:

$ npm --version
8.12.1
$ npm bin
/Users/alexander/Projects/front-end/node_modules/.bin
$ ls -al `npm bin`/rescript
lrwxr-xr-x  1 alexander  staff  20 Jul 25 10:37 /Users/alexander/Projects/front-end/node_modules/.bin/rescript -> ../rescript/rescript

Similarly pnpm

$ pnpm bin       
/Users/alexander/Projects/test/node_modules/.bin

Similarly to get the global folder containing binaries Yarn supports yarn global bin and NPM and PNPM both support passing --global to the <exe> bin command.

While calling the package manager may be considered slow, this only needs to happen once after the lock file (package-lock.json or yarn.lock) changes and can be cached otherwise.

The package manager to use could be read from the packageManager in package.json (if the repository is using Node’s corepack). Alternatively it can be guessed based on the presence of a yarn.lock or other lock file.

I also wanted to test what this would look like in a repository using Yarn PnP. It turns out that if we actually use the package manager, this will just work :partying_face:

$ yarn bin --json
{"name":"bsc","source":"rescript","path":"/Users/alexander/Projects/test/.yarn/unplugged/rescript-npm-9.1.4-75c65d01e7/node_modules/rescript/bsc"}
{"name":"bsrefmt","source":"rescript","path":"/Users/alexander/Projects/test/.yarn/unplugged/rescript-npm-9.1.4-75c65d01e7/node_modules/rescript/bsrefmt"}
{"name":"bstracing","source":"rescript","path":"/Users/alexander/Projects/test/.yarn/unplugged/rescript-npm-9.1.4-75c65d01e7/node_modules/rescript/lib/bstracing"}
{"name":"rescript","source":"rescript","path":"/Users/alexander/Projects/test/.yarn/unplugged/rescript-npm-9.1.4-75c65d01e7/node_modules/rescript/rescript"}
$ /Users/alexander/Projects/test/.yarn/unplugged/rescript-npm-9.1.4-75c65d01e7/node_modules/rescript/rescript -h
Available flags
-v, -version  display version number
-h, -help     display help 
Subcommands:
    build    
    clean
    format
    convert
    dump
    help
Run rescript subcommand -h for more details,
For example:
    rescript build -h
    rescript format -h
The default `rescript` is equivalent to `rescript build` subcommand

As you can see Yarn keeps an unzipped version of the command available for invocation.

2 Likes
12

Hmmm, if you have a workspace with bin folders both in the workspace root and in a package, this only shows the inner bin folder in yarn classic, npm or pnpm. yarn berry is better, since it gives you the full path to each binary.

I think the current solution is not so bad, but it expects node_modules to be there, which is not true for yarn with PnP.

The question is, is there any way to make the build system agnostic from that logic, heck even npm itself? A stupid simple solution would be import maps for all the dependencies, not sure if it is feasible for the binary itself.

In fact there is an old issue that discusses this: https://github.com/rescript-lang/rescript-compiler/issues/3276

13

I’ve been thinking about this. Considering Node Modules, P’n’P monorepo, and even Deno’s multi-module configuration, more things are needed.

(We really need to follow up NPM packages that efforts to support multiple platforms.)

I guess the simplest way is having an independent package manager to emit files required by a platform and its lock files.

1 Like
14

Do you perhaps have a repo that reproduces this? I’m not able to create this scenario in my own testing (with monorepos). Also is this actually a problem or does the package manager also provide rescript in that bin directory (since it should be a dependency of the lower level package too).

I’m very strongly against ReScript having its own package manager.

To gain adoption in the JavaScript community we should make interoperability as easy as possible. The JavaScript community is not waiting for yet another package manager.

I have a slide in my presentation on ReScript and getting ReScript is as simple as running npm install rescript. Lets not make that any more difficult than it needs to be. One of the bigger hurdles for Reason adoption was having to become familiar with dune and the like (and understanding what does what).

In my eyes the solution here is for ReScript to better work with the existing package managers (by fixing the issue fham linked), not by adding another tool for new users to learn and for the ReScript community to maintain.

2 Likes
15

My point is that “node_modules” != “whole JavaScript community”.

Since Deno, Bun, Yarn Berry already use difference format rather than Node.js, even Node.js has experimental URL import support. This means that relying on node_modules resolution is not a long term solution.

It will become increasingly important to understand that Node.js is not the only option for JavaScript, And I think third-party tools are currently the only way ReScript can explore them without much modification.

3 Likes
16

I have to +1 this sentiment as well, because before landing on rescript I also played with ScalaJs and F# fable, and having to learn a whole new build system / package manager just for a hello world was intimidating and caused enough friction for me to drop them

I love that rescript integrates so easily with the tools and flow that I am used to

3 Likes
17

Okay yeah that’s fair. Though in that aspect we’re in agreement. Hence my test also with Yarn Berry for PnP.

Right now ReScript relies on the file system to find the binaries it needs. Instead I’m arguing that we should rely on the package manager to locate the binary that’s installed. That’ll make ReScript more compatible with different runtimes.

My knowledge of Deno is limited but from what I can find it installs executables in a global folder (or a user specified root) and leaves it to the user to add the executable to the path. So it sounds like in the case of using Deno the user would have to ensure rescript can be found with $PATH.

The discussion in this thread (or at least my interpretation of it and argumentation around it) is specifically about ensuring the ReScript tooling can find its own installed executable in a monorepo set-up.

Any discussion about how dependencies would be imported within a JavaScript file is a separate discussion and not specifically related to monorepos. I agree with you that I’d want PnP support in the compiled JavaScript files. However, that’s not a minimum requirement to get monorepos working. It would be interesting to create a separate discussion on how we can push all of https://github.com/rescript-lang/rescript-compiler/issues/3276 forward (I’d be all for it!) but I think this topic should keep focussed on the much smaller issue of “Don’t assume dependencies [ReScript’s compiler] even exists in node_modules.”

18

The goal of this RFC is basically to create a standard all the ReScript tools like the compiler and editor can adhere to. If we need a meta package manager for that, it can implement this very standard after the specification phase.

What I gather from the discussion now, are the following points:

  • the rescript installation can basically be anywhere and neither should we assume the existence of a .bin folder, nor a node_modules folder. It can be an arbitrary folder anywhere in your project or even outside of it (global installation).
  • Existing package managers may help in finding binary paths, but it depends on the package manager (and its version) how likely you are to actually find the binary. Complex workspace setups are prone to fail here. It may still be valueable to support some blessed paths besides the current default.
  • Building on top of this spec, a (meta) package manager can help in maintaining the developer experience, which will inevitably get worse
  • The node_modules way should still work the same as it does now, at least as long this approach is still the default one across the ecosystem.

Some ideas (just a braindump)

  • .env file in project folder (or any folder above, really). Workspace packages may have different .env files and only consume the workspace root one if not. It could even provide a list of binaries if we need to. Could simply be added to the ReScript template. Users may delete it if they are happy with the defaults.

  • Don’t use the .bin folder at all, just have everything inside the (installed) rescript package itself, and use the binaries from there.

  • Default to a globally installed tool, which handles the calling of the correct project-level rescript binaries. This does not have to be its own full-blown package manager like cargo. Set its location in $PATH and the binary will do the rest, based on some well-defined heuristics.

(these ideas might need to build on top of each other)

3 Likes
19

Some discussion here https://github.com/rescript-lang/rescript-compiler/pull/5722

4 Likes