ReScript Office Hours #3

Events
#1

Hi everyone!

We want to keep up the pace and announce our next ReScript Office Hour!

What are ReScript Office Hours?

This is a remote event where everyone in the community to talk about ReScript. This time will be a bit more technical that we would like to hear the feedback about build, especially the monorepo support

When and how to join?

Tuesday, November 3⋅19:00 – 19:40 (HK +8 TimeZone)

Join Zoom Meeting https://us04web.zoom.us/j/74265766999?pwd=aUVqRlVzTWMvZDBwRGJvQmVxOEVmUT09

Meeting ID: 742 6576 6999 Passcode: 7qRgpM

Note we have people live across the world, it’s hard to find time to fit everyone, we will schedule time better for US next time.

The Zoom free version only offers 40 minutes at maximal, to make things more effective, we can have some offline discussions first about mono-repo support in this thread.

See you soon!

7 Likes
Call for testing: bs-platform@8.3.3-dev.1 (better -make-world performance)
ReScript Office Hours #4
#2

There’s a lot of things I could ask for, but my two biggest requirements are

  1. A single bsb -w command covers all monorepo source code
  2. The ability to use namespace: true for each package within the monorepo

I have given up #2 to achieve #1.

2 Likes
#3

Hell yes, monorepo support!

We use yarn workspaces to be able to share common code among projects. E.g. one project has this in the package.json

  "workspaces": [
    "common-base",
    "common-ui",
    "mobile-app",
    "web-app",
    "simulators"
  ],

Where common-base has no UI components, so it can be used in node context too (simulators). common-ui can only be consumed by the mobile-app and web-app.

Every workspace has their package.json (as required by yarn workspaces) and bsconfig.json. When I now work with the compiler running on e.g. the web-app and I need to change something in common-ui I either have to let another compiler run also in this subproject or I recompile everything from web-app which takes a while.

Would be cool to be able to work on multiple subprojects with a single compiler.

#4

Just to give some inspiration for monorepos, jest has a projects config property. You run jest at the top level, and it runs it over all projects.

#5

I have read all feedback available on the mono-repo support.

Let me explain the original design and why it works in current way.

Note the name is a bit misleading, mono-repo structure is simple and well supported, here users want to support develop multiple packages in separate repos while achieve the similar experience to monorepo.

what does -make-world do

  • -make-world will first resolves each package recursively and build them independently, this has to be done whether there are changes to the package or not, the changes are not tracked by the build system, that’s one of the reasons why it is slow

  • The building is on a different mode from toplevel projects (dependency mode):

    • The warning/errors are mostly ignored, this is designed with the mindset that if your dependency
      has a warn-error, it should not get in your way.
    • Custom rules are not run
      The motivation is that your dependency should not impose his build dependency to its consumers.

  • There’s an installation process after finishing building the package. So for a package a, the user wants to hide some interface from its consumer, he can choose not to install that cmi file. This abstraction is implemented in the installation. Installation is basically copying files which can not be as fast as we hope

What we can improve on current situation?

  • We can fix the dependency mode as long as we have user input that some packages are pinned packages that it should go to toplevel mode instead of dependency mode.

  • We could bypass the installation for pinned packages. That means for pinned packages, every interface is exposed (minor semantics difference from unpinned packages)

Watcher

Our built-in watcher is not great due to the limitation of cross platform and minimal dependencies. We want to offer the help to make it easier to integrate 3rd party watchers e.g, watchman

Merge multiple pinned packages into a monorepo?

In theory, the build system could scan multiple pinned packages and treat it as a monorepo, this is a lot of work. However much work we have done here, the semantics would not be the same as multiple packages since the generated js files are path sensitive, even if we make sure all relative paths are still calculated correctly, it is hard to preserve the semantics with bindings, The benefit is of course performance.

Suggestions

If you truly own these packages, I would encourage you adopt a real monorepo setup, then you don’t have these problems at all. For pinned packages, we will discuss how we can improve here.

Bugs

There are some other minor bugs about path resolving which should be fixed later.

3 Likes
Call for testing: bs-platform@8.3.3-dev.1 (better -make-world performance)
#6

Let’s put aside the english definition of “mono”, the term monorepo (at least as far as I’ve heard it, and what I believe all participants in this feature request are asking for) is specifically used to describe a repository where multiple packages are developed together for convenience (for example making cross-package changes with a single PR). They usually depend on each other but are treated as independent logical units.

As a TypeScript developer, being forced into a single namespace is extremely annoying. To give you an idea of the sort of monorepo I’m used to, TinyMCE has 6783 TypeScript modules (including tests) spread across 23 packages. I am now working on a moderate sized (400-500 files) ReScript project with 12 logical packages. If we ever try to introduce ReScript to TinyMCE core, monorepo support will be essential.

I do not want to run 12 copies of bsb, which wouldn’t even work as described by others above who have tried it. I want to use one, as I do in TypeScript. For now I have chosen to develop under a single bsconfig file, which seems to be what you are encouraging us to do, but it forces me to use namespace hacks. Fully 80% of the files in my project are manually namespaced by package e.g. modules/somepackage/src/Somepackage__ModuleName.re because namespace: true is unavailable to these packages and we have to implement it manually. We create the Somepackage.re file with all of the module aliases inside, and then use it from other packages as Somepackage.ModuleName. This is confusing and error-prone because nothing actually prevents using the internal modules directly.

This sounds like it would solve most of my problems :thinking:

I fully appreciate that it might not be possible to implement all of the features a separate bsconfig file per package would imply. And that it might take a few releases to get it to the place we’d like it to be (I complain, but I’ve been developing with manual namespaces for over 12 months).

I do want to note however that JS build tools do support monorepos. The monorepo install tool (yarn in my case) creates symlinks for each source package under node_modules so cross-package imports work perfectly using the package.json name and I expect bindings will as well.

3 Likes
#7

Hi,

Thanks to all who participated in the meeting! the meeting is helpful and effective.

I made a note about the issues we are going to address for pinned packages support.

Bug fixes when dependencies changed

There are several obvious bugs when you change your dependencies but will be fixed, hope this alone would make your life easier:

  • Installation, the installation was not always triggered for some reasons,
    This was correct if you don’t change the depency, but seems more people
    will change the deps, we are going to fix this.
  • Scan the directory in a deterministic way && Lock the deps in the bs-depenencies
    Previously, the scanning order depends on the OS specific behavior, and re-do a make-world,
    will rescan, we will fix this non-deterministic behavior, and lock the bs-dependencies once
    it’s scanned, this probably contributes to your inconsistent assumption error

Toplevel mode and dependency mode

  • The two modes are slightly different, for example, dependency mode does not run generators, it makes sense. However, we will apply toplevel mode to pinned packages too so that it generators are always re-run when the source changes

Watcher issues

  • We plan to make the build system 3rd party friendly so that you can query which directories to watch and integrate with your favorite watcher.

Other issues

  • Concurrent build
    Suppose both A and B depends on C. -make-world A and -make-world B could potentially trigger a concurrent build on C. We are not sure to allow concurrent build in the same directory, but at least we
    will lock the build directory and spits out a meaningful error.

Other high level changes to the build system

  • Proposed UI Changes: subcommands mode
    The current command line flags are a bit messy and some flags are in conflict for different state. We plan to follow other tools, e.g:

    npx bsb clean
npx bsb build
npx bsb format
npx bsb install

Build system bugs are hard to reproduce, if you can contribute some reproducible failure examples, that would be much appreciated! – Hongbo

7 Likes
#8

Very interesting. I was just thinking about monorepos. I’m trying to use JS with Yarn workspaces and Lerna… it seems super complicated (to me). I hope something better comes up!

BTW when will be the next office hours? It would be cool if it were a weekly or bi-weekly event, one-off events are easy to miss!