Oh no, we're toast because of PPX usage

[mrmurphy]

Okay, in truth that subject was clickbait. We’re not toast. But we do have a potential huge load of work in front of us.

The Problem
Day One has 105k lines of Rescript code that runs its backend that we’ve written over the last few years. And we have a boat load of types that use GitHub - reasonml-labs/decco: Bucklescript PPX which generates JSON (de)serializers for user-defined types to generate JSON encoders and decoders to check and transform JSON into types safely.

When we started using that package it seemed like a pretty safe bet. But that was before Rescript came and the native/js nature of Reason was put on the backburner.

Nowadays I regularly encounter stale build issues that always involve decco. And unsurprisingly, upgrading to Rescript 11 brings breaking changes with that package.

I tried diving into the decco repo since I’ve contributed before. But I was never very familiar with the native OCaml ecosystem, and on top of it Decco uses esy a build system that I think has also been put on the backburner with the old Reason syntax. I can’t get the ppx to build.

I’m also thinking that I don’t want the PPX to build. Over the years it has become clear that PPXs in Rescript are not the happy path. I think that if we insist on keeping Decco around, we’re going to encounter more trouble than we save by using it.

On Getting Out
So, we have tons of @decco decorations all through our code on both type definitions and their fields. Does anyone know if there’s any way to use the rescript compiler in a way that will run the ppx, and then spit the source code that was produced back out?

If that’s possible we can keep relying on the Decco rescript module, but remove the use of its PPX, which I think would be acceptable for us.

Otherwise, I think we’re going to have to either re-write all of the encoder/decoders by hand, or write some very complex code parsing/generation script.

[Hossman333]

I really hope there’s a straightforward answer to this.

[tsnobip]

Wouldn’t it be much easier to just search and replace @decco annotations to replace them with ppx that are still supported like spice or rescript schema?

[zth]

Several options here. OTOH just like @tsnobip says, migrating to Spice might be an option? GitHub - green-labs/ppx_spice: ReScript PPX which generates the JSON (de)serializers

Anyway, bear with me with the technical details, but in ReScript v11.1 there’s a flag you can use to reprint a source file and expand a certain PPX while doing it. You need to call the underlying compiler (bsc) directly. Here’s an example:

// Tst.res
@decco
type myType = {
  name: string,
  age: option<int>,
}

Command: ./node_modules/rescript/bsc -ppx node_modules/decco/ppx -bs-no-builtin-ppx -reprint-source src/Tst.res

This will output the above source, but with the PPX expanded:

@decco
type myType = {
  name: string,
  age: option<int>,
}
@ocaml.warning(@reason.raw_literal("-39") "-39")
let myType_encode = v =>
  (
    v =>
      [("name", Decco.stringToJson(v.name)), ("age", Decco.optionToJson(Decco.intToJson)(v.age))]
      |> Js.Dict.fromArray
      |> Js.Json.object_
  )(v)
@ocaml.warning(@reason.raw_literal("-4") "-4") @ocaml.warning(@reason.raw_literal("-39") "-39")
and myType_decode = v =>
  (
    v =>
      switch Js.Json.classify(v) {
      | @explicit_arity Js.Json.JSONObject(dict) =>
        switch (Js.Dict.get(dict, "name")->Belt.Option.getWithDefault)(
          Js.Json.null,
        ) |> Decco.stringFromJson {
        | @explicit_arity Belt.Result.Ok(name) =>
          switch (Js.Dict.get(dict, "age")->Belt.Option.getWithDefault)(
            Js.Json.null,
          ) |> Decco.optionFromJson(Decco.intFromJson) {
          | @explicit_arity Belt.Result.Ok(age) => @explicit_arity Belt.Result.Ok({name, age})
          | @explicit_arity Belt.Result.Error(e: Decco.decodeError) =>
            @explicit_arity
            Belt.Result.Error({...e, path: \"^"(@reason.raw_literal(".") ".", \"^"("age", e.path))})
          }
        | @explicit_arity Belt.Result.Error(e: Decco.decodeError) =>
          @explicit_arity
          Belt.Result.Error({...e, path: \"^"(@reason.raw_literal(".") ".", \"^"("name", e.path))})
        }
      | _ => Decco.error(@reason.raw_literal("Not an object") "Not an object", v)
      }
  )(v)

This is a way to “eject” from a PPX. But, there’s obviously no guarantees that the code the PPX generates will compile, so you might need to figure out some sort of migration/cleanup script if you go this route (https://comby.dev is excellent for this).

[jfrolich]

We recently moved to ppx-spice from decco in a 500kloc codebase, and it was a pretty good migration! (it’s mostly compatible)

[mrmurphy]

Thank you so much, everyone, for your answers! It was very helpful to discover rescript-schema as an example of an updated PPX project that uses dune directly instead of esy. I’m currently in the middle of upgrading Decco to use .ml files with Dune for the native code, and .res files for the Rescript modules. Things are going pretty well so far. We’ll see if it all works out in the end.

[mrmurphy]

Hey @zth! I’m making a ton of progress on updating Decco. The challenge I’m facing now is that the transformations it does seem to produce a curried function:

Interestingly, the function produced here in the output has only one argument:

There are encoder functions that will take more than one argument, so I’ll need to learn how to produce an uncurried function in the PPX. According to my research so far, OCaml itself has no concept of uncurried functions, right? Only functions that take a tuple as an argument.

I’m assuming that curried VS uncurried is something at the Rescript syntax level and that it happens before the PPX step is run. Is that right? I think this because when I copy-paste the output of the PPX preview into a new file and save, the error goes away.

I’m not sure how Rescript detects curried and uncurried functions at the AST level. Will it be enough to adjust Decco to always produce a function that takes a tuple of arguments instead of an argument directly?

I’ll go ahead and see if I can try that.

[mrmurphy]

I did an experiment by writing out an uncurried function in Reason, then using refmt to convert it to ocaml. The resulting code uses the [@u ] decorator in front of the function. I’m guessing then we have a compiler extension that then uses that decorator to know that the function is uncurried? Or perhaps that’s a bucklescript thing. In any case, I’ve updated Decco to produce that decorator on its generated functions, but I’m still getting the compiler error that v_encode is a curried function where an uncurried function is expected. I’ll keep digging, but would love any ideas you could offer.

---

Edit, I just looked at how rescript-schema-ppx handles this, and it seems they use Object.magic to cast the function:

---

Edit again, I see that Spice upgraded already Support uncurried on v11 by mununki · Pull Request #49 · green-labs/ppx_spice · GitHub
Hopefully I can follow what they are doing there, and I see @alex.fedoseev asked here: Quick migration guide to uncurried mode for PPX maintainers?

Hopefully I can figure out how to apply those decorators in the PPX

[mrmurphy]

We’ve found a solution and merged a PR for supporting Rescript 11! Now we’re working on record spread support.

[DZakh]

The example from rescript-relay might be more helpful. I was quite lazy with rescript-schema since tests cover this part well.

[spyder]

Spice or updating Decco was probably the correct answer for you, but many years ago I grew tired of the performance overhead of running a PPX when the percentage of files it actually needed to process was fairly small. Luckily this was early enough in my project that swapping to another solution was manageable.

So I migrated from decco to atdgen, which relies on a weird bucklescript-era generator config, and have been very happy. I’m currently working on a fork to make the process work better on ReScript 11 and plan to post about it on this forum once it’s ready.

[tsnobip]

Atdgen is really nice, it now works with quite many languages, including python, which is pretty useful when you want to share types between languages for API!

It’d be cool to make it output more idiomatic and modern rescript though, like optional fields, unboxed variants, etc.

[spyder]

We’re getting a bit off topic, but

@tsnobip

It’d be cool to make it output more idiomatic and modern rescript though, like optional fields, unboxed variants, etc.

The team is open to a contribution for this, based on a discussion I had with them a couple of weeks ago, but I doubt I’ll have time to do it myself. I just opened a ticket with the details.

Add a ReScript output option

[mrmurphy]

Thanks for the help, everybody!
I’m circling back here to say that we’ve upgraded Decco to support Rescript 11 now, and added support for the new record spreading syntax in types!

Release v2.0.4 · rescript-labs/decco

Heads up that the package name has changed on NPM from decco to @rescript-labs/decco

[zth]

Impressive work! Quite a lot of intricate internal stuff to understand for this to work.

[mrmurphy]

Thank you! Somehow we felt our way through piece by piece

[ashton]

Hey, I use GitHub - green-labs/ppx_spice: ReScript PPX which generates the JSON (de)serializers which is based on Decco, it’s pretty neat and maybe could save you some effort migrating

[mrmurphy]

Thanks @ashton but we’re already done! And I think we may have avoided some bugs that Spice needs to address still.