[ANN] Call for testing - improved syntax highlighting in VSCode extension

[zth]

Syntax highlighting was by far the most upvoted issue reported for the VSCode extension. So, me and @cristianoc have been working on improving it, and we’re finally ready for some external testing.

But, before the test instructions, let’s summarize what we’ve been doing and what design principles we’ve been following:

• Our guiding principle has been to look as similar as TypeScript highlighting as possible. This has 2 reasons: 1) It provides familiarity for TypeScript devs coming to ReScript. 2) TypeScript highlighting is battle tested, well supported over a large amount of themes, and polished.
• We’re leveraging semantic highlighting for most of the new highlighting. However, not all themes support semantic highlighting, which is why you’ll probably still get the best highlighting results using one of the officially recommended themes. But, after this lands, many more themes in VSCode than before should be usable.

Check below for how the results look using the Dark+ theme:

Current highlighting

New highlighting

Current highlighting

New highlighting

Testing instructions

Download and install this vsix. It’s a custom build of the VSCode extension with the new syntax highlighting enabled. Please test and let us know your feedback in this thread. Please remember to be specific in your feedback, and preferably use screenshots. Also remember that we’re trying to stay as close to TypeScript as possible, which might help explain why things look the way they look.

More information about the implementation

Performance

One of the things we set out to explore was whether the performance of semantic highlighting could be good enough. Semantic highlighting is not instant - VSCode will actively ask our language server to provide a set of semantic tokens for an entire document. And when that happens, we need to read that file, parse it to an AST, extract tokens from it, and send those tokens back to the language server via JSON.

VSCode does this asynchronously, and has built in throttles, which means that semantic highlighting can take a (small) amount of time before it’s applied. This is normal and it’s the same for all languages leveraging semantic highlighting (check out TypeScript, as well as Rust).

Still, we’ve focused on how we can keep this feature as fast as possible. We’re quite lucky in that the ReScript parser itself is very fast. Have a look at the video below for an example of the current performance:

We’re looking forward to hearing how performance is for you, and after doing quite extensive testing ourselves, we’re optimistic it’ll be fast enough to give a good experience.

Robustness

Since semantic highlighting relies on parsing a document to an AST, and then extract information from that AST, robustness of the parser comes into play. We’re again lucky that the ReScript parser is robust in terms of parsing and recovering from errors. A fun side effect of this is that our semantic highlighting is more robust in the face of errors than what for example TypeScript is. Check out the sample videos below, and notice how syntax highlighting disappears as there are syntax errors in TypeScript, but not in ReScript:

Summing up

We’re very excited for you to test this and give your feedback! Looking forward to hearing about your experiences.

[carere]

@zth Hello,

First, thanks for this awesome work !!

I see one problem (in my opinion), the coloration of functions is missing in the extension if we compare it to Typescript:

In TS all functions are properly colored (lastValueFrom, execute,merge, …)

I think that we could get the same colorations for the following function below (in ReScript)

Do you think it’s possible to set different colors between arguments and function ?

PS: I’m using Dark+ (default Dark) theme

[Lomand]

Oh my God, that’s simply magnificent!
It is rather difficult for me to communicate how much satisfaction the news about semantic highlighting brought to me today! I’m genuinely asking myself should I go to get some drinks to celebrate the issue or not.

Thanks a lot for your enormous contribution to the Rescript community!

PS: I’ve added the extension to my VScode, and so far my code looks just as I would expect, literally zero complaints. Thanks a lot, @zth @cristianoc, you guys rock!

[fham]

Thank you so much for this.

I got so used to white text now, that the Dark+ light blue now is a bit too much for me. Maybe it really will help to color parameters differently.

Also types now use the same color as modules, I am not sure if it is correct to follow TypeScript here. Interfaces are kinda like Modules though, so I get where that comes from.

Performance on my MB Pro 2018 is not an issue at all. Maybe it could be an issue on older machines, but with modern hardware it’s not noticeable.

Sure enough, this feature is already so great just because of JSX highlighting. Awesome work!

Btw: the above VSIX also contains this feature:

[mxthevs]

This looks awesome! Thank you for this amazing work, @zth @cristianoc

Highlighting is super fast and it is great! The only thing I noticed, is object fields not being colored properly (using Dark+)

[zth]

You’re absolutely right that TS (and Rust) colors these differently. We’d love to do the same, however what made us opt to not try and implement this in the first iteration of the highlighting is the following:

1. Detecting what are functions and not would require us to look at the compiler artifacts/type information from the compiler. While not a problem in itself (the extension does that for things like hover, autocomplete, etc), it would have the same limitation as hover etc does - the file would need to be saved (and actually compile, so no syntax errors etc). And that limitation would likely affect the experience of the highlighting a lot.
2. We’ve kept everything we’ve done contained to just looking at the AST itself so far. This is good because it reduces the complexity and keeps things fast, and performance is especially important in a feature like highlighting.

We’re not ruling out highlighting functions, in fact we’d love to have it. But there’s some background on why it’s not trivial to implement in an accurate way, and why we opted to not try it for the first iteration.

[zth]

Thank you Florian!

For context - the module highlighting is modelled after classes and namespaces (which we felt were closest to modules conceptually) in TS, although I guess interfaces end up looking the same too.

And yes, “open compiled file” is in there - awesome work!

[zth]

Great catch, thank you! We will make sure to fix that.

[carere]

Thanks for tour explanation, it’s clear now.

I thought that we can tackle this with textmate grammar.

Since every functions are call like func( or someValue->func, we may colorized those with regex.

The only drawback is for functions pass as parameter like func1(func2).
Only func1 woulb be colorized, but it may be a good start.

[praveen]

Thank you for your great work. This will definitely enhance the DX.

[amiralies]

Thanks for large amount of work!

this still looks weired in dracula theme. rescript-vscode is the only language extension that i have problems with (using dracula theme).

semantic highlighting delay is also annoying for me. If i were using vscode as my main editor I definitely would disable that, especially on my slower machines.

It’s still odd for me that why we can’t get a good enough/reliable highlighting that is theme agnostic using textmate grammars only like most other language extensions do.

[zth]

A short update - the initial iteration of this has been merged and will be in the next official release. We’ll continue improving the syntax highlighting as we go along.

Thank you all for testing, giving great feedback, and also for all the encouragement you’ve given us as we’ve worked on this. Thanks!

[fham]

I tried current master and now the highlighting of modules does not differ from values (even with Dark+), which looks similar to @amiralies pic.

[spyder]

@amiralies

It’s still odd for me that why we can’t get a good enough/reliable highlighting that is theme agnostic using textmate grammars only like most other language extensions do.

A fairly comprehensive response to this was posted at the start of 2021. I disagree with the reasoning, but that’s the situation.

[zth]

Would you mind opening a ticket?

[fham]

Highlighting of modules on current master is same color as values

[Ryan]

I tried an example with a couple more of the color themes mentioned in the list of recommended themes. Awesome overall…the new highlighting is great!!

On a couple of the recommended themes though, it seems that the types aren’t highlighted correctly. Check it out:

---

Mariana Pro (Warm):

---

Solarized Light

---

For reference, here it is with Light+ (which seems fine).

[cristianoc]

@Ryan

Mariana Pro (Warm)

Thanks @Ryan, see here: https://github.com/rescript-lang/rescript-vscode/pull/406 which takes care of Solarized Light. Not sure whether anything can be done for Mariana Pro. One can test with the package created by CI in the PR.

[Ryan]

Thanks, and just to be clear, I’m totally fine with sticking with one of the themes that “just works”. But once the new vscode extension is released, it may be a good idea to update the recommended themes!