Compiler testing via fuzzing

I wonder if ReScript compiler developers make use of automatic compiler testing techniques like, Csmith for C or LangFuzz for JS. As RS is relatively young, I think there would be many chances for those fuzzing techniques to reveal RS compiler bugs. Currently I’m studying on software testing, so I’ve been curious. Do developers face any need in systematic & automated testing of ReScript compiler?

Any comments will be appreciated.
Thanks in advance :kissing_smiling_eyes:

A couple of relevant areas that come to mind.

  • Can you cash the parser?
    That area is already tackled by static analysis (using https://github.com/rescript-association/reanalyze). No unsafe code is use so the remaining issue is throwing exception. The analysis reports on uses of unchecked functions that might raise exceptions. That said, recent integration of the parser in the editor extension, which effectively uses users as “fuzz testers” revealed one case of crash (unhandled exception) where the source code was explicitly silencing the static analysis.
    The other area handled via static analysis is infinite loops, which were quite common before turning on static analysis for termination.

  • Can you crash the editor extension?
    The code is currently not as strong as the parser, and there have been, and likely are left, cases where it can crash (raise exceptions) under certain inputs.

6 Likes

@cristianoc Thank you for comments:)

Sounds interesting that there existed a corner case which made parser malfunctioning. I have a question as I little confused. Does it mean that rescript parser crashed with unexpected exception which reanalyzer didn’t report? What makes me confusing is, rescript parser is an ocaml program, while reanalyzer seems targeting rescript program.

Reanalyze works for both OCaml and ReScript projects.
Here’s the relevant commit: Fix issue where the parser crashes on empty unterminated comment. · rescript-lang/syntax@ccd4e3d · GitHub

There’s an explicit annotation [@doesNotRaise] that is used to silence the analyser. Either the problematic case was missed in the beginning, or some small change has introduced it afterwards.
In any case, when something goes wrong, there are only a few spots to inspect.

1 Like

Thank you for your kind answer, and besides that concrete example is very helpful for me. Because complex semantic restrictions(e.g. def before use) are not needed when a parser input is made, this task seems to be a good starting point. Even I’m not going to do it right now as I’m working on an urgent project, I’ll post here if I make some progress about it :kissing_smiling_eyes:

1 Like