Thanks for the great meetup yesterday - it gave me some ideas. I’ve been thinking about AI generating bindings before. In essence, on the assumption that you are using the (non-existing) binding in a mostly correct manner, we can extract bindings from the way you are using them. I’m sure you get the idea.
I’m working on a tool to auto-generate the bindings as needed using AI. Ruminating over things, I came up with the following scenarios:
1. The user is using bindings (that don’t exist yet) incorrectly.
An example of this would be sort of doing something like:
let foo = Dayjs.turnBackTime()
There are variations to this that are more or less troublesome than the example above. These will be hard to handle until another layer can be added that maps usage to maybe publicly available API specs.
2. The bindings are used properly, but do not exist at all.
This is the simplest case, it seems. We can just try to compile, notice that we’re referencing a module that doesn’t exist, check whether there is a src/bindings/ModuleName.res file. If there isn’t, have an AI read the erroring file and generate bindings for that particular file in the bindings dir. And hope for the best.
3. The bindings are used properly, and there exists some bindings, but some are missing.
Slightly more complex, possibly, but GPT4 (gpt-4-1106-preview) actually seems to do a better job here. We provide the erroring source file to the AI, along with the bindings file. We ask the AI to adjust it by adding the missing bindings after having read the erroring source file.
4. The bindings are used properly, and we generated bindings, but we screwed up
This will likely be the next case that I have to handle. I’m not done yet. It’s really hard to catch some scenarios because, as you know, the AI is not deterministic (maybe I should not be using the chat completions AI - I will have to experiment but appreciate all input).
I think we can detect it by comparing compiler output. E.g., if we generated bindings, did the error message change before and after generating? If so (unless we end up in scenario 3), we probably screwed up with the generation and generated garbage.
Anyway, I just have a POC so far but it’s doing its job alright. I don’t want to get your hopes up that this is even possible (as of today) for real-world use cases, I’m unsure.
The plan is to have it run “wrapped” around the compiler. It would take over the role of bsc watch and monitor for file changes, and then run continuously as needed as a layer between you and the compiler, reading error messages. Seeing the error messages, it will be able to:
- Automatically fix code (e.g., write bindings)
- Provide additional helpful information on the error message.
- Provide you with suggestion for how you can fix the error and, if a known solution is there, fix it for you.
Sort of like a more automatic GitHub co-pilot, but actually useful.
Anyway, let me present to you, revalkyr (working title, lol - GPT4 came up with the name for me and actually wrote most of the first prototype that I binned for me):
- We have the source file Main.res like this:
src
└── Main.res
Its contents:
let main = async () => {
let fmt = (await Ky.get("http://foo.bar/dateformat"))->Ky.text
let d = Dayjs.make()->Dayjs.format
Js.Console.log(d)
}
let () = ignore(main())
As you can see, we can’t compile it because we don’t have the bindings for Dayjs and Ky. We then run revalkyr and this happens:
Compilation failed because bindings for the module 'Ky' are missing (detected in src/Main.res on row 4).
Attempting to generate bindings for Ky (with src/Main.res in mind).
We'll put the generated bindings in src/bindings/Ky.res
Bindings were generated for Ky in src/bindings/Ky.res so we will attempt to compile again.
Compilation failed because bindings for the module 'Dayjs' are missing (detected in src/Main.res on row 5).
Attempting to generate bindings for Dayjs (with src/Main.res in mind).
We'll put the generated bindings in src/bindings/Dayjs.res
Bindings were generated for Dayjs in src/bindings/Dayjs.res so we will attempt to compile again.
After that, the src tree looks like this:
src
├── bindings
│ ├── Dayjs.res
│ └── Ky.res
└── Main.res
And we have exactly the bindings we need for the project to run:
Dayjs.res:
// Dayjs.res
type dayjs
@module("dayjs") external make: unit => dayjs = "default"
@module("dayjs") external format: dayjs => string = "format"
Ky.res:
// Ky.res
type response
// Assuming Ky.get returns a Promise of response type
@module("ky") external get: string => Js.Promise.t<response> = "get"
@send external text: response => Js.Promise.t<string> = "text"
There may be inconsitencies in naming, comments, etc. - but who cares if all your bindings can be autogenerated. It builds and runs now. Over time, we could likely converge towards some standard bindings that we could include for known modules (sort of like npm @types). It would probably be good enough in most cases.
Beyond that, I think we are at a point in time where we can make huge gains in terms of community and exposure if we can integrate more AI tooling, something that I think is more difficult and risky for the larger, established players to do. I’ll keep playing with this for a bit, once I get it a few steps ahead – IF I do – I will just open source it on GitHub.
Feel free to chime in with your thoughts. Also, sorry for wall of text.