A new post by Simon Willison nearly perfectly captures my current thoughts on how to understand the shortcomings of using LLMs to write code.

Hallucinations in code are the least harmful hallucinations you can encounter from a model. The real risk from using LLMs for code is that they’ll make mistakes that aren’t instantly caught by the language compiler or interpreter. And these happen all the time!

One thing I’ve learned building Conductor is compilation errors are the easiest to fix. Most of the time all that’s needed is to give the LLM the error and it will fix itself. If that doesn’t work, I can prompt my way towards a fix. Self-healing or prompting solve 99% of all of these types of errors. Making sure that the code actually does what I intended is an entirely different thing.

Also from Simon:

A general rule for programming is that you should never trust any piece of code until you’ve seen it work with your own eye—or, even better, seen it fail and then fixed it.

This is increasingly less common with the proliferation of LLM-enabled code generation. We’re thinking of ways to “bake in” this process into the systems we’re building but it’s a really hard problem.