Progress on MetalScript and random thoughts
This has been a good week for MetalScript.
This post is more of an informal ramble. Read it if you extra time on your hands, but I don’t think I’m going to say anything profound here. Also read this if you’re doing a product competing with MetalScript, since I’m spilling some of the implementation here ;-)
A few weeks or months ago1, I switched over from MiniLanuage to MetalScript. MiniLanguage is a new language I started about 9 months ago to test-drive a number of the MetalScript principles in a simplified context. It’s in MiniLanguage that I’ve created a working end-to-end pipeline from source text to output binary. But rather than perfect MiniLanguage with all the bells and whistles, I wanted to move back to the real project.
Breakthrough — bridging compile time and runtime
I had a breakthrough this week in terms of structuring code that implements the spec in a way that splits between runtime and compile time. In particular, the ParseScript operation (and some of the surrounding code) has been a bit of a pain in the butt — the behavior of the function according to the spec is to return a Script Record, which contains both a reference to the realm2, and also a reference to the script code.
The script code is purely a compile-time construct, while the realm is purely a runtime construct. So ParseScript is kinda split between runtime and compile time. It’s really awkward to consolidate code that is split in this way, while still trying to make it maintainable. And I’ve been bashing my head against a wall for a while on the right way to do it.
Something finally clicked this week, and I found a way to have both runtime behavior and compile time behavior defined in the same lines of code. The way it works is essentially that my implementation of ParseScript returns a monad, that contains both the compile time component of the return value, as well as the sequence of runtime IL operations required to get the runtime component of the return value. The caller can immediately use the compile time component, and then is obliged to also emit code that invokes the runtime component.
For simplicity, I opted to implement the monad as just a tuple, as described by the MixedPhaseResult type below.
/** Used as the result of an operation that has both a compile time and runtime
* component. The compile time component of the operation is the function
* itself, and its result is the first element in the returned tuple. The
* runtime component is represented as an IL function that is the second part of
* the tuple. */
type MixedPhaseResult<T> = [T, il.ILFunction];
...
// https://tc39.github.io/ecma262/#sec-parse-script
function parseScript(unit: il.Unit, sourceText: string): MixedPhaseResult<ParseScriptResult> {
...
}
...
const [parseResult, parseScriptRT] = parseScript(unit, sourceText);
const scriptRecord = code.op('il_call', parseScriptRT, realm, hostDefined);
(Side note: Yes, of course MetalScript is being written in TypeScript, and the plan is to make it self-hosting so I can compile MetalScript with MetalScript in order to distribute an efficient binary executable of the compiler).
The beauty of this approach is that it allows me to have a single parseScript function in my implementation, which as you can see above has an embedded comment that references the exact location in the spec, and the full behavior of the corresponding piece of the spec is fully encapsulated in the body of parseScript. This one-to-one relationship between spec functions and implementation functions is going to be super useful from a maintenance perspective — keeping up to date with the latest spec — which as stated in a previous post matches one of my goals with MetalScript.
I’ve used this technique in a number of other places that bridge the gap between compile time and runtime, and I think the result is beautiful.
MetalScript Unit Tests
I’ve also started writing unit tests for MetalScript. Previously I avoided unit tests because there was too much uncertainty and I landed up completely changing my mind on things too often to make unit tests a useful addition. Now after spending 9 months in MiniLanguage, I feel I’m reaching a point of stability in the underlying concepts and have started adding unit tests.
The first unit tests I have working use the above parseScript and related functions to translate a source text to IL. And as of today I have this working for 2 simple test cases — an empty script and a “Hello, World” script.
I can’t show you the IL itself, because it would give away too many of the internals. Maybe when I’m further along in the project and there is less risk of having my ideas stolen, I will give up more details. But believe me when I say the IL is a thing of beauty.
The hello-world script translates to 62 lines of IL (including some whitespace), which is a lot, and emphasizes how many operations are actually required to perform simple tasks in JavaScript, and how much of an accomplishment it is to get to this point. Bear in mind that this IL language is designed by me with the intention of compiling easily, not to be a compact representation of the program, since the IL will never get to the target device.
Personal Note: Be comfortable with your work
A personal lesson I’m learning with this and other projects, is to do what it takes to feel comfortable with what you’ve done. In MetalScript, it’s a constant battle in my mind as to whether I should cut corners to save time and get to a POC quickly, or whether I should take it slow and make sure that every piece is as simple, understandable, reliable, and maintainable as possible.
There are arguments for both at different stages of a project, but if you plan on the project becoming something big, then I really believe you need to do what it takes to feel emotionally comfortable with what you’ve done. The reason is that when you leave a piece of code, and in the back of your mind you think of it as hacky, fragile, or overly complicated, and when you wrote it you just had to pray that it worked, then you aren’t going to want to go back to it, and you will become generally demotivated by your work. But if you leave a project or piece of code feeling comfortable about it, then it will be much easier to go back “home” to it in future.
So when I say that you should spend time on your work until you feel comfortable with it, I’m not talking about spending time making the most advanced piece of code you can be proud of, that has a gazillian features and can do backflips and handstands and handle a bunch of different use cases. I’m talking about thinking really carefully about how to remove complexity from your code and distill it down to its bare essence. You want to use your superpowers to remove complexity, not to handle it. Understanding a complicated design is only the first step; reducing it to a simple design is the end goal.
If your code is clean, simple, and has a good readme and guiding comments to help newcomers get into it, then you will feel more comfortable when you are the newcomer getting back into it after some time.