It’s been a while since my last update! Unfortunately, this is because there’s not a lot of observable progress to speak of. But today, here’s another update for those who are interested. I’ll preface this by saying that this update is not particularly interesting.

TL;DR: my personal life has got in the way, and MetalScript has been more difficult than I anticipated.

Personal Life

Since I’m the only person working on MetalScript, and I’m working on it in my spare time, the progress on MetalScript is intrinsically dependent on the state of my personal life. So here’s the status on that. (Or skip to the technical section).

Firstly, I’m going through a divorce. It’s been emotionally taxing and takes time and money (and money = time, and emotional strain = loss of time).

Secondly, I left my traditional job in January. I’m still doing contract work, to lessen the blow on my savings accounts, but overall, working for money is a lesser part of my life now (at least until my savings run out). In theory, this will give me more time to work on MetalScript!

Thirdly, I’ve moved house and country. Kinda. I’ve done half of the move: I’ve moved out of my previous apartment in January, but I haven’t moved into anywhere else yet¹. For the last 5 months or so, and for the foreseeable future, I’ve been living a nomadic life, jumping between staying with family and friends and Airbnbs on different continents². I carry in my suitcases the possessions I really need, like, umm, my laptop, monitor, mechanical keyboard, Homepod, headphones, and VR headset³. Oh, and clothes and stuff. Most of my other possessions have been sold or donated to charity⁴ (or given away in the divorce).

Technical

My last progress update⁵ was in October. At that time, I had a working demo showing a small snippet of code compiling (see pages 22-26 of the presentation I made available in my October post). The summary of progress since then is that now MetalScript cannot compile anything at all, as I’m rewriting large parts of the symbolic interpreter. Most of the time I’ve spent on the project over the last 6 months has been in refactoring and progressing on the design concepts.

Limitations of the previous work

Given that it was a work-in-progress in October, there were some limitations to the kind of code that could be compiled. One such limitation is the fact that it only works when the output does not have multiple runtime functions — the symbolic interpreter did not yet have a way of passing runtime data between different runtime scopes (function scopes).

A related limitation is that the symbolic interpreter could not compile a program that required runtime heap allocations.

These are the challenges that I’ve been solving recently and will continue to do so over the next few months.

Design Work

The vast majority of my time on MetalScript over the last 6 months has been spent on design work (or what I call “thought-work”).

My design process is to journal extensively, to hash out different approaches and refine unclear concepts. In the MetalScript project, I’ve accumulated over a million characters across all the markdown files that make up my journalling history.

I used to be able to write code directly from the ideas in my head. Using a fuzzy language like English (or pseudocode, IMO) for describing behavior is for people who have fuzzy thoughts⁶. If your ideas are clear in your mind, then expressing them directly in code is trivial⁷.

With MetalScript, and other projects where I push the boundaries on my capabilities, I find that I’m less and less able to write code directly. A lot of the ideas I conceive are so foreign and vague to me, just an abstract notion or gut feeling, that I can’t even express them well in English, let alone in code. Hence why I write pages and pages of journaling (mostly English prose), often rehashing the same idea 10 or 20 times over the course of weeks or months, each time making it slightly clearer, until it reaches the clarity required to express it in code.

The thought-work I’m working through at the moment is related to the symbolic interpreter, and how it handles ambiguity in pointer values.

Moving to POD Types

Previously, a lot of the internal data models of the compiler used direct cross-references in their in-memory representation. For example, a jump instruction may contain a direct reference to the block to which it jumps, or an add operation may contain a direct reference to the operands.

While this is convenient when either looking up the reference (e.g. performing the jump) or in constructing the reference, it makes some things more difficult when it comes to ease of maintenance/programmability:

• Data in this model cannot simply be dumped to the console or a file as a JSON, because JSON embeds the structures that are referenced, which may even be recursive.
• Cloning structures with internal or cyclic references is more difficult.
• When debugging, it’s a lot harder to visualize what’s going on without human-readable IDs (direct references have an implicit machine-readable ID which we call the “address”).
• Related to debugging, if you need to save IL or VM state as a file (for debugging or just persistence), if there are internal references then you need to calculate ephemeral names for everything on the fly, in order to represent the cross-references in a persistable way.

Given the phase of the project, and the need for simplicity and debugability over performance, I decided to overhaul the data structures to be pure POD-types that don’t include internal cross-references. In this model, all cross-references are implemented as string or numeric IDs (depending on the context).

Simplified IL Model

The symbolic interpreter is a complicated beast, and to give it the best chance of success, it helps if the IL that it processes is as easy to interpret as possible. Earlier this year, I undertook a large sub-project to redesign and simplify the IL model (the set of IL operations and their semantics). The new IL model is able to express JavaScript source code in a form that is much easier to interpret, which makes downstream phases of the compiler simpler. Simplicity is critical when working on such a large and complicated project.

A change in my personal work process

For me, the thought-work (iterative journaling) is incredibly taxing, mentally and emotionally, due to its complexity, inherent uncertainty and lack of clear progress indicators. I find I burn out after a few hours in a day of doing such work.

The solution I’ve adopted recently is to split my work day: I spend a few hours on concept work, followed by the rest of the day on what I would characterize as easy grunt work, such as implementing the ECMAScript specification. For example, yesterday in my “easy time” I implemented object destructuring, so now the following JavaScript is compiling to IL:

const { foo } = require('./module');

For interest’s sake, this is what the IL looks like (the highlighted lines correspond to the destructuring, and the preceding lines are the function call to require):

    t0_0 = sys_op("resolveBinding", "require", false, undefined);
    t0_1 = sys_op("getValue", t0_0);
    t0_2 = sys_op("getBase", t0_0);
    t0_3 = sys_op("withBaseObject", t0_2);
    t0_4 = list_new();
    t0_5 = copy("./module");
    t0_6 = sys_op("getValue", t0_5);
    t0_7 : list_push(t0_4, t0_6);
    t0_8 = sys_op("throwIfNotCallable", t0_1);
    t0_9 = sys_op("call", t0_1, t0_3, t0_4);
    t0_10 = sys_op("getValue", t0_9);
    t0_11 = reg_get("context");
    t0_12 = record_get(t0_11, "lexicalEnvironment");
    t0_13 = sys_op("requireObjectCoercible", t0_10);
    t0_14 = list_new();
    t0_15 = sys_op("resolveBinding", "foo", false, t0_12);
    t0_16 = sys_op("getV", t0_10, "foo");
    t0_17 = sys_op("initializeReferencedBinding", t0_15, t0_16);
    t0_18 : list_push(t0_14, "foo");

This work methodology is going well for me so far, because it provides a balance between the sensation of tangible progress and the more mentally-taxing job of getting the fundamental principles in place.

What’s next?

• I’m working on a module loader for CommonJS modules, so I can get some basic multi-module code running. There are some interesting challenges here. For example, the imported module is allowed to call MCU.start in its root scope, thus suspending the process mid-require.
• I need to finalize the new design for the symbolic interpreter and get it working

---

1. The world is surprisingly ill-adapted to people who have an email address but not a physical address. ↩

2. I’ve even experienced an airport shower for the first time. I was surprised at how unpleasant it wasn’t — there was hot water, and space to put my luggage. But I did empty half a bottle of hand sanitizer on the floor before getting in because waterproof sandals are not something I own at the moment.  ↩

3. Actually, I only use the VR headset on flights — it’s great to disconnect from the real world and watch Netflix in a massive home theater in the virtual mountain-tops ↩

4. “Bare-metal living”, you might say. ↩

5. That is, my last published progress update. I’ve written several updates in the interim which I never published in the end because I felt they didn’t say anything substantial. ↩

6. Or for those who can’t natively think in terms of the programming language and thus have to mentally translate between their primary (natural) language and their “secondary” (programming) language ↩

7. Technical documentation as a form of inter-human communication is a different story. ↩