One of the statements about MetalScript that people seemed to disagree with is the idea that it can take as long as a week to integrate a C/C++ library into a firmware project.

TL;DR I’ve spent many years professionally doing both JavaScript and C/C++ firmware. In my experience, integrating and becoming familiar with a C/++ firmware library can take often days or in some cases weeks in some cases, while a JavaScript library often takes minutes to use.

Let me first say that if you are an expert in C and/or C++ firmware and have chosen not to use JavaScript for any major projects, then you may not be the person who will want to use MetalScript. You have spent years mastering a complicated craft, and although you have probably seen or worked with people who use JavaScript, you’ve chosen to stick with C/C++ because you probably believe it’s better. You may even look down on people who program in JavaScript — “real men” know how to do their own memory management, avoid signed integer overflow, and use CRTP to write code that is both well-structured and performant at the same time.

If that describes you, then keep doing what you’re doing. MetalScript is not for you, it’s for people who love JavaScript and who want to write real firmware with it.

Let me also say this:

I am a C/C++ firmware developer, and have been programming firmware for the last 20 or so years.

Up until about 5 years ago, my impression was that JavaScript was a pretty poor choice of language for various reasons, until I actually learned it and used it in real projects. Now that I’ve actually used it, I’m converted to its merits¹. I am qualified to compare C/C++ against JavaScript because I am proficient in both and have used both in many real-world projects.

The best thing about JavaScript is not the language itself. The language is good — it used to be pretty bad, but with ES6 and modern features, it is becoming a really good language to work in. But the thing that makes the JavaScript experience great is npm.

The npm package repository contains hundreds of thousands of packages for JavaScript. In addition to the packages themselves, there is a culture that drives useful conventions, such as:

• Packages generally have their source code in GitHub
• Packages generally have a readme file in the root written in markdown. And because everyone does it this way, npm and GitHub both display the readme on the main page for the package
• The readme typically contains a brief description of what the package does, as well as how to install it (even though the installation process is almost always the same)
• The readme often contains a set of examples to get you started
• The readme often contains a set of options for advanced usage, or links to proper API documentation.

Not all packages will be this well presented, but the vast majority are, and it’s hard to understate the importance of this conformity. It means that finding and getting started with a completely new library can happen within just a few minutes. The relevant information is all upfront, and the examples are typically self-contained so they can be pasted right into your code and they “just work”.

To try to demonstrate my point, I’m going to compare some examples. It’s difficult to come up with fair examples because firmware libraries are not going to be typically found in npm. So I will do this with two different examples — one of them using actual libraries and one of them a made-up scenario.

Example 1: Calculating a CRC

For the first example, I’m going to try to calculate a CRC in both C++ and JavaScript. This is something that is well suited to a third-party library, and so I expect to find code that already does it in both JavaScript and C++.

This comparison will probably be the best possible case for a C/C++ library. A CRC can be calculated without any platform dependence or customization. It should be as simple as finding a function online that does it and pasting it into the code. Let’s see how we go.

JavaScript

I’ll start with JavaScript, and will then compare the experience in C/C++.

• At 10:22 AM, I Google “npm crc calculation” (Note: I recorded these timestamps while doing it, but not while blogging, so as to minimize interference)
• Look at first result – a package called crc on NPM — open the page
• The first thing on the front page: the list of features for the library. Yes, this looks like what I want
• The second thing on the front page: the command to install it. At 10:23 AM I run the command npm install crc in my project folder. I have gone from “thinking that a library might exist”, to successfully installing it, in about 1 minute.
• The third thing on the front page: the example code to use it. At 10:23 AM (still), I create a test script with two lines of example code:

const crc = require('crc');
 
console.log(crc.crc32('hello').toString(16));

At 10:24 AM, I run the code – node test.js . It works.

But actually I didn’t want CRC-32, I wanted CCITT-16. I adjust the test code to crc.crc16ccitt(‘hello’) and it still works.

Finished by 10:25 AM — from imagining some functionality to having it integrated in 3 minutes. This is not unusual IMO, once you are familiar with the workflow and know where to expect everything by common convention.

In C

At 10:37 AM, the first thing I’m going to do is Google “CRC calculation in C”. There is no standard repository that I can search, so I’m open to anything on the internet.

I look at the first page of links. None of them jump out at me as what I’m looking for.

I look at the first link. Scrolling through it, I can see some diagrams, some code, and lots of writing. Perhaps if I want to understand CRCs, this is not a bad place to be. But really I would prefer it if someone else understood CRCs, and I just leverage their expertise.

Should I just copy-paste one of the example pieces of code? I read skim-read pieces of the document to try to get an idea of whether this is a bad idea or not. The fact that it says “bit by bit” as one of the headings makes me think that it’s leading the reader through the implementation and starting with a less-than-ideal implementation. Better not use that one.

What about the other code snippet they include in the article? It’s not clear what kind of CRC this is for. Should I read the article? Should I cut my losses and move to the next link? Should I copy-paste and hope this is the right one? Time is ticking, and this is a race.

10:41 AM. I cut my losses on this page and move to the next search result. It’s got code — that’s good. But again, it’s got a “simple example” and an “advanced example” — not examples of usage, but examples of CRC functions. What does simple and advanced mean? Does one do more stuff than the other? Is one more efficient than the other? Do I have to read the code to find out? Time is ticking.

Glancing through the code (now the 3rd and 4th pieces of somebody else’s code that I’ve had to look at), I see that the simple example doesn’t use a lookup table, and the advanced one does. Likely I’m on another educational page that’s trying to teach the reader about how to do CRCs.

Why is this kind of thing the first two search results? Surely people are more commonly wanting to use CRC code than to write and understand their own implementation? Does it say something about the culture of C that the top links on google for “C CRC calculation” are to help people to write their own implementation from scratch?

Maybe my search terms are poor. Maybe I should have used the term “ccitt16” in the search query. Maybe some other changes would also help? I remember that the next two search results are stack overflow questions — let me have a quick look at them before I go back to try other search terms.

[Bla bla bla] I’ve created a function to calculate a CRC16 checksum, but it doesn’t seem to be outputting correct values, [bla bla bla]

(I’m skim reading because I’m in a race against the JavaScript guy who integrated a working library in a quarter of the time it’s taken me not to get anywhere)

The guy wants to fix his function. The top answer has a bunch of explanation that I don’t have time to read, and then some code that is prefaced with “so, your function might look like”. Those are not words that inspire confidence in me. It sounds like his goal is to help the questioner figure out where he went wrong, rather than writing production-quality code that many other people will depend on.

Should I look at the other answers? Should I abandon these search terms and try something else? Should I look at the other SO question?

Let me have a quick glance at the other SO question before deciding.

Since CRC is so widely used, I’m surprised by having a hard time finding CRC implementations in C. [bla bla bla]

Totally agree with ya bro.

The top answer provides a bunch of links. A lot more reading, and a lot more implementations to choose from. But now at least we’re getting somewhere.

I actually landed up picking the implementation in the second answer of the first SO question. Not because it was carefully considered as the best choice, but because I was in a rush and it had a couple of nice properties at a glance:

• It was short, so I felt less intimidated
• The answer was only prefaced with 2 sentences, so there wasn’t much reading for me to do
• In one sentence I see the words crc16 CCITT
• In the other sentence, I see the word “tested” and a link (the link makes it official! ?)

10:50 AM. I paste the code into a C file and write a main function to test it. 

int main() {
  const char* str = "hello";
  short crc = crc16(str, strlen(str))
  printf("%04x", crc)
}

10:53 AM. I try to compile it, but GCC is not in my environment path. This has nothing to do with the library, so let’s just pretend I compiled it and it worked.

It took me 15 minutes, as opposed to JavaScript’s 3 minutes.

Postmortem

Speed to find library

Most of the time spent in C was spent finding the library. While it’s true that this will generally be slower in C than in JS (since there is no common convention and central catalog of such libraries), finding the library will never take a whole week, so this small example doesn’t account for the majority of the time that I claimed it takes to get a C firmware library integrated.

Why is that? What’s special about the CRC example that makes it unrepresentative of the norm?

I think the answer is that if you are picking an example from the subset of libraries that work both in today’s JavaScript world (i.e. it will be intended for server or browser) and also the world of C/C++ firmware, you are actually left with a small collection of libraries which do not exhibit most of the complexities that arise in firmware, which biases the comparison. As I mentioned earlier, a CRC-calculating function is much easier to make platform-agnostic, and so should be the best possible candidate for a hassle-free library in C.

Speed to install/integrate

I’d say that both the C and JS versions in this example were pretty similar to integrate into a test script (representative of a larger application). However, from experience, I’d say that almost every npm library is just as easy to integrate as this CRC library, typically only taking a few minutes to get going with the basic examples. In JS, I believe this example is representative of the general experience.

In C however, we’ve picked an example that is trivial to integrate — just copy and paste.

Most C/C++ firmware libraries are integrated at the source code level because of the wide range of possible target architectures, and the source code often requires extensive customization or dependency implementation or port layers in order to get it to work for your particular setup (and reading through documentation to understand how to do that). To compound the issue, the reality is that many firmware compilers don’t support the full C/C++ spec, normally for performance or architecture reasons. These are not criticisms of C/C++ per se, but nevertheless, are part of the typical experience of using C/C++ in a firmware environment.

Confidence

How confident am I in the JS vs C library?

The C “library”, if we can call it that, is some code in a Stack Overflow answer that one guy wrote. His testing involves running it a few times and checking that it matched some online web page. I don’t feel great about that. Maybe it’s okay because other people reading it may have spotted the problem and put in a comment if there was something wrong with it.

On the other hand:

• The npm crc package has been downloaded over a million times in the last 7 days. All those feet treading on the same path will harden that path. If there any bugs, they will be found quickly.
• It comes standard with a suite unit tests, and both the GitHub and npm pages display that all the tests are passing.

Furthermore, the package manager allows me to quickly update my dependencies, to make sure that I get the latest bug fixes at any time.

Documentation and Ease of Use

I think this answers itself. There is no documentation and no example code with the C version — it is a code snippet in a SO answer, so what do you expect?

The JS version has exactly the documentation that you’d expect from most npm libraries — it is concise and describes the key things you need to know in order to use it. It doesn’t try to tell you the theory behind CRCs, or anything that isn’t directly relevant to being productive as quickly as possible.

Am I talking about the language or the package manager?

I’d like to just clarify something because I know this is going to be brought up. I say that I’m comparing C/C++ vs JavaScript but then go on talking about things that are not part of the language at all (Google, Stack Overflow, npm, culture and ecosystem). Is that valid?

Yes, I think this is valid. When you choose to develop in JavaScript or C/C++, you’re not just adopting a language. You’re adopting all the tools, community, and culture surrounding the language. Productivity is affected by all of these factors, and they come together as a whole. You can say all you want about how you think C++ is a better language if you think that, but at the end of the day, it’s about getting shit done, and the JavaScript “whole” is better for that then the C++ “whole”.

Example 2: A modem driver

Here, I’m picking an example that adds complexity more typical of firmware development, but the tradeoff is that this example isn’t real — it’s merely a vision I have for the future. I don’t think the current state of JavaScript firmware development is mature enough for this to be a reality today.

For this example, I will assume the following hypothetical scenario:

• We have a product that has an MCU and a u-blox cellular modem
• Due to a shortage of UARTs on the MCU, the product connects the modem to a UART extender
• Objective: connect to the internet to send an HTTP POST, receive the response JSON, decode, and output to the message therein to the debug UART

Before I even start, if you are a firmware programmer, give a moment to think about how you would do this. If I contracted you to write firmware for a device that does this, how long would it take you to write?

The Vision

There are two domain-specific pieces of information that we absolutely need to specify somewhere in any firmware, no matter what language:

1. Information describing the behavior we require, such as the fact that at startup we want to connect to the internet, POST a message, and print out the response
2. Information describing the device configuration, such as the fact that we have a u-blox modem, and the fact that it is connected on the multiplexer, etc.

We can summarize the required behavior with the following hypothetical JavaScript code:

// app.js
import * as request from 'request-promise-native'; // third-party library to perform HTTP requests

export async function run(device) {
  await device.modem.connectToInternet();
  const reply = await request({
    url: 'http://my-service.com/test-url',
    method: 'POST',
    json: 'please give me a message to display'
  });
  console.log(reply);
}

We can summarize the required device configuration with the following hypothetical JavaScript code:

// device.js
import { UBloxModem } from 'ublox';
import { Max14830 } from 'max-14830'; // UART extender driver

export const uartExtenderI2C = mcu.i2c('G7');
export const uartExtender = new Max14830(uartExtenderI2C);
export const modemUart = uartExtender.uart(3);
export const modem = new UBloxModem(modemUart, 'LISA-U200');
export const debugUart = mcu.uart(2);
export const debugConsole = new UartConsole(debugUart, { baud: 115200 });

Then we also need some glue code:

// main.js
import * as device from './device';
import { run } from './app';

// The console we want to use for output messages
global.Console = device.debugConsole;

// The device to use for connecting to the internet
global.internet = device.modem.internet();

// Transition from compile time to runtime
mcu.start();

run();

Perhaps the reality won’t be so easy, and I’m oversimplifying it. But I can imagine getting a library like this off npm and being able to get working with it on a firmware device within a few a hours.

How long would it take to do the same thing in embedded C? Days, weeks, months?

Conclusion

In between writing this article and publishing it, I ran into another real-world example. I needed a modbus connection from my C firmware to my electron JavaScript application, and I was implementing both sides. The JavaScript side was working within an hour, as one would expect. The C side took days of implementing hundreds of lines of porting layer, managing states, and banging my head against the wall.

The reality is that JavaScript is simply a much more productive tool to use, and a large part of that is because of how easy it is to reuse third-party code and to share your own so that others can reuse. 

---

1. Although if you want to use it for a real project these days, please use TypeScript so you can get static type checking ↩

A MetalScript progress update, for those who are interested. This one is all boring technical details — if this is your first time on the site, perhaps read something more interesting like what MetalScript is and the reason why I’m creating it. Or for those who like technical details, read on…

[Edit: Previously this had “November 2018” in the title, even though I wrote this in September — who knows what was going through my head when I wrote that.]

Stack Unwinding

To recap, last time I said I had my IL compiler compiling the following code file to IL and getting it to run in the virtual machine:

// blinky.js
MCU.start();
setInterval1

Phew! That’s quite a list. I must point out though, that this isn’t even comprehensive. Initializing the realm is a lot more work than implied by the little IHDR rectangle in diagram. Similarly for most other operations — I’ve only shown the highlights here.
The main point I want to convey is that even though MCU.start() appears to be the first thing executed when you look at the source code, there is actually a lot of VM work leading up to that point, and by the time the VM is suspended at MCU.start(), the VM stack is already 5 frames deep. It is at this point that the MCU hardware must take over — the entry point for the MCU reset interrupt vector is the continuation of the MCU.start() call2. But ideally we do not want to somehow translate the VM stack into the corresponding hardware stack at the point where the program starts. We could, but it would be significantly more difficult to compile this way.
Instead, the “unwinding” process that I’ve finished now translates the functions in the call stack to continuation passing style (https://en.wikipedia.org/wiki/Continuation-passing_style). These constructed continuation functions are just normal IL, whose behavior starts where the original function left off.
This isn’t as easy as it sounds. It has to work for all valid user code, including user code that might look like this:
function myFunc() {
  const thingsToDo = [];
  thingsToDo.push3;
  // (... add more things to do ...)
  while (thingsToDo.length) {
    const thingToDo = thingsToDo.shift();
    thingToDo();
  }
}
myFunc();

The challenge in the above code is that the continuation has to continue in the middle of the loop. It has to restore the state correctly, while resuming the old behavior on the second iteration of the loop (the first iteration is split between compile time and runtime, and the second occurs completely at runtime).
Symbolic Interpreter
The phase I’m working on now is what I call information threading which is done by something I call a symbolic interpreter. This is the part where the compiler tries to understand the code that’s been written, in the same way that a human might mentally follow through the code.
For example, in the blinky example, a human following through the code will know what the call to setInterval refers to — they know the state of the machine, they know that it contains a setInterval JavaScript function, and they know whether or not they’ve done something nasty to that function such as replacing it with their own wrapper that does something different to the standard setInterval function. So a human following the code would be able to predict with 100% accuracy what the code should do (and so can a machine).
But there are cases where you can’t predict the behavior exactly. For example, calling toggle. You can predict what function is called when you invoke toggle, but you don’t say whether it will turn the LED on or off — at different times the same line of code can have different effects, depending on previous runtime state. This just means that you need to record that state somewhere at runtime.
The symbolic interpreter exactly this. It keeps track of the state of the program over “time”, keeping track of what can and can’t be known at each point in the program. This information will later be used to emit an implementation of the source code in the target language (in this case LLVM IR).
I have a lot of the symbolic interpreter up and running. It’s currently able to fully interpret simple IO-free programs, meaning that if it’s given a script that doesn’t perform any IO (e.g. one that doesn’t read or write to GPIO for example), it can infer the behavior of the program completely.
But in the case of IO-free programs, there’s no observable behavior at all (observable behavior requires IO). Interesting things can only happen when you hit an IO boundary, such as turning on an LED, sending commands to a robotic arm, or writing “Hello, World!” to the console. So this is what I’m working on at the moment — IO boundaries for the symbolic interpreter.
What’s Left?
I’m currently working through the “Blinky” example. I want to get the blinky code going all the way through the compilation pipeline, as in the following diagram (blocks in green are complete).

I don’t think the type assignment phase will be difficult. Taking the output from the symbolic interpreter, it should be easy to have an algorithm choose some basic binary types. Likewise, I’m optimistic that the LLVM IR emitter will be easy, given the chosen binary types. I’m not referring to the complete implementations for any of these phases. I only need to reach a proof of concept for the blinky example first. This will give me a chance to go back and review the good and bad decisions I’ve made, before fleshing out each phase to fully support all cases.
There is a little bit of work not implied by the diagram above, such as leveraging LLVM to produce output object files, linking, and developing a minimal runtime (event loop and GC). But I think these will be relatively straight forward for the first iteration of POC.
The above work is all to get to a minimal POC where I can demonstrate end-to-end compilation of at least one input file. Then as I’ve said in previous posts, it will be somewhat longer to reach MVP.
News: TC53
I’m very pleased to say that I’m traveling to Boston4 next month to participate in the inaugural meeting of ECMA TC53, a new “technical committee” (TC) being established for JavaScript on “smart wearable systems and sensor-based devices”. I’ve been invited to speak a bit about MetalScript, among other things, and so I’m working hard to reach a good milestone before the meeting. I’m incredibly excited to go and be part of what will certainly be a group of highly intelligent people paving the way forward for a better future5.
 
 
) => gpio.pin('B4').toggle(), 500);

I said that next up was the “unwinding” phase — taking the suspended virtual machine state and unwinding the virtual stack and register states to create a unit of code that doesn’t require any stack or register information. I’m glad to say, this is complete!

Let me explain how it works. At the point where MCU.start() is invoked, there’s actually a bunch of things in the VM stack, as illustrated by the following diagram:

 

Most of this, unsurprisingly, is not user code. I’ve used blue to represent system functions (aka ECMAScript “abstract operations”), and red to represent user functions. Here’s a brief explanation of some of the system functions:

• RunJobs – this is the entry point to an ECMA application and has the main event loop that we all know and love.
• IHDR (InitializeHostDefinedRealm) – this creates all the “realm” objects, which includes things like the “Array” object and the “Object” object — things that are “built in”
• EnqueueJob – RunJobs gets the JS source text and calls EnqueueJob to add a script evaluation job to the job queue (for each entry JavaScript file, although in most applications I’ve seen there’s only one such file, normally called “app.js” or something).
• ScriptEvaluationJob – this is a job wrapper in the spec that is used to evaluate a previously-enqueued user script
• Parse (ParseScript) – this parses the script and returns information required to evaluate it.
• ScriptEvaluation – given information from the parsing stage, ScriptEvaluation sets up the context needed to run the main script code, and then runs it.
• GDI (GlobalDeclarationInstantiation) — initializes variables and functions for the script (recall that JavaScript variables and functions are hoisted)
• ScriptBody – then we finally get to executing the body of the script — the actual user code.
• Call – The user code invokes MCU.start(), which is a JavaScript function call, and there are a number of system-level things that need to happen for calls, such as checking that the target is callable, deciding what this-value to use, etc. All this work is encapsulated in the “Call” system function ((There is a separation in the MetalScript compiler between IL functions and JS functions, and likewise between IL function calls and JS function calls. IL does not have or need closures, variadic functions, etc, and is used to represent both user code and ECMAScript “abstract operations”. ↩

• It is exactly the continuation of the MCU.start() call. Unlike in C, there are no operations on the MCU that precede the user code. Operations like initializing the MCU clock source or copying .data into RAM are logically part of MCU.start() and can be parameterized and configured accordingly in future by options passed to the MCU.start() call. I’ve said this before, but I feel it’s important enough to mention again ↩

• ) => MCU.start( ↩

• Boston is 17,000 km from here in Melbourne — almost as far away as is possible without going into outer space. I’m going to be spending about 3 times longer on the airplane than in the meetings! ↩

• In a dream I had about it a few nights ago, I could swear that Gandalf was one of the members of the committee. ↩

For those following along, here’s an update on my progress on MetalScript in the last month.

It’s been a bit of a slow month for MetalScript, just because I have a lot going on and MetalScript takes a bit of a backseat to other important things. I expect things to stay like this for a while longer.

Website

I spent a weekend hacking together a stub of a website. See http://metalscript.com. It’s really a placeholder for a website, not a real website. I spent a morning giving myself a crash course in Hugo and Slate for the first time, as well as trying to remember how to use Blender to create the somewhat rudimentary graphic depicting “JavaScript on a Microcontroller”. And lots of CSS hacking.

The website is not meant to attract customers, because of course there is no product yet. It’s meant to be a representation of the grand vision of MetalScript. It paints the picture of what I’m trying to get to, and so helps me stay focused on the goal. It’s also a place where I can start collecting user documentation so that when I launch the MVP there will already be documentation in place, and the documentation doubles as a spec of what it needs to do.

I didn’t spend much time on it, and it leaves lots to be desired. The markdown at the bottom of the home page is particularly jarring, and the whole thing is not very mobile friendly. But I’ll improve it over time.

I also couldn’t make up my mind on the color scheme, and landed up changing it a number of times. Originally I was thinking pink/orange (excuse my primitive “engineer’s vocabulary” for colors) because I think it has really awesome energy to it, and paints the picture of something new and exciting and different…

But when I was playing around with the MCU graphic, I started with yellow instead because it seemed to be “the JavaScript color” (if you Google images for JavaScript).

But quite frankly, yellow is ugly. Sorry to those who like it. It’s such a stark, flat color, and doesn’t say anything I want it to for MetalScript.

I went through a number of greenish shades which I dismissed. And while it’s probably not final, the website is currently based around an aqua blue.

Not everything matches, because I kept changing my mind about exactly where on the spectrum between blue and green I wanted to be.

Blue is nice because it’s calming but professional. This is very much what I intend MetalScript to be — a tool for professionals, but one without the discomfort and complexity associated with C and C++.

I also spent embarrassingly long on the front image. I was trying to convey the idea of putting JavaScript “onto” or “into” a microcontroller, but was worried about misrepresenting what I am doing as if I am selling a microcontroller with a JS brand (I can’t assume that anyone visiting my website has prior expectations about what MetalScript is). My solution in the end was to create a somewhat unrealistic and stylized representation of an MCU, to try help get that idea across that the MCU in the picture is not a real thing I’m trying to sell.

Just because it’s unrealistic and stylized doesn’t mean it’s not a challenge for a Blender amateur like me. There are lots of details I tried to get right that a professional may have achieved better results in a lot less time. Take for example this comparison between an earlier draft (left) and a later one (right)…

(Click for a larger view). I wanted to convey the idea of JavaScript “radiating” out of the microcontroller, as if the touch of MetalScript imbued it with superpowers. In the image on the left, the JS logo is actually radiating, but radiation (light emission) is pretty boring-looking in reality. On the right, I decided to just “pretend” to radiate by actually constructing emissive volumetric rays floating above the “JS”. This was much harder than I thought, such as trying to provide an alpha (transparency) gradient from the bottom to the top so that the rays appear to just “fade out” gracefully as they move further from the apparent source, and not abruptly end when the volume ends. Half the time while doing this, I landed up having anti-emisive rays, if that’s a real term, that sucked in blue light (looking red-ish) — a side effect of not providing the correct origin point or 3D orientation for the gradient volumetric texture.

(Other differences to note are the corners, the color, and the reflectivity of the surface).

By no means is anything final. I will likely change my mind several more times before MVP, and anyway, as soon as there is money I will probably get this all looked at by someone with better skills than mine.

Technical

But to the technical stuff I’ve been doing in the last month…

IL Compiler

I’ve added the necessary features to my IL compiler (compiles JS to IL) to support the following traditional example:

// blinky.js
MCU.start();
setInterval(() => gpio.pin('B4').toggle(), 500);

This is the hello-world of microcontrollers, to create a blinking LED. Of course, nothing yet blinks since there is a lot of work left to do on the compiler before I have actual machine code running on the device. But nevertheless it demonstrates a number of key features that the IL compiler needs, such as the ability to instantiate closures, call functions, access properties, etc.

Virtual Machine

Probably the biggest piece of work done in the last month is getting the virtual machine up and running.

It’s been tested printing “Hello, world!”, which is an important milestone. You might think that “Hello, world!” is a stupidly simple example, but actually the virtual machine needs to execute hundreds of IL instructions to achieve that simple task, most of which is setting up the realm (i.e. the global builtin “stuff” that JavaScript needs).

Perhaps more interestingly, I have the above blinky example running on the virtual machine. I say “more interesting” because the blinky example demonstrates another key feature: suspending the virtual machine before the next stage of compilation. So what I have currently working is taking a program (blinky) that runs in the VM until the suspension point, and then the suspended machine is serialized as IL, including registers, stack, and heap allocations (the virtual heap includes functions). That’s a pretty cool point to be at!

[Edit] Re-reading this, I think I should clarify what I mean by “virtual machine”. I’ve previously made it clear that MetalScript programs don’t run on a virtual machine, so why is there a virtual machine? The reason is simply because in the two-phase execution model of MetalScript, where a program is executed at both compile time and runtime — it is the compile-time execution that is done on a virtual machine. The runtime execution is bare-metal.

What’s next?

The next stage is a post-processing step that will be executed on the suspended virtual machine state, that implements the behavior of “resuming” the JavaScript job that was suspended when MCU.start was called. This post-processing step is somewhat like a continuation passing style transformation. All the functions that were suspended in the call stack will be turned “inside out” so that their entry point is where they left off when the virtual machine was suspended.

Believe it or not, the VM stack is 5 levels deep at the point where the blinky example is suspended, even though the call is done from the root scope of the script. This is yet another illustration of the fact that “it’s more complicated than it looks”.

The result of this step will be a body of IL that does not need to contain any definition for the current register states or call stack, which will be much easier for the next phase of the compiler to deal with.

I’m also finalizing the design of what I call the “symbolic interpreter” which essentially will step through the IL to figure out what it does (and thus how to implement it in machine code). This is the most complicated and critical piece of the whole project. As I’ve said in previous posts, through the side experiment I called “MiniLanguage” I’ve gone far enough with this idea to feel confident of the direction I’m going, but MetalScript is an order of magnitude more complicated than MiniLanguage so it’s going to take some time to iron out the details.

I’ve also spent some time thinking about how setInterval is going to work. There are any number of possible approaches I could take, but I’ve decided that the quickest path for the moment will be to implement it as “special” behavior that is part of the bare-minimum runtime library. Previously I said that the runtime library would only have the GC and event loop, but now I think that timers are also going to benefit from being built-in in this way. Among other things, this means I can postpone thoughts about implementing interrupts in pure JavaScript (which is completely plausible, but I don’t want to bloat the MVP with features that are not critical to using MetalScript, when there are perfectly reasonable pragmatic alternatives).

Looking further into the future, my plan is to keep pushing the blinky example through each phase of the compiler until I eventually get it out the other end and have an actual blinking LED. This would be a point of massive celebration.

I intend MetalScript to compile JavaScript code as either an executable or as a library. When I introduced the idea of MetalScript in a previous post, I gave a simple code example for how I imagine it to look when MetalScript compiles (and runs) a JavaScript program as an executable. In this post I’d like to present how I envisage it to look when a JavaScript program is compiled as a library, and a few juicy details about how this will work.

What do I mean by “library”?

Different people may mean different things when they say “library”. What I mean, in this context, is that the MetalScript compiler will be able to take a JavaScript file input, and produce a corresponding .o object file or .a static library, which can then be used by a traditional linker to merge the compiled JavaScript code into a larger compiled project.

Why would anyone want to do this?

This would certainly be an “advanced mode” use of the compiler. It brings back many of the things I hate about C/C++, such as linkers, intermediate files, etc, and so I would never recommend it as a starting point for new projects. The use case would be for people who have an existing legacy C or C++ codebase and who want to bring in a “sprinkle” of JavaScript to see how it feels, or perhaps even to bring existing NPM libraries into a C project.

What will it look like?

Let me get straight to an example. Here is one which would compile to a library that adds two numbers together. I’ve written it out intentionally verbose for expository purposes.

// mylibrary.js
import * as c from 'c-ffi';

function add(a, b) {
  return a + b;
}

c.exportFunction('foo',       // Exporting a function with the public name "foo"
  add,                        // The function we want to export
  { 
    ret: c.Int32,             // The return type
    args: [c.Int32, c.Int32]  // The argument types
  }
));

And then to compile, you would execute something like the following command line:

metalscript --library -o mylibrary.a mylibrary.js

To use it, you might link it alongside the following C code:

#include <stdio.h>
#include <stdint.h>

extern int32_t foo(int32_t a, int32_t b);

int main() {
  int c = foo(1, 2);
  printf("The result is: %i\n", c);
  return 0;
}

Let’s break it down…

First, I consider that there will be a built-in MetalScript module which here I’ve called c-ffi (short for “C foreign function interface”), which provides the helpers and IO functions that are required to create library definitions allowing C to interface with our library. Think of how node has builtin modules for accessing the “outside world”, such as the fs module for accessing the file-system. The c-ffi module is providing analogous access to the outside world, except that here, the “outside world” is the surrounding C code in the larger project (for which MetalScript is not responsible).

c.exportFunction(name, fn, signature)

The c.exportFunction function¹ creates a new definition in the library symbol table — i.e. it creates a new symbol with external linkage. This symbol needs to have a name and in the above example I’ve given it the name 'foo'. I’ve called it foo and not add, just to make a point: the name in the symbol table is unrelated to the JavaScript idea of a function name. To drive this point home further, consider that functions in JavaScript don’t need to have names at all, so the example could have been written like this (notice the lambda definition):

import * as c from 'c-ffi';
c.exportFunction('foo', (a, b) => a + b, { ret: c.Int32, args: [c.Int32, c.Int32] });

I don’t understand. Does the compiler recognize c.exportFunction as special syntax, like C’s extern keyword?

No, the variable c here is a JavaScript object and c.exportFunction is a property on that object which is a function, just like any other JavaScript method. There is nothing special about the syntax c.exportFunction. You could have written c["ex"+"port"+"Function"] instead of c.exportFunction and it would have worked exactly the same.

What may be confusing to you, is that we have what appears to be a mix of compile time and runtime code in the same file² —  the c.exportFunction statement executes at compile time, while the a + b expression executes at runtime.

Recall from my initial post on MetalScript that this is one of the key features of MetalScript — the program transitions between two phases, initially executing in the compiler and then moving to execute on the target device. Normally, when MetalScript is in compiling a whole program (not just a library), the transition from compile time to runtime happens when the program calls mcu.start, and the entry point to the runtime program is the continuation of where the running process left off when it called mcu.start. In the case of a library however, the programmer has chosen to go the “advanced route” and implement the global entry point (e.g. the reset interrupt vector) in another language, so the JavaScript code won’t have an mcu.start statement. What this means is that the whole script executes at compile time to completion, producing the export definitions as an artifact (as a result of calling c.exportFunction), and then the compiler uses the exported definitions as the entry points to the static library that it produces.

If you are used to C or C++, this might make your hackles stand on end. You might think that this has to use some ugly, hacky, complicated and fragile heuristics to figure out what’s compile time and what’s runtime, and to separate these “magically”, until it doesn’t work and everything breaks³.

But I’m here to assure you that I’ve gone into sufficient detail in the MetalScript design to know it won’t need to resort to such hackery. The code you write will all be interpreted according to the ECMAScript standard.

Here’s an example that may help put your mind at ease. Let’s say that instead of creating a library with one function foo that adds two numbers together, let’s say we want to create a library with 10 functions that add the numbers 1 to 10 to a given argument respectively, as represented by the following C code:

extern int add1(int x); // returns x + 1
extern int add2(int x); // returns x + 2
extern int add3(int x); // returns x + 3
extern int add4(int x); // returns x + 4
extern int add5(int x); // returns x + 5
extern int add6(int x); // returns x + 6
extern int add7(int x); // returns x + 7
extern int add8(int x); // returns x + 8
extern int add9(int x); // returns x + 9
extern int add10(int x); // returns x + 10

We can create such a library with the following JavaScript, which again I’ve written verbosely for illustration:

import * as c from 'c-ffi';

function createAddN(n) {
  function addN(x) {
    return x + n;
  }
  return addN;
}

const exportSignature = { ret: c.Int32, args: [c.Int32] };

for (let i = 1; i <= 10; i++) {
  const exportName = `add${i}`;
  const implementation = createAddN(i);
  c.exportFunction(exportName, implementation, exportSignature);
}

Some key points I’d like to highlight about this example:

• The call to c.exportFunction only occurs once in this code (lexically), but because it is executed 10 times, there are 10 exports in the library. It is not some mystical static analysis that is producing these exports, but a real JavaScript engine executing code according to the defined standard.
• The names in the symbol table in this example are dynamically computed.
• You may notice that the program essentially exports addN 10 times, and each time it has a different behavior (adds a different number). This is because, in JavaScript, lexical functions (the code that makes up your function) are actually like function “templates”, which are instantiated every time the surrounding code is executed. Since createAddN is called 10 times, there are actually 10 distinct function instances of the addN function, each with a different closure, and each of these can be exported separately⁴.
• A more subtle point is that the value for the parameter n in createAddN is part of a closure that is carried over from compile time to runtime. Again, this is perfectly valid JavaScript code, and MetalScript doesn’t use hackery or trickery, it will work just fine.

If you aren’t yet convinced that this is a good idea, let me give you one more example. Let’s say that for debug purposes, you wanted to log any calls to exported functions. You could add the following at the beginning of your JavaScript program:

// Replace c.export with our own definition that logs incoming calls
const actualExport = c.exportFunction;
c.exportFunction = function (name, fn, signature) {
  function wrapper(...args) {
    console.log(`Calling function ${name} with arguments ${args}`)
    return fn(...args)
  }
  actualExport(name, wrapper, signature);
}

This snippet of code wraps the c.exportFunction function with our own implementation. Each time our own exportFunction function is called, it creates a wrapper function and exports that wrapper function instead. The wrapper function calls the original function, but it also logs the call to the console so we can see it happening.

If this doesn’t blow your mind, maybe it’s because you’re coming from “JavaScript land” and think that this example is obvious because it’s just vanilla JavaScript.

Does this mean I can export at runtime?

No (not surprisingly).

You may have thought that since c.exportFunction is a function, and functions can be called at runtime, that we can therefore call c.exportFunction at runtime to create new export definitions while the program is running — but this isn’t the case. Or maybe you thought that since this couldn’t work at runtime (because you know that libraries can’t change after they’ve been compiled), that the whole idea is invalid, but that isn’t true either.

The distinction between compile time and runtime is not invisible to the user code (nor should it be). The MetalScript APIs have different behavior depending on what phase the program is in, as one would expect. For example, the c.exportfunction will only work at compile time (the call will throw an exception at runtime), and reading GPIO will only be meaningful at runtime.

This doesn’t violate the spec in any way — it’s perfectly valid for the behavior of APIs to change as the state of the world outside changes. For example, in node I can open a file once and it might work fine, and then delete the file (i.e. change the world state, either through code or by hand), and then the next time I try to open the file, the API will give an error.

In MetalScript, you can imagine that your program is running inside a glass teleportation capsule, and there is a button in the capsule that allows you to teleport from “compile-time universe” to “runtime universe” (the button is mcu.start). There is also a pending job in a lowest-priority JavaScript job queue which will perform the teleportation automatically if your program never presses the button (i.e. your running program will be teleported automatically when all the scripts run to completion). The things you can see out the window of the capsule change at the moment of teleportation — the scaffolding around the launch pad of the capsule in compile-time-world vanish, and you find yourself inside the living beast of the MCU⁵:

• The module resolver disappears, so require statements can no longer be used to load dependencies (you need to get all your dependencies “on board” before launching the capsule)
• The c-ffi export definitions are now cast in stone (or flash to be more accurate). You can no longer add or modify export definitions.
• Similarly, the memory layout, choice of GC algorithm, etc. all become fixed at the point of launch. In the teleportation analogy, you can think of this as choosing your life support for the journey before you leave.
• The heartbeat of the beast comes to life — timers start ticking, and GPIO becomes available.
• The outside world can start talking to you — functions that you previously exported may now start receiving calls⁶.

The function signature

We’ve talked about how the symbols are registered in the library export symbol table, but we haven’t yet talked about how the “signature” of the function works.

It’s quite simple: the signature provided to c.exportFunction specifies the binary interface through which C functions (or any external code) can call the JavaScript function. I like to think of it like the following diagram. There is “JavaScript land”, made up of your library code and its imported dependencies (or builtin-dependencies), and there is “C Land”, which is all the other code in the system which is not under the control of MetalScript⁷.

Calls from C into JavaScript are done by declaring the extern definition in C, which the C compiler uses to understand the signature of the call (how to pass arguments etc), and provides information to the linker about the name of the thing that is called. On the JavaScript side of the fence, the export information given to c.exportFunction is used by MetalScript to implement a stub that receives the arguments from C-Land, performs any parameter conversions, and forwards the call to the given JavaScript function.

The types that are provided in the signature, such as c.Int32 dictate the semantics of the conversions/marshaling between C and JavaScript. The Int32 argument type is one that is received from C as an int32_t and promoted to a first class JavaScript number value. In the opposite direction, when the Int32 result is sent back to C land, the dynamically-typed JavaScript value is coerced to an int32_t using familiar JavaScript rules, such as those used when writing to an Int32Array.

What it doesn’t look like

I’d like to conclude this article by highlighting what the MetalScript-way-of-doing-things doesn’t look like:

• It doesn’t require a manifest file or configuration files to describe the exports
• It doesn’t require many lines of hand-written boilerplate C code to perform conversions
• It doesn’t require any special C macros or classes, such as in native abstractions for node (NAN)
• It doesn’t force you to write pseudo-dynamically-typed C code, where you need special C function calls to query the type of a JavaScript value
• It doesn’t require any language extensions to JavaScript to handle binary types or export definitions
• It doesn’t require you to have an understanding of some virtual machine that the JavaScript code is running on (because there isn’t one)

If you’ve never tried to interface between a JavaScript script and native code (e.g. in node or espruino or something else), then perhaps the above points don’t make sense to you. You may think, “why would you need special macros or language extensions, or reams of boilerplate code?” And to that I say, “I agree”.

P.S. There are lots of details I haven’t covered here. If there’s something specific that you find confusing or disagree with, feel free to leave a comment below, or drop me an email.

---

1. Obviously the names and structures may change. ↩

2. If it wasn’t confusing you, perhaps it is now that I’ve brought your attention to it. ↩

3. For example, I believe that webpack uses what I would call “hacky” processes to find “require” statements statically, and so it would be tricked by any number of legal JavaScript codes, such as if require were aliased with a different name ↩

4. You could also export the same function instance 10 times if you wanted to, for example if you wanted to export the same JavaScript behavior with different C signatures, such as exporting an Int32 variant and a Double variant of the same function ↩

5. If you are compiling a MetalScript stand-alone program, then you will be lucky enough to be inside “the beast” of the MCU. If you are compiling as a library, then “the beast” will be a foreign C program, which is far more vicious and its insides are toxic — radioactive with stray pointers, and tumorous growths of leaked memory. ↩

6. And interrupts may start firing, which is a topic for another article ↩

7. To tie this back to the teleportation capsule analogy, the outer circle in the diagram is the capsule, and what I’m calling “C Land” is the outside world after teleportation. The export definition is part of the “capsule window” through which we can interact with the outside world ↩

I recently posted about MetalScript, a JavaScript compiler I’m creating to allow people to write firmware in JavaScript. One of the responses I received is that there already exists a solution for running JavaScript on a microcontroller: Espruino. Why am I creating something new?

TL;DR: Espruino uses an interpreter, which takes up memory, runs slowly, and ES conformance is sacrificed for speed. MetalScript is a compiler, so all the memory is available to the application, which will run quickly, and there is no need to sacrifice conformance for speed.

For those who don’t know, Espruino is a JavaScript interpreter that is designed to run on an MCU. This means that it can take JavaScript source code (either on the serial port as you type it, or stored as text in MCU flash), and the interpreter running on the MCU itself will execute it to produce the desired behavior. Espruino actually sell a joint solution which includes both the hardware and the interpreter running on it, but in this article I will be focusing only on the interpreter.

Let me start by saying that I think Espruino is great at what it does. It really does achieve a lot in a comparatively small code footprint. Espruino apparently uses 100-200kB of flash (ROM), plus whatever size you need for storing code, and apparently it can operate in under 8kB of RAM. This is pretty impressive, and I commend Gordon and the Espruino team for their ability to pull this off.

But having said that, I’ve considered Espruino for two different projects in the past and chosen not to use it for either of them, for reasons which I’ll unpack here. The one project was an IoT smart parking meter, and would have used Espruino as a scripting engine for user workflows. In this scenario, there was a mature existing codebase of firmware which we had no intention of changing, but we wanted to augment it with the ability to run simple scripts for customization purposes.

The other project was an IoT gateway device for a sensor network. We had existing hardware, but the existing firmware code was in a serious state of disrepair and we were looking at rewriting it from the ground-up. Here I was considering the possibility of JavaScript being the primary language for the firmware.

Let me make it clear that this is not a comparison between tools, but rather about why Espruino did not address my specific needs, and how I intend to make a tool that would have addressed those needs, and so might help others with similar needs in future.

VM Size

In both projects where I considered Espruino, we were limited by the amount of RAM and ROM available. But particularly in the parking meter solution, the size and memory overhead of the VM was a concern. While its very impressive that Espruino only takes 8kB of RAM and 100kB of ROM, this would still have consumed most of the resources on both devices, leaving very little for anything else.

Particularly on the smart parking meter, it would have been completely disproportionate to have a single feature consume this level of resources, bearing in mind that we only needed to run one or two short scripts to manage the flow between different screens.

How will MetalScript address this?

MetalScript will compile code to run bare-metal, meaning that it doesn’t require any supporting runtime infrastructure such as a VM or interpreter. This means that like in C, an empty JavaScript program will consume almost no RAM or ROM — probably in the order of 1 kB of ROM and 100 B of RAM, for the event loop, garbage collector, and bare-bones startup instructions.

Additionally, RAM usage of a firmware application will be made smaller because the compiler can infer types. For example, an integer field or variable number might take 4 bytes (as opposed to Espruino’s 16 bytes). Type inference will not always succeed, so real-world performance will likely be somewhere in between.

Code Size

Since the performance page on the Espruino website deals with this issue specifically, I’m going to take a little more time to address it. The page has quite a nice demonstration of the program sizes for an example function that draws a Mandelbrot fractal. It provides compelling evidence that JavaScript source code can be similar in size than the compiled binary produced by GCC from an equivalent C function. When minified, the size of the JavaScript source text is close to half the size of the GCC output.

This means that if you are taking C functions and writing them in JavaScript, like in the example, you could actually save on flash memory by writing your code in JavaScript in Espruino.

But I think this misses the point of what makes JavaScript so great. I don’t want to write JavaScript so that I can write C-style code in JavaScript syntax. I want to leverage all the things that make JavaScript great.

Let me use a different example to highlight what I mean. In C, if I want to write a function that converts a string to title-case, I might do it like this:

void titleCase(char *sentence) {
  bool capitalizeNext = true;
  for (char *c = sentence; *c != 0; c++) {
    if (capitalizeNext)
      *c = toupper(*c);
    capitalizeNext = (*c == ' ');
  }
}

int main() {
  char sentence[] = "hello, world!"; // Does this initialization work? Not sure.
  titleCase(sentence);
  puts(sentence);
}

If I was to do it in JavaScript, I would do it like this:

1. I happen to know that there is an NPM library called change-case which can do this. I found this by spending about 30 seconds Googling for it sometime back.
2. Look at the readme on the main page — yes, it supports what I want
3. yarn add change-case ¹
4. const { titleCase } = require(‘change-case’)
5. console.log(titleCase(‘hello, world!’));

The JavaScript way is better for a developer for many reasons, which JavaScript developers are probably all familiar with:

• It’s less of your code, so it’s more maintainable (other people can maintain the dependencies, and their work will be leveraged by the 1000s of people that use their library).
• The writers of the library have probably spent time to think about edge cases that you may not have thought about.
• The C version is brand new code, and has not had time to mature to iron out any bugs and edge cases that come with real-world usage.
• The JavaScript version is used by many people — there are 762 dependents of this package listed on NPM². This is a form of hardening and improves reliability — the more dependents there are on the code, the more stable and bug-free the code will be.

What code size would this be in Espruino? I don’t know. The project pulls in 18 dependencies. The code for the path we care about is pretty long, involving a case normalization step, followed by a regular expression to add in the title case characters, and a number of different files. I don’t know how Espruino’s module system works (I haven’t looked at it), but there may be overhead there as well. I wouldn’t be surprised if this library pulled in 50 kB of JavaScript code, and maybe if you used Webpack to minify and remove unused code, that might go down to 2 kB — just a complete guess. Either way, it is likely one or two orders of magnitude larger than the compiled C code.

It’s true that I have picked a particularly pathological case to demonstrate my point. But I think the principle stands true. JavaScript developers are more effective than C and C++ developers in large part due to the fact that most of the code in their projects they do not need to write themselves. Conversely, JavaScript library writers are more effective than C++ library writers, because their productivity is multiplied by a larger factor across the thousands of people using their library. But one of the costs here is that libraries need to cater for a wide range of usage scenarios, which makes them bigger and heavier. This is particularly bad for an interpreter.

That tells you why I think it’s a problem for Espruino. But how would MetalScript deal with this?

I see package support as one of the biggest advantages of using JavaScript, so from the beginning this has been one of the objectives of the MetalScript project. There are a number of angles that I’m using to attack this problem:

1. As described in the previous post, a MetalScript program executes in two phases: it starts executing at compile time in a VM running in the compiler, and then gets suspended and the suspended VM is compiled. Dependencies are loaded at compile time, and so even a complicated dependency tree will not incur runtime loading overhead (e.g. all the require statements and initialization code for a library like this will execute at compile time)
2. Unreachable code is automatically eliminated by the garbage collector in the compile-time VM, since code in JavaScript is just like any other data value. There is no need for a separate dead-code elimination step — it comes for free. So all the extra unused functionality of a library will just fall away.
3. A special case of the above two points, is that if a library is only used at compile time, it can be used and freed before the program is even loaded onto the MCU. In C, there have been a number of times where I’ve needed to write a stand-alone tool to pre-generate lookup tables or complex constant structures to be used by a firmware application — in MetalScript, this can just be done in normal code because of the two-phase execution. So if you can find ways to use the libraries at compile time and cache the results for use at runtime, those libraries will actually take up zero flash and RAM.
4. MetalScript uses global optimization techniques to push constants and other type information through the control flow graph. This allows parts of the application that don’t change to be removed from the binary output. In particular, this allows libraries to use dependency injection and options hashes with zero runtime overhead, provided that the running application does not need to change the injections/hooks and options during the execution of the program.

Syntactic Style Matters

This might be my biggest turn-off with Espruino. Since Espruino has a parser and interpreter running on the device to interpret the source text, things like comments and whitespace affect the speed of execution (see here). I don’t need to say much about this. Anyone coming from most a modern programming background will understand why this is a problem.

The solution they propose is minification. Maybe that works for you, maybe it doesn’t. If Espruino has strong support for debugging with source maps, maybe this is fine (does it?).

Performance

I don’t think I need to spend much time justifying this point. On the Espruino website, the same example that talks about whitespace also presents an interesting performance metric. Apparently the following code produces a 4 kHz square wave (it loops 4000 times per second):

while (1) {A0.set();A0.reset();}

In C, I would imagine similar-intentioned code to be about 1000 times faster. Perhaps this example is particularly bad, but I think it goes without saying that interpreting text on the fly is not going to be quick.

MetalScript will deal with this by being compiled. I imagine MetalScript will be comparable in performance to C for this example, since most of the program remains constant and would be optimized out by the symbolic executor I spoke of earlier (the variable and property lookups, and the function calls).

Conformance

Espruino claims 95% compatibility with the ECMAScript specification, and apparently something of a mix between ES5 and ES6 at this point. 95% is pretty poor in my opinion. This means I might often run unit tests on my code using node.js and it all passes, and then find the code behaves differently when I download it to the device.

The FAQ says that if you’re writing “normal JavaScript” then you probably won’t have a problem, but it’s not clear what exactly it does or doesn’t support, so it’s a bit hit-and-miss.

My biggest concern with lack of conformance is that it will impact the ability to include third party libraries. It’s all very well writing your own code from scratch to use Espruino, where you can work around various unsupported features or deviations from the spec. But when it comes to NPM packages that are not specifically designed for Espruino, you get what you get, and its the difference between spending 5 minutes to use an off-the-shelf library to do something vs spending a month writing and debugging it yourself.

I can’t even say for sure that the earlier change-case library example will actually work in Espruino — please can anyone who has an Espruino try including it into a project and tell me what the experience is like? Does it work? How big is it? How easy to integrate compared to installing a dependency for node.js?

Professional Workflow

The quick start guide get’s you up and running pretty quickly with code on the REPL in a terminal, or single scripts in their Chrome app IDE. But then what? If I’m a real professional developer, and want to create a real-world product, the REPL will be an interesting novelty for the first 5 minutes of the project, and then I’ll be asking,

How do I do real development on this thing?

What do I mean by “real development”?

• Creating a project in a way that it can grow to thousands of source code files
• Setting up and a professional IDE to run, edit, and debug the source code using all the modern tools you would expect for such.
• Structuring your project, and bringing in drivers and libraries as needed

Perhaps someone with more experience with Espruino can tell me where the missing starter guide is for a JavaScript professional who wants to make a real product. As it stands, Espruino and its Web IDE look like educational tools for children and hobbyists³.

Unfortunately, I would say that if Espruino is not good for professionals, then it’s also not good for beginners. The reason is that if you are a beginner in something, you are rather going to want to learn to use a tool that will carry over to larger and more professional projects when you outgrow your noob shoes.

How is MetalScript intended to be different?

• MetalScript will be a command-line tool with a similar CLI to node.js. It will be easy for existing JavaScript developers to get started with this interface since it’s familiar, but critically it will be the same interface that can be used at scale in professional work.
• The interface to MetalScript will include the node-inspector debugger API, so that IDEs such as VS Code can be used seamlessly with MetalScript for a full, modern editing and debugging experience.
• The two-phase execution feature means that modules can be unambiguously resolved and loaded at compile time, so only the path of the entry script needs to be provided to MetalScript, like it is with nodejs. There is no need to have separate project files, manifest files, or configuration files, to work with larger projects.

Conclusion

A lot of what I’ve said is speculation. I’m speculating about Espruino based on what I’ve read online, and I’m speculating about how MetalScript will be based on the way I’ve designed it and the way the proof of concept is going. When MetalScript is working, this topic should be revisited.

If you’re an Espruino user, I’d love to hear your experience with it. Have I given it a fair chance? Where are the areas where you think it excels or suffers? What are the lessons that MetalScript should take from it?

Please feel free to share your rants and disagreements with me in the comments.

 

---

1. or npm install --save change-case ↩

2. These are only the ones listed, which only include public projects that are also registered on NPM — think how many thousands more projects use this library that are not listed ↩

3. This is not necessarily a bad thing, but it is not what I need ↩

People who know me, or have been reading my blog for a while, probably know that I’ve been working on a JavaScript firmware compiler, which I’ve called MetalScript. For the first time ever, I’m going to share a little about it with you, my beloved blog readers.

If you’re not interested in the story behind MetalScript, skip about halfway down this post to the Enter: MetalScript heading. I’ll warn you ahead of time that this post is pretty long, because this is a topic I’m really passionate about.

My Story

I first started to learn to work with firmware and microcontrollers when I was 12. My dad is an electrical/software engineer, and maintained a workshop in one of our garages, with many different kinds of electronic components and tools. He showed me how to put together a basic microcontroller to control some LEDs for an electronic dice¹ I did for a school project. I must admit that I didn’t absorb much during that project — I was becoming a teenager, and had just moved schools, and although I was quite proficient at programming by that stage, electronics was still a dark art to me. But through the following few years I become more familiar with it, and it’s nevertheless quite interesting looking back on the experience of I had when I first was introduced to the tools.

The first microcontrollers I learned to work with were in the Microchip PIC family². If I recall correctly, a simple circuit for a PIC to control an LED might have just a handful of components:

• A battery (or bench power supply)
• A voltage regulator, to convert the power from the battery into the 5V needed to run the MCU. Note that some microcontroller don’t need a regulator and can work directly from the battery.
• The microcontroller, which is a like a tiny computer in a single chip (without any peripherals on its own such as a display or keyboard etc), having its own RAM, ROM, and CPU built into the single chip.
• A crystal oscillator — this is the one part of the “tiny computer” that couldn’t be put into the MCU chip (in some MCUs it can be, but not the one I was working on). It’s not important what it does, but just that it’s one of the “ingredients” in creating a simple MCU circuit.
• An LED (with a resistor to limit the current), as an example, so we actually have something for the microcontroller to control

That’s fewer parts than some kitchen appliances, to give you a tiny computer that you can write software for! (And as I say, some MCUs require even less to get started, if they have internal RC oscillators and are more permissive on the input power).

I don’t have a photo of those early circuits (digital cameras weren’t really a thing back then), but here’s a photo from a few years later of a slightly more complicated board, that has 4 LEDs and a monochrome graphics display (the graphics display is the most complicated part, requiring a lot of data lines and a charge pump to drive negative voltages for the backlight of the display).

Side note: this photo was taken in my bedroom³.

I won’t go into detail on how these are made. There are plenty of online resources to guide you through DIY electronics. The main thing I want to highlight is how simple it was. Once you know how, you can solder together⁴ a working microntroller circuit from scratch in probably 30 minutes⁵.

So what about the code?

For these Microchip PIC microcontrollers, you downloaded some free software (MPLAB I think it was called). You went through a wizard to create a new project, where you selected the type of microcontroller, and some other basic stuff, and it created a new project file and a main C file (or assembly — depending on your choices). There were just 2 files you cared about — the code file, and project file. You write your C code into the code file, and click a little button in the IDE to put the code on the MCU and make it run.

I’d characterize this process as “relatively simple”. You write code, and you click the button to download and run it, and that’s it.

To know how to write the code, you would read the datasheet — I emphasize that this was a single datasheet, that concisely described pretty much everything you need to know for the project, including pin-outs, memory layout, IO registers, example code in C, electrical specifications, some reference circuits, and even a description of the instruction set for the device. I’m hugely disheartened these days when I have to refer to 5 different documents written by different companies in order to figure out how to use something.

Writing firmware wasn’t as easy as writing desktop application code, but it was ok. The IDEs were a bit worse, the debuggers a bit less predictable, and there was more work to understand the platform you’re on. It wasn’t a blissful experience, but it was marginally acceptable.

The state of firmware development today

The state of firmware development over the last two decades in my mind has gone downhill from “marginally acceptable” to “a hideous monster” . Microcontrollers have become more powerful, which is a great thing, but with it has come the opportunity for astronomical amounts of complexity.

To get a minimal working example (a flashing LED) using a modern embedded architecture, you are quite possibly going to need:

1. A CPU
2. RAM
3. A flash chip or SD card
4. Possibly a ROM chip to hold the bootloader
5. A ton of supporting circuitry

If you want to assemble the circuitry yourself, bear in mind that many modern day CPUs and MCUs don’t come in DIP packages (the kind I showed in the photo earlier that you can solder into some veroboard yourself), they come in impossible-to-hand-solder packages like BGA. So you’re going to need to buy some adapters, but finding the right ones is going to be tricky. You may just want to buy a module that includes all of this complex circuitry pre-packaged but uncustomizable. Or you can spend 3 weeks having a proper PCB fabricated. There is no good option.

With this setup, you no longer just have C code that is compiled to run directly on the device. Rather, you have a software stack, starting with a bootloader such as U-boot, which reads a file system to load a Linux kernel, and the Linux kernel boots the operating system (the set of device drivers and such), and at some point invokes a startup script, which invokes your application.

If the application wants to toggle an LED, it can no longer set a bit in an IO register. Rather it will probably use a software framework which tells the Linux kernel that it want to output to the GPIO, which dispatches the request to the GPIO driver, which in turn toggles the bit on your behalf. I can’t even confirm to you that this is exactly what happens, despite doing months of research and having spent a fair amount of time writing my own application code in such an environment, because it’s so bloody complicated.

I’ve found that actually compiling and getting the application code onto the device is equally nightmarish. You are now essentially working with two computers: your development computer and a tiny embedded computer running a different (but almost as complicated) operating system, and getting code on your computer to run on the device might involve network drives, or FTP servers, remote debug servers, etc. This is all required just to get a device that blinks and LED.

You might be able to hide from these details for a short while. You may buy a development kit which already has all these things pre-loaded on it. But sooner or later you will be exposed to these details. Say, you need to install a driver for something, or you need to update/configure the kernel, or you need to figure out how to boot from a remotely-hosted image to debug something, or whatever. The abstraction is leaky, and sooner or later you’re going to need to enter that rabbit hole, and it ain’t no wonderland.

I’m not exaggerating when I say that this seems to be the modern way of turning on and off a few LEDs. I spoke to a guy the other day who tells me that the LED street signs here in Melbourne are each running Linux. In fact, each sign runs a full web server stack that you could log into if you wanted. These signs are literally just controlling a panel of LEDs, and somebody decided that the best way to do this was to run a full Linux operating system. I would speculate that they chose this route so that they could run the web stack to make the devices remotely configurable, but I would argue there are better ways to do it, if the tools existed. Imagine the size of the team and expertise that were probably required to do this software.

I don’t care much that it uses more power or costs more to run Linux on a sign, since the incremental cost is negligible compared to the cost of the sign, and power is not an issue here. What I’m concerned about is the deep expertise required to do it, which makes it difficult for the average guy to do something similar, and difficult for beginners to get into. And to reiterate, buying a prepackaged module doesn’t solve the complexity, it just delays your exposure to it.

The JavaScript Perspective

I’m lucky enough to be a software engineer who works full-full stack. That is, I write software for everything from microcontrollers, to websites, and in between. It exposes me to a wide range of tools. One of the those tools is JavaScript⁶. Whenever I mentally context switch from JavaScript development to embedded C or C++ development, it feels like I’m driving into a pit of smelly mud. C++ is full of landmines with memory corruption and undefined behavior. Error messages are complicated. The build process is complicated, involving C preprocessing, template preprocessing, compilation, object files, linker scripts, etc. It cripples your ability to write proper abstractions, or reusable code. I’ve talked about these problems in past blog posts.

JavaScript doesn’t have these issues. There is no preprocessing, no template processing, linker scripts or intermediate object files. A JavaScript program cannot corrupt memory, and there is no such thing as undefined behavior⁷. You can easily write abstractions to reduce the complexity of your code.

JavaScript is also easy for beginners to learn. But perhaps most importantly, it’s easy in JavaScript to integrate third party code in the form of reusable libraries. In C or C++ it might take a week to get set up and familiar with a reasonably-sized foreign library, while in JavaScript it can be a matter of minutes.

Enter: MetalScript

What I am trying to achieve with MetalScript is to bring back the simplicity of the “good ol’ days” of microcontroller development, but bringing it up to modern development standards by supporting JavaScript rather than C or C++.

I want to see a world where you can write a firmware program in JavaScript, and with a single command, run the JavaScript program on the microcontroller. No preparation required (other than plugging the device in). You don’t need Linux, or multi-stage bootloaders. You don’t need project files and linker scripts. You just need your main.js file, and a microcontroller to put it on.

This is really important, so I think it bears repeating. What I am trying to make, is

A tool to that allows you to put a JS program onto an MCU and run it.

No fuss. No linker files. No manifest files, or configuration files of any sort. No make files. No pre-downloading a runtime or an interpreter or a VM or operating system. No 10 step installation process with third party tools and dependencies which “almost” work together.

How will it work?

Similar to the how a C program was run in the old Microchip MPLAB compiler/IDE I spoke of earlier, MetalScript will put a JavaScript program onto your device by first translating it to a format that the device can natively execute. This step is traditionally thought of as compilation, and that’s why I refer to MetalScript as a compiler.

The compilation will produce a result that is self-contained, in that it requires no extra dependencies to be loaded onto the device in order for it to execute. It runs bare metal (hence the name MetalScript), not requiring any supporting interpreter or operating system. In a typical use, especially by beginners, the build artifacts will be loaded directly onto the device and the user won’t even know about it. More advanced workflows will exist for those who want to compile as a library or create an image.

Runtime Library

When C firmware is compiled and loaded onto a device, it’s actually not your C code that starts to run when the device is powered up. The compiler typically inserts some assembly code called the startup script into the executable, and this does some useful initialization such as wiping the memory and initializing global variables. MetalScript is in a way lower level than C, in the sense that the startup process is invoked and controlled by user-written application code, by calling a function called mcu.start(). A typical LED blinking example might be written as follows:

// Run startup routine with the default options (this initializes the processor
// clock, allocates heap and stack memory, initializes the garbage collector, etc)
mcu.start(); 

// Create a timer to call `tick` every 200 ms
setInterval(tick, 200); 

function tick() {
  mcu.gpio('D6').toggle(); // Toggle GPIO pin D6 to blink LED
}

MetalScript is similar to a typical C toolchain, in that it incorporates a standard runtime library during the compilation process. In C, this provides common functions such as strlen, but in JavaScript this provides the garbage collector and event loop.

No Preprocessor

In C, there are two languages: the preprocessor language, and the C language. The preprocessor language is executed at compile time, while the C language corresponds to statements that are computed at runtime. This is complicated and difficult for a beginner to learn, and it’s even more complicated if you consider linker scripts and make files.

In MetalScript, by contrast, there is only one language — everything is JavaScript. Code that happens before mcu.start() is executed at compile time, while everything after mcu.start() is executed at runtime. The set of operations supported at compile time and runtime are different. For example, the code can require  (like #include ) other modules at compile time but not runtime. Timers can be set at compile time or runtime, but they will not start ticking until the MCU is started (i.e. until runtime).

The compiler achieves this by executing the JavaScript program as a process in a virtual environment during compilation, and then when the program calls mcu.start(), the state of the process is suspended and it is this suspended process that is actually compiled, not the source code files (the source code becomes part of the process when it is loaded via import or require statements). For more information about this, take a look at my post on using MetalScript in a C project, which provides more technical details.

There are a few advantages of having it done this way. One of them is very fast runtime startup time, since all the heavy initialization work is already done at compile time, which may include building arbitrarily complex structures in memory for the initial state of every module or driver. Another advantage is that user-written JavaScript code has the capacity to control the compilation and startup process, such as influencing flash memory layout, choice of garbage collector, processor clock source, etc. In a small example, this could all happen within the main.js file, and it can have code, variables, and third party libraries shared between all phases of the program⁸.

A big advantage of this approach is that there is only one language to learn — no need for newcomers to learn macros, C++ templates, linker scripts, and make files. And definitely no need to learn how to write Linux drivers.

Whole Program Optimization

MetalScript compiles the whole program at once, which is one of the reasons why it is initially intended to target firmware and not desktop or server applications, since firmware is necessarily smaller and more self-contained. This gives MetalScript the opportunity to use global optimization techniques, being able to trace values as they flow across the program to do constant propagation and type inference.

This in turn allows library writers to create very versatile libraries, that cover a broad range of use cases, without worrying about performance. This is often done in JavaScript by providing a set of options at the point of construction of a module. When the library is compiled into the firmware application, the global optimizer is able to specialize the library for the specific use by propagating constant option values through the library and eliminating unused code.

The performance of MetalScript will be good, both because of the fact that it is compiled rather than interpreted, and because of these whole program optimizations.

Where should it be used?

I think MetalScript will be useful for small-to-medium sized microcontrollers, in the range of 2kB to 500kB of RAM⁹. In larger devices, the size of the program may negatively impact compilation times. The garbage collector and runtime will have some overhead, and so devices that are too small may not leave enough space for program operation.

For the initial version of MetalScript, I will be targeting one specific microcontroller (not chosen yet), and then this can be extended in future.

MetalScript will not be suitable for systems with hard-real-time constraints, since the performance of any particular piece of code depends on the rest of the application code (as a result of the whole program optimization), and there is no particular upper bound on how slowly a specific piece of code may run, even though performance is expected to be good on average.

Is it proper JavaScript?

Yes.

We’re talking real, modern ECMAScript script, conforming to the specification. By design, you will be able to include pretty much any existing pure-JavaScript NPM package that doesn’t depend on the browser or node APIs. You can use node to execute unit tests if you like, and you can write common libraries or shared code that will run on the browser, the website backend, and the MCU, to capture concepts and models that are common to your IoT domain (if that’s what you’re doing).

The only thing that will knowingly not be supported is eval, since that would require an interpreter running on the device.

TypeScript?

Yes.

As mentioned above, MetalScript will consume real JavaScript source code, and so it will work fine with the TypeScript transpiler. In fact I would strongly encourage you to write your firmware in TypeScript if you can. It will not affect runtime performance, but it will help you eliminate certain types of coding mistakes.

How much will it cost?

In order to continue improving it, working on new generations of the compiler, better library support, and keeping up to date with the latest ECMAScript features, I will need to figure out how to get some money from the project in order to support ongoing development. I haven’t yet figured out how is the best way to do this. If you have ideas, let me know. Likely the compiler will be free for certain types of use and level of support.

When can I have it?

This is a very big project, and I have been working on it for some years now. I have a proof of concept that demonstrates most of the core features, such as the compile-time execution, suspending the process, and running through a number of compilation steps to get an output ELF file. There is a lot more to do — I am still working on the type inference phase, debugger support, and then need to implement all the JavaScript features.

I am not working against any specific timeline, but since I’m doing this in my spare time, it could take a while still. If you’re in a hurry to have it, contact me (see my contact details on the about page) — I will accept motivational pleas, constructive criticism, monetary donations, or a helping hand. I don’t have a Patreon account, but if you want to support me through something like that then let me know and I’ll set one up.

Probably the best way for you to help me out, is to let me know that this is something you want, and share it with your friends on Facebook or Twitter or your favorite forums etc. The biggest impediment to my progress is trying to maintain the motivation to stick with it, day in and day out, and the best way for me to stay motivated will be for me to know that there are people out there who are waiting for it and counting on me to deliver.

---

1. Technically the singular of dice is die, but I think “die” is a pretty overloaded word so I’m intentionally using the incorrect word for clarity — after all, the point of writing is to communicate. ↩

2. It’s confusing as heck that a company named Microchip made microchips ↩

3. I painted my bedroom half sky blue and half terra cotta (although my wive says it was more like orange — take your pick), split down the middle with various patterns at the junction between the two sides. Did you know that when I applied to university, I applied for both art and electrical engineering? ↩

4. I have much fonder memories of soldered veroboard circuits than “breadboard” circuits, since the latter tends to get damaged easily ↩

5. It helps a lot having component trays at the workbench filled with all the common components you need for this kind of thing ↩

6. not “Java” — please do not call JavaScript “Java” for short, since these are two completely different languages ↩

7. There is a small amount of implementation defined behavior, but this is a different thing in practice ↩

8. It could also be done in multiple files for the purposes of code organization, but I highlight that it can be done in a single file to emphasize that this is not just a case of “linker scripts written in JS” but rather a completely new paradigm. ↩

9. I am classifying 500kB of RAM as “medium” because I’m projecting into the future ↩