People who know me, or have been reading my blog for a while, probably know that I’ve been working on a JavaScript firmware compiler, which I’ve called MetalScript. For the first time ever, I’m going to share a little about it with you, my beloved blog readers.

If you’re not interested in the story behind MetalScript, skip about halfway down this post to the Enter: MetalScript heading. I’ll warn you ahead of time that this post is pretty long, because this is a topic I’m really passionate about.

My Story

I first started to learn to work with firmware and microcontrollers when I was 12. My dad is an electrical/software engineer, and maintained a workshop in one of our garages, with many different kinds of electronic components and tools. He showed me how to put together a basic microcontroller to control some LEDs for an electronic dice¹ I did for a school project. I must admit that I didn’t absorb much during that project — I was becoming a teenager, and had just moved schools, and although I was quite proficient at programming by that stage, electronics was still a dark art to me. But through the following few years I become more familiar with it, and it’s nevertheless quite interesting looking back on the experience of I had when I first was introduced to the tools.

The first microcontrollers I learned to work with were in the Microchip PIC family². If I recall correctly, a simple circuit for a PIC to control an LED might have just a handful of components:

• A battery (or bench power supply)
• A voltage regulator, to convert the power from the battery into the 5V needed to run the MCU. Note that some microcontroller don’t need a regulator and can work directly from the battery.
• The microcontroller, which is a like a tiny computer in a single chip (without any peripherals on its own such as a display or keyboard etc), having its own RAM, ROM, and CPU built into the single chip.
• A crystal oscillator — this is the one part of the “tiny computer” that couldn’t be put into the MCU chip (in some MCUs it can be, but not the one I was working on). It’s not important what it does, but just that it’s one of the “ingredients” in creating a simple MCU circuit.
• An LED (with a resistor to limit the current), as an example, so we actually have something for the microcontroller to control

That’s fewer parts than some kitchen appliances, to give you a tiny computer that you can write software for! (And as I say, some MCUs require even less to get started, if they have internal RC oscillators and are more permissive on the input power).

I don’t have a photo of those early circuits (digital cameras weren’t really a thing back then), but here’s a photo from a few years later of a slightly more complicated board, that has 4 LEDs and a monochrome graphics display (the graphics display is the most complicated part, requiring a lot of data lines and a charge pump to drive negative voltages for the backlight of the display).

Side note: this photo was taken in my bedroom³.

I won’t go into detail on how these are made. There are plenty of online resources to guide you through DIY electronics. The main thing I want to highlight is how simple it was. Once you know how, you can solder together⁴ a working microntroller circuit from scratch in probably 30 minutes⁵.

So what about the code?

For these Microchip PIC microcontrollers, you downloaded some free software (MPLAB I think it was called). You went through a wizard to create a new project, where you selected the type of microcontroller, and some other basic stuff, and it created a new project file and a main C file (or assembly — depending on your choices). There were just 2 files you cared about — the code file, and project file. You write your C code into the code file, and click a little button in the IDE to put the code on the MCU and make it run.

I’d characterize this process as “relatively simple”. You write code, and you click the button to download and run it, and that’s it.

To know how to write the code, you would read the datasheet — I emphasize that this was a single datasheet, that concisely described pretty much everything you need to know for the project, including pin-outs, memory layout, IO registers, example code in C, electrical specifications, some reference circuits, and even a description of the instruction set for the device. I’m hugely disheartened these days when I have to refer to 5 different documents written by different companies in order to figure out how to use something.

Writing firmware wasn’t as easy as writing desktop application code, but it was ok. The IDEs were a bit worse, the debuggers a bit less predictable, and there was more work to understand the platform you’re on. It wasn’t a blissful experience, but it was marginally acceptable.

The state of firmware development today

The state of firmware development over the last two decades in my mind has gone downhill from “marginally acceptable” to “a hideous monster” . Microcontrollers have become more powerful, which is a great thing, but with it has come the opportunity for astronomical amounts of complexity.

To get a minimal working example (a flashing LED) using a modern embedded architecture, you are quite possibly going to need:

1. A CPU
2. RAM
3. A flash chip or SD card
4. Possibly a ROM chip to hold the bootloader
5. A ton of supporting circuitry

If you want to assemble the circuitry yourself, bear in mind that many modern day CPUs and MCUs don’t come in DIP packages (the kind I showed in the photo earlier that you can solder into some veroboard yourself), they come in impossible-to-hand-solder packages like BGA. So you’re going to need to buy some adapters, but finding the right ones is going to be tricky. You may just want to buy a module that includes all of this complex circuitry pre-packaged but uncustomizable. Or you can spend 3 weeks having a proper PCB fabricated. There is no good option.

With this setup, you no longer just have C code that is compiled to run directly on the device. Rather, you have a software stack, starting with a bootloader such as U-boot, which reads a file system to load a Linux kernel, and the Linux kernel boots the operating system (the set of device drivers and such), and at some point invokes a startup script, which invokes your application.

If the application wants to toggle an LED, it can no longer set a bit in an IO register. Rather it will probably use a software framework which tells the Linux kernel that it want to output to the GPIO, which dispatches the request to the GPIO driver, which in turn toggles the bit on your behalf. I can’t even confirm to you that this is exactly what happens, despite doing months of research and having spent a fair amount of time writing my own application code in such an environment, because it’s so bloody complicated.

I’ve found that actually compiling and getting the application code onto the device is equally nightmarish. You are now essentially working with two computers: your development computer and a tiny embedded computer running a different (but almost as complicated) operating system, and getting code on your computer to run on the device might involve network drives, or FTP servers, remote debug servers, etc. This is all required just to get a device that blinks and LED.

You might be able to hide from these details for a short while. You may buy a development kit which already has all these things pre-loaded on it. But sooner or later you will be exposed to these details. Say, you need to install a driver for something, or you need to update/configure the kernel, or you need to figure out how to boot from a remotely-hosted image to debug something, or whatever. The abstraction is leaky, and sooner or later you’re going to need to enter that rabbit hole, and it ain’t no wonderland.

I’m not exaggerating when I say that this seems to be the modern way of turning on and off a few LEDs. I spoke to a guy the other day who tells me that the LED street signs here in Melbourne are each running Linux. In fact, each sign runs a full web server stack that you could log into if you wanted. These signs are literally just controlling a panel of LEDs, and somebody decided that the best way to do this was to run a full Linux operating system. I would speculate that they chose this route so that they could run the web stack to make the devices remotely configurable, but I would argue there are better ways to do it, if the tools existed. Imagine the size of the team and expertise that were probably required to do this software.

I don’t care much that it uses more power or costs more to run Linux on a sign, since the incremental cost is negligible compared to the cost of the sign, and power is not an issue here. What I’m concerned about is the deep expertise required to do it, which makes it difficult for the average guy to do something similar, and difficult for beginners to get into. And to reiterate, buying a prepackaged module doesn’t solve the complexity, it just delays your exposure to it.

The JavaScript Perspective

I’m lucky enough to be a software engineer who works full-full stack. That is, I write software for everything from microcontrollers, to websites, and in between. It exposes me to a wide range of tools. One of the those tools is JavaScript⁶. Whenever I mentally context switch from JavaScript development to embedded C or C++ development, it feels like I’m driving into a pit of smelly mud. C++ is full of landmines with memory corruption and undefined behavior. Error messages are complicated. The build process is complicated, involving C preprocessing, template preprocessing, compilation, object files, linker scripts, etc. It cripples your ability to write proper abstractions, or reusable code. I’ve talked about these problems in past blog posts.

JavaScript doesn’t have these issues. There is no preprocessing, no template processing, linker scripts or intermediate object files. A JavaScript program cannot corrupt memory, and there is no such thing as undefined behavior⁷. You can easily write abstractions to reduce the complexity of your code.

JavaScript is also easy for beginners to learn. But perhaps most importantly, it’s easy in JavaScript to integrate third party code in the form of reusable libraries. In C or C++ it might take a week to get set up and familiar with a reasonably-sized foreign library, while in JavaScript it can be a matter of minutes.

Enter: MetalScript

What I am trying to achieve with MetalScript is to bring back the simplicity of the “good ol’ days” of microcontroller development, but bringing it up to modern development standards by supporting JavaScript rather than C or C++.

I want to see a world where you can write a firmware program in JavaScript, and with a single command, run the JavaScript program on the microcontroller. No preparation required (other than plugging the device in). You don’t need Linux, or multi-stage bootloaders. You don’t need project files and linker scripts. You just need your main.js file, and a microcontroller to put it on.

This is really important, so I think it bears repeating. What I am trying to make, is

A tool to that allows you to put a JS program onto an MCU and run it.

No fuss. No linker files. No manifest files, or configuration files of any sort. No make files. No pre-downloading a runtime or an interpreter or a VM or operating system. No 10 step installation process with third party tools and dependencies which “almost” work together.

How will it work?

Similar to the how a C program was run in the old Microchip MPLAB compiler/IDE I spoke of earlier, MetalScript will put a JavaScript program onto your device by first translating it to a format that the device can natively execute. This step is traditionally thought of as compilation, and that’s why I refer to MetalScript as a compiler.

The compilation will produce a result that is self-contained, in that it requires no extra dependencies to be loaded onto the device in order for it to execute. It runs bare metal (hence the name MetalScript), not requiring any supporting interpreter or operating system. In a typical use, especially by beginners, the build artifacts will be loaded directly onto the device and the user won’t even know about it. More advanced workflows will exist for those who want to compile as a library or create an image.

Runtime Library

When C firmware is compiled and loaded onto a device, it’s actually not your C code that starts to run when the device is powered up. The compiler typically inserts some assembly code called the startup script into the executable, and this does some useful initialization such as wiping the memory and initializing global variables. MetalScript is in a way lower level than C, in the sense that the startup process is invoked and controlled by user-written application code, by calling a function called mcu.start(). A typical LED blinking example might be written as follows:

// Run startup routine with the default options (this initializes the processor
// clock, allocates heap and stack memory, initializes the garbage collector, etc)
mcu.start(); 

// Create a timer to call `tick` every 200 ms
setInterval(tick, 200); 

function tick() {
  mcu.gpio('D6').toggle(); // Toggle GPIO pin D6 to blink LED
}

MetalScript is similar to a typical C toolchain, in that it incorporates a standard runtime library during the compilation process. In C, this provides common functions such as strlen, but in JavaScript this provides the garbage collector and event loop.

No Preprocessor

In C, there are two languages: the preprocessor language, and the C language. The preprocessor language is executed at compile time, while the C language corresponds to statements that are computed at runtime. This is complicated and difficult for a beginner to learn, and it’s even more complicated if you consider linker scripts and make files.

In MetalScript, by contrast, there is only one language — everything is JavaScript. Code that happens before mcu.start() is executed at compile time, while everything after mcu.start() is executed at runtime. The set of operations supported at compile time and runtime are different. For example, the code can require  (like #include ) other modules at compile time but not runtime. Timers can be set at compile time or runtime, but they will not start ticking until the MCU is started (i.e. until runtime).

The compiler achieves this by executing the JavaScript program as a process in a virtual environment during compilation, and then when the program calls mcu.start(), the state of the process is suspended and it is this suspended process that is actually compiled, not the source code files (the source code becomes part of the process when it is loaded via import or require statements). For more information about this, take a look at my post on using MetalScript in a C project, which provides more technical details.

There are a few advantages of having it done this way. One of them is very fast runtime startup time, since all the heavy initialization work is already done at compile time, which may include building arbitrarily complex structures in memory for the initial state of every module or driver. Another advantage is that user-written JavaScript code has the capacity to control the compilation and startup process, such as influencing flash memory layout, choice of garbage collector, processor clock source, etc. In a small example, this could all happen within the main.js file, and it can have code, variables, and third party libraries shared between all phases of the program⁸.

A big advantage of this approach is that there is only one language to learn — no need for newcomers to learn macros, C++ templates, linker scripts, and make files. And definitely no need to learn how to write Linux drivers.

Whole Program Optimization

MetalScript compiles the whole program at once, which is one of the reasons why it is initially intended to target firmware and not desktop or server applications, since firmware is necessarily smaller and more self-contained. This gives MetalScript the opportunity to use global optimization techniques, being able to trace values as they flow across the program to do constant propagation and type inference.

This in turn allows library writers to create very versatile libraries, that cover a broad range of use cases, without worrying about performance. This is often done in JavaScript by providing a set of options at the point of construction of a module. When the library is compiled into the firmware application, the global optimizer is able to specialize the library for the specific use by propagating constant option values through the library and eliminating unused code.

The performance of MetalScript will be good, both because of the fact that it is compiled rather than interpreted, and because of these whole program optimizations.

Where should it be used?

I think MetalScript will be useful for small-to-medium sized microcontrollers, in the range of 2kB to 500kB of RAM⁹. In larger devices, the size of the program may negatively impact compilation times. The garbage collector and runtime will have some overhead, and so devices that are too small may not leave enough space for program operation.

For the initial version of MetalScript, I will be targeting one specific microcontroller (not chosen yet), and then this can be extended in future.

MetalScript will not be suitable for systems with hard-real-time constraints, since the performance of any particular piece of code depends on the rest of the application code (as a result of the whole program optimization), and there is no particular upper bound on how slowly a specific piece of code may run, even though performance is expected to be good on average.

Is it proper JavaScript?

Yes.

We’re talking real, modern ECMAScript script, conforming to the specification. By design, you will be able to include pretty much any existing pure-JavaScript NPM package that doesn’t depend on the browser or node APIs. You can use node to execute unit tests if you like, and you can write common libraries or shared code that will run on the browser, the website backend, and the MCU, to capture concepts and models that are common to your IoT domain (if that’s what you’re doing).

The only thing that will knowingly not be supported is eval, since that would require an interpreter running on the device.

TypeScript?

Yes.

As mentioned above, MetalScript will consume real JavaScript source code, and so it will work fine with the TypeScript transpiler. In fact I would strongly encourage you to write your firmware in TypeScript if you can. It will not affect runtime performance, but it will help you eliminate certain types of coding mistakes.

How much will it cost?

In order to continue improving it, working on new generations of the compiler, better library support, and keeping up to date with the latest ECMAScript features, I will need to figure out how to get some money from the project in order to support ongoing development. I haven’t yet figured out how is the best way to do this. If you have ideas, let me know. Likely the compiler will be free for certain types of use and level of support.

When can I have it?

This is a very big project, and I have been working on it for some years now. I have a proof of concept that demonstrates most of the core features, such as the compile-time execution, suspending the process, and running through a number of compilation steps to get an output ELF file. There is a lot more to do — I am still working on the type inference phase, debugger support, and then need to implement all the JavaScript features.

I am not working against any specific timeline, but since I’m doing this in my spare time, it could take a while still. If you’re in a hurry to have it, contact me (see my contact details on the about page) — I will accept motivational pleas, constructive criticism, monetary donations, or a helping hand. I don’t have a Patreon account, but if you want to support me through something like that then let me know and I’ll set one up.

Probably the best way for you to help me out, is to let me know that this is something you want, and share it with your friends on Facebook or Twitter or your favorite forums etc. The biggest impediment to my progress is trying to maintain the motivation to stick with it, day in and day out, and the best way for me to stay motivated will be for me to know that there are people out there who are waiting for it and counting on me to deliver.

---

1. Technically the singular of dice is die, but I think “die” is a pretty overloaded word so I’m intentionally using the incorrect word for clarity — after all, the point of writing is to communicate. ↩

2. It’s confusing as heck that a company named Microchip made microchips ↩

3. I painted my bedroom half sky blue and half terra cotta (although my wive says it was more like orange — take your pick), split down the middle with various patterns at the junction between the two sides. Did you know that when I applied to university, I applied for both art and electrical engineering? ↩

4. I have much fonder memories of soldered veroboard circuits than “breadboard” circuits, since the latter tends to get damaged easily ↩

5. It helps a lot having component trays at the workbench filled with all the common components you need for this kind of thing ↩

6. not “Java” — please do not call JavaScript “Java” for short, since these are two completely different languages ↩

7. There is a small amount of implementation defined behavior, but this is a different thing in practice ↩

8. It could also be done in multiple files for the purposes of code organization, but I highlight that it can be done in a single file to emphasize that this is not just a case of “linker scripts written in JS” but rather a completely new paradigm. ↩

9. I am classifying 500kB of RAM as “medium” because I’m projecting into the future ↩