Debugging Embedded Code
At first I didn’t think this would be something useful to blog about. But the more people I speak to, the more I realize that I might be one of the few who has my perspective on how to debug embedded microcontroller/firmware code on small devices1. So let me share my “secret formula” with you.
Problems
For those who don’t traditionally write embedded microcontroller applications but are still following along for interest, let me quickly highlight some of key problems we face debugging embedded code.
Problem 1: Hardware
The most obvious problem with developing embedded code is that these microchips are generally embedded into a piece of hardware, such as your washing machine, and the code’s primary purpose is often closely related to controlling the hardware. You can often use what is called an in-circuit debugger to actually connect your IDE/debugger to the firmware program running on the embedded device2.
But lets say you’re writing code for an x-ray machine, and lets say that it needs to pulse the x-ray cathode for 10 ms. You do not want to accidentally put a breakpoint during those 10 ms. The breakpoint will suspend the execution of the firmware, but will generally leave the hardware in whatever state it was in – which may be a state you really don’t want it to be in for an extended period of time3.
There are other reasons why hardware can be a problem for debugging, such as when the program outputs physical voltages but does it incorrectly. You may need to start resorting to using a multimeter or oscilloscope just to see what the program is doing. You can’t add a software “debug watch” to a piece of hardware.
Problem 2: Download time
Another problem with debugging is the time it takes to cycle through your modify-build-execute-debug process, because it has an extra “download” phase. In other words, every time you make a change, you have to download the compiled executable to the embedded device. This can take anywhere from a few seconds to minutes, depending on the size of the program. It seems a common pattern for developers to do something like this:
- Compile program
- Download program
- Execute in debug mode
- Try put it into the problem state
- See the problem (perhaps by a breakpoint and watches)
- Go back to your code
- Add diagnostic
printfstatements or more breakpoints - Go back to step 1
I’ll call this the compile-debug cycle, and it can take minutes to get through it each time.
Problem 3: Non-Standard Compilers
Another problem, which perhaps isn’t quite as obvious, is that is seems that an overwhelming number of compilers targeted at embedded microcontrollers do not conform to the C or C++ standards. They often only support a subset of standard C/C++, and they include many extensions for things which are difficult or impossible to do in standard C. This can be a good thing: how do you declare an interrupt routine in standard C? But for debugging code it can cause problems. In the same way that hardware dependent code forces you debug code on the physical hardware, compiler dependent code forces you to debug with the specific debugger provided with the compiler. Why is this a problem? I’ll get to that in a moment.
Solution
So how do I solve these problems?
The answer is stupidly simple. Perhaps even insulting. It is simply:
Don’t write embedded code
Avoid writing code that accesses hardware, or that needs to be downloaded to run, or that uses non-standard extensions of C/C++. You could almost say, “Don’t write embedded code at all”. Embedded code has all of these problems, so just don’t do it!
Instead, write platform-independent code. Use all the modern techniques of dependency injection, unit testing, etc. Get your code working in isolation first, in an environment where your compile-debug cycle is in the order of seconds rather than minutes. Get it working on your local PC! This eliminates step 2 – “Download program” – from the above compile-debug cycle, and makes everything easy.
Also, ideally you should be debugging unit tests, not your fully integrated software. This eliminates step 4 in the above compile-debug cycle, because your unit tests automatically generate the problem state. If they don’t, then you may need to write more tests.
The unit tests for the module you’re working on should automatically run at the click of a button or keyboard shortcut, and should be integrated into the IDE so you don’t have to leave your code to run tests. Personally, I divide my code into small enough modules that my unit tests for the module-under-development can compile and run in under a second, and causes the IDE to jump immediately to the first failing test assertion.
Doing it this way you can easily shorten the compile-debug cycle overhead by a factor of 100 (“overhead” meaning tasks not related to finding or fixing the actual problem). If you also write the code in such a way that you can debug-step backwards then you can also reduce the number of times you need to repeat the compile-debug cycle.
But, Hardware!
Of course, you do need to access hardware at some point. And you probably need to use compiler pragmas and intrinsics to squeeze out the most performance or do unusual things. But you can isolate these cases, and confine them to very thin layers on which your main application code runs. For example, instead of having application-level code directly access an IO port, it can access it through a function which is implemented by an ultra-thin HAL (hardware abstraction layer). The HAL should do as little as possible, and its interface should be pure C (or C++). When I say “hardware”, I also mean other platform specific dependencies, such as a compiler quirks, third-party libraries, and the OS.
The rest of your application code should be platform independent. Don’t use features that aren’t standard C/C++. Obviously only use the subset of the standard that’s actually supported by your compiler. Don’t rely on unspecified “features” of the language, such as the size of a pointer, the result of signed integer overflow, the endianness of an integer, or the alignment of fields in structure (But you weren’t anyway, right?).
Use dependency injection to inject the HAL into your application code4. This is just generally good advice for any program. And how else would you do unit testing? For different build configurations you can inject mocks and interactive simulators just as easily as the HAL on your release build.
Remember that every line of HAL code might take 10 times longer to develop, so you really want to minimize how much HAL there is.
Conclusion
So, the solution to the long debug-compile cycle with embedded code is simply to avoid “embedded” code as much as possible. I do this, and it really works. I can go weeks developing code for a microcontroller without ever powering up the physical device, and then take a few hours or so to integrate it into the final compiled application. The result is that I can use all the modern techniques and practices to develop high quality firmware in a fraction of the time.
Why don’t many embedded developers seem to do this5? I really don’t know. But perhaps its related to the fact that developing for such limited systems is a lot like developing for the desktop computers of 20 years ago, before object orientated patterns and unit testing were really around. Or perhaps it’s because C and C++ make it very difficult to do zero-cost dependency injection. Perhaps its just because the embedded microcontroller industry is much slower to move, since it targets a much smaller audience.
Whatever the reason, it doesn’t have to be that way, and hopefully I’ve given you a few ideas of how to accelerate your development process. If you have any other ideas or questions, feel free to leave them in the comments below.
I’m talking about anything that’s too small to run a proper operating systems like Linux – something in the order of kilobytes of RAM ↩
Again, I’m talking about quite low level programs here. If the “firmware” is running a full OS, such as Linux, then it may have its own debugging tools such a remote GDB server ↩
I’m not saying you would use a breakpoint at all in a real x-ray machine, but the point applies to other scenarios ↩
I’m not going to cover how to do dependency injection for this type of application, but techniques you could consider are: using the linker to “inject” a dependency C API or mock layer into your application; using CRTP to inject dependencies into C++ code statically; and of course plain old virtual classes and/or function pointers ↩
I may be generalizing. But realize again that I’m referring to embedded firmware programs, perhaps running in sub 100 kB of RAM, and not smartphone or embedded Linux development ↩