One of my favorite hobbies is hacking low-cost embedded systems that run Linux. These systems are absolutely everywhere: the combination of powerful yet inexpensive processors, mass production of consumer goods, and the flexibility of Linux means that hobbyists can buy or build an embedded device capable of running Linux, often for less than $10.
I’ll say that again: not only is embedded Linux within reach for makers and hobbyists, it’s also cheap to throw into your next design.
I’d like to show you just how diverse this ecosystem is, and then I’d like to empower you to start tinkering with these systems. This article series will walk you through choosing hardware and compiling your first system image from source code. Then, we’ll dive deeper into how to pick components for building a custom Linux board, how to deliver updates in a reliable, production-ready way, and how to customize the Linux kernel.
In short, this series of articles will get you up and running with the embedded Linux ecosystem. Ideally you should have some Linux command line knowledge, a good understanding of the overall components of an embedded system (flash memory, processor, peripherals), and a good chunk of free time for learning and experimenting.
Let’s start with some high-level concepts. Having a big-picture view will help prevent being overwhelmed when we start going into more detail later.
Embedded Linux concepts
If you’re already familiar with microcontrollers, you’re by-and-large used to having everything in a single package. You buy an STM32F1 or whatever, and you get flash program memory, RAM, a processor core or two, and some peripherals. If you buy a part specifically designed for it, you might get one or two really nice peripherals tailored to an application — built-in Bluetooth, USB 3.0, or a MIPI camera interface. The device probably has a development studio in which you write your code and send it over to run “bare metal” without a real operating system to get in its way. Finally, all this tends to come in a fairly hobbyist-friendly package: SOIC or QFP packages are usually available for at least one part in a microcontroller line.
The architecture is a little different with Linux-capable processors. Typically, these processors cram in nearly every peripheral you can think of, and many you haven’t heard of. In exchange, they frequently lack built-in RAM, and always lack storage. On top of that, starting one of these devices up is a more involved setup: the processor typically has a built-in boot ROM that is responsible only for loading your bootloader.
From there, the bootloader will load Linux, which will configure the rest of the peripherals and run a preset series of programs. Networking, application logic, and user interface are all available, just like a bigger Linux system. As far as the application is concerned, the environment looks identical to a desktop Linux system — all the APIs are the same, the filesystem looks normal, and it can talk to the outside world using normal networking protocols.
All this is stored in a tiny system image that can be as small as 4MB, depending on the needs of the device. But this is a key point, so I’ll say it again: every software component running on Linux is nearly identical to the version you’d run on a desktop! Linux provides all the abstraction your applications need, so your job is mostly reduced to getting the bootloader and Linux kernel ported to your hardware.
Components
Almost every one of these components is replaceable. The choices made for each of these constitute a large portion of the design decisions you make while choosing an embedded system.
Microprocessor
Of course, your processor will determine a lot of your system’s capabilities. Most of the small, cheap embedded Linux systems you’ll be interacting with will either be using an ARM or MIPS core, and most of the industry is moving further and further toward ARM. (Keep your eye on RISC-V; it’s an open-source instruction set architecture that isn’t quite ready for prime time but has already seen a lot of industry interest.)
On top of the architecture, each silicon vendor adds many different peripherals. Much like their microcontroller brethren, these are small bits of silicon that do one thing, like USB or SPI. They are often configured via memory-mapped registers, and often, manufacturers will add more capabilites to a new product (a “part”) by copy-pasting peripheral IP from an older part, or by adding more copies of the peripheral to the new part.
Memory
These processors typically don’t ship with memory. This is provided separately. The usual kinds of memory you might be familiar with apply here: SDRAM is often used on the low end, while DDR, DDR2, and DDR3 are used for more powerful parts. Although I haven’t yet seen a whole lot of devices using DDR4, I’m sure we’ll get there. The processor provides a module to manage the memory; this module is either initialized by the manufacturer’s boot code or by the first stage of the bootloader.
Note how I said processors typically don’t ship with RAM embedded in their package? Sometimes, they do. If you’re buying an embedded system on a module, or a ready-to-go development board, you don’t care about this, because someone else has already done the hard work of putting the memory on the PCB.
But if you are a hobbyist building your own from individual parts, you care very deeply about these parts (whether or not you know it yet). That’s because laying memory out on a PCB is difficult for a few different reasons. So when I’m talking about hardware in my next article, I’ll be sure call out some parts I’ve found that ship their own memory.
Storage
Storage, distinct from memory, is the involatile place your code and data is stored. On embedded systems, it will almost certainly be some form of flash memory. Even in cases where the system boots from the network, typically this is still assisted by a bootloader stored in flash.
There are multiple kinds of flash memory. You are likely already familiar with SD or microSD cards. These provide raw storage, plus a flash controller whose job is to abstract the raw flash into a well-behaved storage medium (more on this in a second). SD cards are not known for their reliability, as anybody who has used a Raspberry Pi can tell you. However, later in this series, I’ll explain how best to work around questionably reliable storage media.
eMMC is an embeddable version of an SD card. Conceptually it is very similar; it has raw flash and a controller bundled into a single part. Unfortunately it is usually sold in nasty BGA packages like is pictured, which aren’t a problem for large companies but are pretty much unusable for hobbyists because you can’t easily solder it.
You can also buy raw flash without a controller attached to it. This is typically cheaper than eMMC and is sold in two varieties, NOR and NAND. NOR flash is slow to write, very cheap, and not very dense. Boot ROMs almost always know how to read from NOR flash accessed over SPI. If you’re not writing very often, maybe only once to burn firmware to the device, it’s a great option. NAND flash is faster, denser, and a little more expensive. You can buy NAND that’s accessed via SPI, but there are also faster parts that can be accessed over a dedicated NAND bus.
Raw flash isn’t really all that nice. You cannot write to it willy-nilly; flash can only be written once before you must erase it in large blocks. Furthermore, flash can be written a limited number of times before it wears out (between 1000 and 100000, depending on the technology).
A software guy’s first instinct is to throw software at a hardware problem, and that’s what we’ll do here, later on. A Linux subsystem called UBI can help work around these limitations and make raw flash nicer to use.
Software
If you’re developing for one of these systems, you provide the bootloader, kernel, and filesystem. Thankfully, once Linux has booted, it often supports many of the peripherals on a system out of the box using a wonderfully consistent set of driver interfaces — in stark contrast to how often you must write drivers yourself on a microcontroller.
In general the software typically follows a very understandable pattern that you can apply anywhere.
You don’t have to micromanage all these pieces at once. There are embedded Linux distributions that provide a complete toolkit to help you build a firmware image with all of this included. Later on, we’ll start using a distribution and compile our own complete system.
Bootloader
The bootloader is the first program that the engineer or hacker has control over. It is only as complex as is needed to load the kernel and get it running. In practice, the bootloader can still be pretty complex.
Your embedded Linux system will almost certainly be using Das U-Boot, the so-called “universal bootloader.” (It really does run on nearly everything!) The bootloader has stripped-down drivers for the onboard storage, perhaps a couple of other peripherals, and just enough code to read the kernel into memory and start executing it.
A lot of times, people don’t mess with the bootloader that comes with their board, and they just follow the conventions it’s expecting. That’s fine, but often you want your system to do something the stock bootloader can’t. With the right tools, you have no reason to be afraid of modifying the bootloader — it is a program like any other.
Boot ROM
I should mention the Boot ROM here. The Boot ROM is a small chunk of code embedded in the processor, provided by the manufacturer. It is very, very simple, and typically it immediately runs the real bootloader from a couple different storage media — often this “boot order” is specified in the processor’s datasheet.
Boot ROMs have one other cool trick — they often speak USB! This allows you to connect a brand-new, unprogrammed system to a computer over USB and run code, or flash a storage medium. If it’s present, this feature makes a board nearly impossible to brick. Different manufacturers call this mode different things. NXP/Freescale calls it Download Mode. Allwinner calls it FEL mode. You will generally need a special program on your workstation to use this mode, and capabilities vary between different processors.
Kernel
You already know what Linux is, I hope! Linux must be ported to each architecture, each part, and each board. All these drivers ship with the kernel source. There’s a lot of them, but because of the tendency of manufacturers to reuse IP, it’s managable. Typically, a port consists of the following:
- Architecture code provides low-level routines for very basic things like register manipulation, synchronization primitives, etc. Porting Linux to a completely new architecture is an immense amount of work, and you will likely not have to do this anytime soon.
- Drivers make up the bulk of the kernel source code. This is because Linux ships drivers for every device supported by Linux, in one source code tree. Most of them are not needed by your embedded system — for example, a MIPS router has absolutely no use for a driver for Intel QuickSync. So most of these drivers are not even compiled for very small systems.
- The device tree is a very important part of the port that explains to the kernel how hardware is actually connected to the system. Device trees are the “config files” for Linux drivers. This means that Linux drivers are easy to reuse — if you add a device tree entry for a piece of hardware on your system and compile the relevant driver, the driver will find the entry and set itself up. We’ll talk a lot about device trees later.
Userland
Like I mentioned already, the “rest” of the software on an embedded system is pretty much identical to its desktop counterparts. Together, this “everything else” is frequently called “user land,” because it’s the land where user software can roam freely, without worries of the nasty hardware gremlins that lurk underneath the peaceful abstraction of the Linux API.
We’ll go over userland in more detail later. The major components you need to know right now are the filesystem, the init system, and the shell.
The filesystem is important because some are better for embedded systems than others — remember the SD card reliability we talked about? Filesystems can help with this. If the system is using eMMC or an SD card, it looks like a normal “block device,” and you can use the standard ext2/3/4 or stuff like the flash-friendly filesystem (f2fs), which helps with reliability in various ways. If you’re on raw flash, you need to use more esoteric filesystems that are designed for raw flash, such as JFFS2 or UBIFS. My personal favorite is to use squashfs on an UBI partition; we’ll talk more about this when we start customizing our firmware image!
Next, the init system is responsible for managing userland. You might have heard of systemd? It’s used on bigger embedded systems. However, it’s too big for the really small systems, which usually use a SysV init scheme of simple shell scripts. Most embedded Linux distributions provide this, and again, when we’re customizing firmware, I’ll show you how to add your programs to run automatically.
And finally, the shell is what you’ll interact with. Traditionally, this is done over a UART serial connection, but occasionally you might have the luxury of a screen. If you get your system to a shell prompt, you have definitely gotten your system up and running!
Next steps
Okay. That’s a lot of info all at once. Thanks for staying with me!
With that out of the way, here’s my rough plan for the series:
- Hardware: We’ll choose a development board. Spoilers: I’ll use a Raspberry Pi Zero for demo for the first few software tutorials. But here, I’ll also point out various other cheap Linux-capable hardware. If you’re interested in building your own, I am also recommending hacker-friendly parts that you can add to your own PCB.
- Buildroot: For a quick win, we’ll download Buildroot and compile a complete operating system from scratch. We’ll flash it and boot it on real hardware. I’ll be sure to explain what’s happening at each step.
- Customizing the firmware image: here we’ll depart from what the Buildroot developers' defaults and begin to make changes to the Raspberry Pi’s firmware. Each individual step is pretty straightforward, and all combined, your finished system can look completely different from the stock one. It’s all up to the engineer.
- Going smaller: We leave the sunny skies of Raspberry Pi land and start working on an image for a very small board with just 4MB of storage. It’s quite impressive how much functionality you can cram in—take that, node_modules!
- Hacking on U-Boot and Linux: The available code for your board’s processor is out of date. What do you do?
- Building a board from scratch: We’ll go back to the parts I recommend in the Hardware stage and start building our very own dev board. (This part is subject to change as I figure out the best approach!)
In the next article we’ll look around at some hardware. If you’d like to follow along, you can subscribe.
In the meantime, if you’d like to Linux-enable your project, product, or program, leave a comment or drop me a line—I’d love to hear from you and I’m always happy to hear about cool stuff.