Abstract : How-to setup a Raspberry Pi and a x86 host, both installed with Arch Linux, to make the Pi to boot on a filesystem stored on the host, and manage this filesystem (package installations/upgrades) from the host, taking advantage of its “horse” power, and saving writes to the SD card — but not only.
Embedded devices use flash memory to store the linux kernel and the root filesystem they use to boot. One concern when developping an embedded system, is that you should avoid to burn the device’s flash memory each time you want to test a new code : first to improve productivity, second to avoid to “burn out” your device’s flash memory.
The Raspberry Pi is a bit specific since it uses removable micro-SD card as a flash memory device. Since it is easily replaceable, the write cycle limit of the flash memory is far less a concern than on a R.Pi. I think this is not a reason to waste write cycles, and anyway, the productivity argument remains.
First, we want to export our root filesytem onto an NFS mount.
Secondly we want to manage our root filesystem using the Arch Linux standard tools : pacman for officially supported packages and Arch Building System (PKGBUILD / makepkg) for other packages (either found AS IS in the AUR, and or written/modified by ourselves).
The Host is an x86 machine running an Arch Linux distribution. It can be either natively or in a VM, and it should work with any Arch derivativate, like Manjaro.
Obviously, your Raspberry Pi should be installed with Arch linux ARM (or a derivative).
Create the NFS-shared root FileSystem (from an already booting SD card)
If not done yet, install an NFS server on the Host, and enable/start it.
To copy the root filesystem, the easiest way is to mount physically the SD card into the host, and mount it :
create an hosting directory for the root FS (let it be /srv/nfs/R.Pi) :
$ sudo mkdir -p /srv/nfs/R.Pi && sudo mount /dev/mmcblk0p2 /tmp/RPI
Copy the root filesystem content into the NFS share (-a flag for ‘archive’ copy just like tar):
$ cp -a /tmp/RPI /srv/nfs/R.Pi
Then add it to the NFS exported directories by adding following line to /etc/exports
A word on the options :
- * means any client can connect. You can white-list authorized clients here.
- rw : I want a Read/Write root filesystem when I boot from NFS, since it will be for development purposes. It is unlikely, but you should want to set ro (a use case could be a Pi using a tiny — or maybe damaged — SD for boot only, on a network that has a NAS that host other partitions…)
- sync : it is a default option, probably only to avoid a warning when the default changed from async to sync. Quoting the man page exports (5) : “To help make system administrators aware of this change, exportfs will issue a warning if neither sync nor async is specified.”
- no_root_squash : linked to rw option, once again let’s state exports (5) man page :
- no_subtree_check : probably superfluous (this option is the default, according to the man page). TODO : Try without and EDIT
Finally restart the nfs-server unit.
Setup the PI kernel command line
Next step is to adjust kernel command line parameters : mount boot partition of the Pi (to edit cmdline):
$ sudo umount /tmp/RPI
$ sudo mount /dev/mmcblk0p1 /tmp/RPI
First make some copies of your cmdline :
- cmdline.txt.ORIG : absolute reference (optional, but ensure against any unwanted edit of cmdline.txt.flash).
- cmdline.txt.flash : base reference file for flash boot. Keep unchanged at first
- cmdline.txt.NFS : We’ll edit it to boot on NFS share.
The two cmdlines :
root=/dev/mmcblk0p2 rw rootwait console=ttyAMA0,115200 console=tty1 selinux=0 plymouth.enable=0 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 kgdboc=ttyAMA0,115200 elevator=noop
root=/dev/nfs nfsroot=192.168.0.11:/srv/nfs/R.Pi/,tcp,vers=3 rw rootwait console=ttyAMA0,115200 console=tty1 selinux=0 plymouth.enable=0 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 kgdboc=ttyAMA0,115200 elevator=noop
Command line diff summary :
- root= parameters differs from /dev/mmcblk0p2 to /dev/nfs
- Additional nfsroot=192.168.0.11:/srv/nfs/R.Pi/,tcp,vers=3 to specify where is the root NFS. Remember that HOST_IP=192.168.0.11
- Note that I use NFS parameter “vers=3” : it is probably superfluous, and a relicate of experimentation I did about problem I got at work (my target board was still booting nfs version 2 by default, since using a kernel 2.6, while my up-to-date archlinux host nfs-utils deprecated version 2, so I had to force vers=3. TODO : try without and edit)
- ip=dhcp parameter is “mandatory” : not that dhcp is mandatory (static IP is an option), but network must be UP before the kernel attempt to mount the root NFS. I guess one could boot using wifi key as long as required modules are statically linked, or loadables without root fs mounted yet (initramfs ?).
- rootfstype=nfs… Heh, I don’t know why it should be specified, and neither test without, but does not eat much bread, and seems to be the obvious… TODO : Study it and edit
Now the Pi should be able to boot.
I used the following procedure to debug the Pi NFS access to the Host (set all debug flags) :
$ sudo rpcdebug -m rpc -s all && sudo rpcdebug -m nfsd -s all && sudo journalctl -fl
NOTE that sudoing the journalctl call is quite important, regular users don’t see logs for system daemons like nfs-server
When debugging acheived, switch off logging to the journal, clearing the debug flags :
$ sudo rpcdebug -m rpc -c all && sudo rpcdebug -m nfsd -c all
Package Management from Host
Now that we have a system that boot from the Pi on the host root FS (while the kernel is still hosted on SD card), we may want to take advantage of the Host to manage (even when the Pi is offline) the root filesystem, for package upgrade, which is the concern here, but also for other purposes, like file-system analysis for system hardeninng, etc.
The solution I have settled on my system is a script-command, pacman.pi, hosted is ~/bin (my $PATH includes it). Here is the script :
sudo mkdir -p /tmp/pacman.pi.cache
sudo mount --bind $PI_ROOT/etc/pacman.d/ /etc/pacman.d/
sudo pacman -v --debug -r $PI_ROOT --config $PI_ROOT/etc/pacman.conf --cachedir /tmp/pacman.pi.cache --logfile /var/log/pacman.pi.log $*
sudo umount /etc/pacman.d/
Obviously, $PI_ROOT environment variable is to be set conviniently in root user default environment, hereit is to be set with “/srv/nfs/R.Pi”).
Binding /etc/pacman.d/ to the RPi root fs is the workaround I found to get the mirror working for synchronizing database, no option seems to be provided so far… Maybe a potential contribution to the Arch !
NOTE that specifying the pi root FS DOES NOT implies using etc/pacman.conf and databases located inside it. Actually it is a big mess up ; and -v and –debug option are here in order to check that pacman reads the right config file, databases, etc.
Nevertheless, there are many wins here :
- The cache for packages is on the host (save writes / gain in speed),
- The root filesystem too (but it could be an NFS export from the Pi, either…),
- Avoid logs in the Pi.. But let’s speak about /var a bit…
When /var is rather a mounted partition…
Avoiding to have /var on the root partition is helpful to save writes to the flash since it is made for logs, caches, and so on. Furthermore, if you want the base root filesystem to be read-only (for security), mounting /var on some writable partition allows logging anyway.
In all cases, if /var is a separate partition, it will be needed to mount or bind the /var of the Pi into $PI_ROOT/var of the host.