So, it’s definitely been a while since my last post! Sorry about that, life, finds a way! (quoting jurassic park).
Okay so a long time ago in a galaxy far away my manager asked if we could NFS boot a yocto image and I was like, sure! (I had no idea but fully intended to learn). So, I setup a boot system with a rpi hosting my NFS server. I’ll give those instructions in another article, the point here is to rather look at the linux client, not the server.
So, how does linux boot? Well, the first thing it loads is the kernel, which is a very small image in KB scale that has enough knowledge of life to know it needs a root file system which then provides all the other utilities that the user needs. Now, in small systems you can skip this next step and straight up boot, in most cases in fact you can.
But, when you do a network boot, the network server can give so much before the client has to do the rest. So, once the kernel loads it looks for init. Init is a program that sets up everything else, and initfs is a small filesystem you can give to the kernel whereby it has the basic tools to then load everything else.
Anyway, so the challenge my manager then presented was “What happens if the NFS server disconnects? What happens if I need to run a program? or if i need file IO?”. So I ran some tests and the unfortunate answer is this:
- NFS filesystems continue to depend on the NFS connection
- NFS is slow for real time when you try to execute a program as it has to go fetch it from the server
- If you do File IO, it’s actually network IO.
So, for us, this was somewhat unacceptable, this is afterall a real time system. So I ventured out into the known universe of google and found this:
And this was a great starting point, however, this isn’t the end, I need yocto to understand how to implement this of course. So let’s go back to what I said earlier and what I’ve said before about yocto:
- Yocto simply builds a linux kernel based on your machine specs and then makes the utilities you ask for to be in your root fs.
So, we need somehow to make a yocto build which does this. This was quite a useful article in getting that next piece of the puzzle:
https://www.kernel.org/doc/html/latest/admin-guide/initrd.html
So I knew that I need to pass the kernel command line arguments, these are arguments during its boot process. You may be familiar with them or not, here’s an example of what you might see if you click e during a linux grub boot and then configure:
Let’s diagnose whats happening here, the line, linux /boot/vm… is saying, first, load the kernel vmlinuz-2.6.31-19-generic, thats your kernel. Next it states root as a UUID, it is stating that the rootfs is on a drive with that UUID so if that drive is not connected, the kernel will fault. Finally it says, quiet, splash, quiet means don’t show the kernel messages during load like this:
Splash, well, if you have a splash screen, use it.
So, first issue is that root, i’m not loading from a drive locally, so what do I put? Also, how does the kernel interpret these parameters? Mostly, it doesn’t! These are global variables that every program has access to in most cases.
Something I’ve said before, linux is a file based operating system, keep to that mindset, so whenever you have a question it, grep it. So if you do enough googling you’ll stumble upon this:
https://github.com/openembedded/meta-openembedded/tree/master/meta-initramfs
This is a small filesystem for a preliminary ramfs, a good start. Now let’s start navigating it at the important parts. If you get to here:
You start to realise yocto embeds it’s initrd structure. But this won’t tell you much, what it does tell us is that in the same branch of this folder hierarchy in the main yocto openembedded repo, we may find something related to this, so lets go.
Cool, so we’re now here and there’s a whole bunch of stuff:
https://github.com/openembedded/openembedded-core/tree/master/meta/recipes-core/initrdscripts
If you go to framework you see this:
SUMMARY = "Modular initramfs system"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
RDEPENDS:${PN} += "${VIRTUAL-RUNTIME_base-utils}"
RRECOMMENDS:${PN} = "${VIRTUAL-RUNTIME_base-utils-syslog}"
PR = "r4"
inherit allarch
SRC_URI = "file://init \
file://exec \
file://nfsrootfs \
file://rootfs \
file://finish \
file://mdev \
file://udev \
file://e2fs \
file://debug \
file://lvm \
file://overlayroot \
"
S = "${WORKDIR}"
do_install() {
install -d ${D}/init.d
# base
install -m 0755 ${WORKDIR}/init ${D}/init
install -m 0755 ${WORKDIR}/nfsrootfs ${D}/init.d/85-nfsrootfs
install -m 0755 ${WORKDIR}/rootfs ${D}/init.d/90-rootfs
install -m 0755 ${WORKDIR}/finish ${D}/init.d/99-finish
# exec
install -m 0755 ${WORKDIR}/exec ${D}/init.d/89-exec
# mdev
install -m 0755 ${WORKDIR}/mdev ${D}/init.d/01-mdev
# udev
install -m 0755 ${WORKDIR}/udev ${D}/init.d/01-udev
# e2fs
install -m 0755 ${WORKDIR}/e2fs ${D}/init.d/10-e2fs
# debug
install -m 0755 ${WORKDIR}/debug ${D}/init.d/00-debug
# lvm
install -m 0755 ${WORKDIR}/lvm ${D}/init.d/09-lvm
# overlayroot needs to run after rootfs module but before finish
install -m 0755 ${WORKDIR}/overlayroot ${D}/init.d/91-overlayroot
# Create device nodes expected by some kernels in initramfs
# before even executing /init.
install -d ${D}/dev
mknod -m 622 ${D}/dev/console c 5 1
}
PACKAGES = "${PN}-base \
initramfs-module-exec \
initramfs-module-mdev \
initramfs-module-udev \
initramfs-module-e2fs \
initramfs-module-nfsrootfs \
initramfs-module-rootfs \
initramfs-module-debug \
initramfs-module-lvm \
initramfs-module-overlayroot \
"
FILES:${PN}-base = "/init /init.d/99-finish /dev"
# 99-finish in base depends on some other module which mounts
# the rootfs, like 90-rootfs. To replace that default, use
# BAD_RECOMMENDATIONS += "initramfs-module-rootfs" in your
# initramfs recipe and install something else, or install
# something that runs earlier (for example, a 89-my-rootfs)
# and mounts the rootfs. Then 90-rootfs will proceed immediately.
RRECOMMENDS:${PN}-base += "initramfs-module-rootfs"
SUMMARY:initramfs-module-exec = "initramfs support for easy execution of applications"
RDEPENDS:initramfs-module-exec = "${PN}-base"
FILES:initramfs-module-exec = "/init.d/89-exec"
SUMMARY:initramfs-module-mdev = "initramfs support for mdev"
RDEPENDS:initramfs-module-mdev = "${PN}-base busybox-mdev"
FILES:initramfs-module-mdev = "/init.d/01-mdev"
SUMMARY:initramfs-module-udev = "initramfs support for udev"
RDEPENDS:initramfs-module-udev = "${PN}-base udev"
FILES:initramfs-module-udev = "/init.d/01-udev"
SUMMARY:initramfs-module-e2fs = "initramfs support for ext4/ext3/ext2 filesystems"
RDEPENDS:initramfs-module-e2fs = "${PN}-base"
FILES:initramfs-module-e2fs = "/init.d/10-e2fs"
SUMMARY:initramfs-module-nfsrootfs = "initramfs support for locating and mounting the root partition via nfs"
RDEPENDS:initramfs-module-nfsrootfs = "${PN}-base"
FILES:initramfs-module-nfsrootfs = "/init.d/85-nfsrootfs"
SUMMARY:initramfs-module-rootfs = "initramfs support for locating and mounting the root partition"
RDEPENDS:initramfs-module-rootfs = "${PN}-base"
FILES:initramfs-module-rootfs = "/init.d/90-rootfs"
SUMMARY:initramfs-module-debug = "initramfs dynamic debug support"
RDEPENDS:initramfs-module-debug = "${PN}-base"
FILES:initramfs-module-debug = "/init.d/00-debug"
SUMMARY:initramfs-module-lvm = "initramfs lvm rootfs support"
RDEPENDS:initramfs-module-lvm = "${PN}-base"
FILES:initramfs-module-lvm = "/init.d/09-lvm"
SUMMARY:initramfs-module-overlayroot = "initramfs support for mounting a RW overlay on top of a RO root filesystem"
RDEPENDS:initramfs-module-overlayroot = "${PN}-base initramfs-module-rootfs"Okay so here we’re being told there’s multiple boot methods but not much else on how that is selected. so let’s go to a method. Live boot bitbake file simply installs a shell script:
SUMMARY = "Live image init script"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
DEPENDS = "virtual/kernel"
RDEPENDS:${PN} = "udev udev-extraconf"
SRC_URI = "file://init-live.sh"
PR = "r12"
S = "${WORKDIR}"
do_install() {
install -m 0755 ${WORKDIR}/init-live.sh ${D}/init
install -d ${D}/dev
mknod -m 622 ${D}/dev/console c 5 1
}
FILES:${PN} += " /init /dev "
# Due to kernel dependency
PACKAGE_ARCH = "${MACHINE_ARCH}"So what is that shell script it installs?
#!/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
ROOT_MOUNT="/rootfs"
ROOT_IMAGE="rootfs.img"
MOUNT="/bin/mount"
UMOUNT="/bin/umount"
ISOLINUX=""
ROOT_DISK=""
# Copied from initramfs-framework. The core of this script probably should be
# turned into initramfs-framework modules to reduce duplication.
udev_daemon() {
OPTIONS="/sbin/udev/udevd /sbin/udevd /lib/udev/udevd /lib/systemd/systemd-udevd"
for o in $OPTIONS; do
if [ -x "$o" ]; then
echo $o
return 0
fi
done
return 1
}
_UDEV_DAEMON=`udev_daemon`
early_setup() {
mkdir -p /proc /sys /run /var/run
mount -t proc proc /proc
mount -t sysfs sysfs /sys
mount -t devtmpfs none /dev
# support modular kernel
modprobe isofs 2> /dev/null
$_UDEV_DAEMON --daemon
udevadm trigger --action=add
}
read_args() {
[ -z "$CMDLINE" ] && CMDLINE=`cat /proc/cmdline`
for arg in $CMDLINE; do
optarg=`expr "x$arg" : 'x[^=]*=\(.*\)'`
case $arg in
root=*)
ROOT_DEVICE=$optarg ;;
rootimage=*)
ROOT_IMAGE=$optarg ;;
rootfstype=*)
modprobe $optarg 2> /dev/null ;;
LABEL=*)
label=$optarg ;;
video=*)
video_mode=$arg ;;
vga=*)
vga_mode=$arg ;;
console=*)
if [ -z "${console_params}" ]; then
console_params=$arg
else
console_params="$console_params $arg"
fi ;;
debugshell*)
if [ -z "$optarg" ]; then
shelltimeout=30
else
shelltimeout=$optarg
fi
esac
done
}
boot_live_root() {
# Watches the udev event queue, and exits if all current events are handled
udevadm settle --timeout=3 --quiet
# Kills the current udev running processes, which survived after
# device node creation events were handled, to avoid unexpected behavior
killall -9 "${_UDEV_DAEMON##*/}" 2>/dev/null
# Don't run systemd-update-done on systemd-based live systems
# because it triggers a slow rebuild of ldconfig caches.
touch ${ROOT_MOUNT}/etc/.updated ${ROOT_MOUNT}/var/.updated
# Allow for identification of the real root even after boot
mkdir -p ${ROOT_MOUNT}/media/realroot
mount -n --move "/run/media/${ROOT_DISK}" ${ROOT_MOUNT}/media/realroot
# Move the mount points of some filesystems over to
# the corresponding directories under the real root filesystem.
for dir in `awk '/\/dev.* \/run\/media/{print $2}' /proc/mounts`; do
# Parse any OCT or HEX encoded chars such as spaces
# in the mount points to actual ASCII chars
dir=`printf $dir`
mkdir -p "${ROOT_MOUNT}/media/${dir##*/}"
mount -n --move "$dir" "${ROOT_MOUNT}/media/${dir##*/}"
done
mount -n --move /proc ${ROOT_MOUNT}/proc
mount -n --move /sys ${ROOT_MOUNT}/sys
mount -n --move /dev ${ROOT_MOUNT}/dev
cd $ROOT_MOUNT
# busybox switch_root supports -c option
exec switch_root -c /dev/console $ROOT_MOUNT /sbin/init $CMDLINE ||
fatal "Couldn't switch_root, dropping to shell"
}
fatal() {
echo $1 >$CONSOLE
echo >$CONSOLE
exec sh
}
early_setup
[ -z "$CONSOLE" ] && CONSOLE="/dev/console"
read_args
echo "Waiting for removable media..."
C=0
while true
do
for i in `ls /run/media 2>/dev/null`; do
if [ -f /run/media/$i/$ROOT_IMAGE ] ; then
found="yes"
ROOT_DISK="$i"
break
elif [ -f /run/media/$i/isolinux/$ROOT_IMAGE ]; then
found="yes"
ISOLINUX="isolinux"
ROOT_DISK="$i"
break
fi
done
if [ "$found" = "yes" ]; then
break;
fi
# don't wait for more than $shelltimeout seconds, if it's set
if [ -n "$shelltimeout" ]; then
echo -n " " $(( $shelltimeout - $C ))
if [ $C -ge $shelltimeout ]; then
echo "..."
echo "Mounted filesystems"
mount | grep media
echo "Available block devices"
cat /proc/partitions
fatal "Cannot find $ROOT_IMAGE file in /run/media/* , dropping to a shell "
fi
C=$(( C + 1 ))
fi
sleep 1
done
# Try to mount the root image read-write and then boot it up.
# This function distinguishes between a read-only image and a read-write image.
# In the former case (typically an iso), it tries to make a union mount if possible.
# In the latter case, the root image could be mounted and then directly booted up.
mount_and_boot() {
mkdir $ROOT_MOUNT
mknod /dev/loop0 b 7 0 2>/dev/null
if ! mount -o rw,loop,noatime,nodiratime /run/media/$ROOT_DISK/$ISOLINUX/$ROOT_IMAGE $ROOT_MOUNT ; then
fatal "Could not mount rootfs image"
fi
if touch $ROOT_MOUNT/bin 2>/dev/null; then
# The root image is read-write, directly boot it up.
boot_live_root
fi
# determine which unification filesystem to use
union_fs_type=""
if grep -q -w "overlay" /proc/filesystems; then
union_fs_type="overlay"
elif grep -q -w "aufs" /proc/filesystems; then
union_fs_type="aufs"
else
union_fs_type=""
fi
# make a union mount if possible
case $union_fs_type in
"overlay")
mkdir -p /rootfs.ro /rootfs.rw
if ! mount -n --move $ROOT_MOUNT /rootfs.ro; then
rm -rf /rootfs.ro /rootfs.rw
fatal "Could not move rootfs mount point"
else
mount -t tmpfs -o rw,noatime,mode=755 tmpfs /rootfs.rw
mkdir -p /rootfs.rw/upperdir /rootfs.rw/work
mount -t overlay overlay -o "lowerdir=/rootfs.ro,upperdir=/rootfs.rw/upperdir,workdir=/rootfs.rw/work" $ROOT_MOUNT
mkdir -p $ROOT_MOUNT/rootfs.ro $ROOT_MOUNT/rootfs.rw
mount --move /rootfs.ro $ROOT_MOUNT/rootfs.ro
mount --move /rootfs.rw $ROOT_MOUNT/rootfs.rw
fi
;;
"aufs")
mkdir -p /rootfs.ro /rootfs.rw
if ! mount -n --move $ROOT_MOUNT /rootfs.ro; then
rm -rf /rootfs.ro /rootfs.rw
fatal "Could not move rootfs mount point"
else
mount -t tmpfs -o rw,noatime,mode=755 tmpfs /rootfs.rw
mount -t aufs -o "dirs=/rootfs.rw=rw:/rootfs.ro=ro" aufs $ROOT_MOUNT
mkdir -p $ROOT_MOUNT/rootfs.ro $ROOT_MOUNT/rootfs.rw
mount --move /rootfs.ro $ROOT_MOUNT/rootfs.ro
mount --move /rootfs.rw $ROOT_MOUNT/rootfs.rw
fi
;;
"")
mount -t tmpfs -o rw,noatime,mode=755 tmpfs $ROOT_MOUNT/media
;;
esac
# boot the image
boot_live_root
}
if [ "$label" != "boot" -a -f $label.sh ] ; then
if [ -f /run/media/$i/$ISOLINUX/$ROOT_IMAGE ] ; then
./$label.sh $i/$ISOLINUX $ROOT_IMAGE $video_mode $vga_mode $console_params
else
fatal "Could not find $label script"
fi
# If we're getting here, we failed...
fatal "Target $label failed"
fi
mount_and_boot
Voila! We’re getting somewhere. Okay so let’s look at nfs, so here it is:
https://github.com/openembedded/openembedded-core/blob/14241ed09f9ed317045cf75a6d08416d3579bb8d/meta/recipes-core/initrdscripts/initramfs-framework/nfsrootfs
#!/bin/sh
nfsrootfs_enabled() {
if [ ${bootparam_root} != "/dev/nfs" ] || [ -z ${bootparam_nfsroot} ]; then
return 1
fi
return 0
}
nfsrootfs_run() {
local nfs_opts
local location
local flags
local server_ip
nfs_opts=""
if [ "${bootparam_nfsroot#*,}" != "${bootparam_nfsroot}" ]; then
nfs_opts="-o ${bootparam_nfsroot#*,}"
fi
location="${bootparam_nfsroot%%,*}"
if [ "${location#*:}" = "${location}" ]; then
# server-ip not given. Get server ip from ip option
server_ip=""
if [ "${bootparam_ip#*:}" != "${bootparam_ip}" ]; then
server_ip=$(echo "$bootparam_ip" | cut -d: -f2)
fi
if [ -z "$server_ip" ]; then
fatal "Server IP is not set. Update ip or nfsroot options."
fi
location=${server_ip}:${location}
fi
flags="-o nolock"
if [ -n "$bootparam_ro" ] && ! echo "$bootparam_rootflags" | grep -w -q "ro"; then
if [ -n "$bootparam_rootflags" ]; then
bootparam_rootflags="$bootparam_rootflags,"
fi
bootparam_rootflags="${bootparam_rootflags}ro"
fi
if [ -n "$bootparam_rootflags" ]; then
flags="$flags -o $bootparam_rootflags"
fi
mount -t nfs ${flags} ${nfs_opts} ${location} ${ROOTFS_DIR}
}So what is all this bootparam_ something? Well, these are the kernel line arguments you pass! Being passed from the kernel to your init program as bootparam_ and whatever the passing argument was called. So you see how it asks whether ro is set (read only), what the ip is, etc… allllllll useful stuff. So, now we know how to get data from the kernel into a initrd script. We also know we can simply add them by updating our frameworks bitbake with a .bbappend and we know that this shell script should capture whether it is the right one and then mount the root file system.
One other note is look at how the first code segment I showed installs things, it gives them numbers, this is a prioritization order, the higher the number, the later in the process it is called. Each shell script will check the bootparam’s and confirm whether it is them who should run. Mostly they use the root parameter to confirm this.
So I know from my preliminary work that I don’t want NFS directly, I want this hybridised style of NFS and RAM. I know how to do it because of the article and from some initial testing of loading into a initramfs debug with the right tools (tar, wget, etc…) I could manually do this. So how do I make that into a script, well, eventually I came up with this:
#!/bin/sh
httprootfs_enabled() {
if [ ! -z ${bootparam_url} ]; then
return 0
fi
return 1
}
httprootfs_run() {
local name
echo "Running HTTP Sync Module..."
mount -t tmpfs -o size=95% none ${ROOTFS_DIR}
name = basename ${bootparam_url}
wget ${bootparam_url}
mount -o loop basename ${ROOTFS_DIR}
}
Super simple right? So let’s walk through, in any init script you need an enabled function which appends to the script name and a run function, this is the bare minimum. The enabled function is called first to confirm whether this is the right script by init to use. Here I simply say, if url is true, then we are http booting. In run I also say, hey, I’m booting this way! This was a kinda debugging step, and then I create a temporary filesystem which is 95% of my RAM, I then wget the bootparam_url and mount it, assuming it’s a tar, or something that you can mount as one file using -o loop.
So this worked well, it was booting from http just like ubuntu can. But, that’s not what I needed, I needed a NFS store system and I needed that so that if I eventually did want to write back to disk I could, like the rsync that the first article showed. I also want to share a large filesystem, almost 10GB so http might not work for my purposes. So I created this new system:
#!/bin/sh
ramnfsrootfs_enabled() {
if [ ! -z ${bootparam_ramnfs} ]; then
return 0
fi
return 1
}
ramnfsrootfs_run() {
echo "Running RAMNFS Sync Module..."
mkdir /ijkijk
mount -t nfs -o nolock ${bootparam_ramnfs} /ijkijk
mount -t tmpfs -o size=95% none ${ROOTFS_DIR}
cd /ijkijk
tar cv . | (cd ${ROOTFS_DIR} ; tar x)
cd /
umount /ijkijk
}
If ramnfs is a set boot parameter then this method gets used. Next I follow similar steps to article one, and voila, I have my mount system for my rootfs. Now, how do I get yocto to do this? I create an initramfs-framework_1.0.bbappend with the following:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
inherit allarch
SRC_URI += "file://httprootfs \
file://ramnfsrootfs \
"
S = "${WORKDIR}"
do_install_append() {
install -m 0755 ${WORKDIR}/httprootfs ${D}/init.d/85-httprootfs
install -m 0755 ${WORKDIR}/ramnfsrootfs ${D}/init.d/85-ramnfsrootfs
}
PACKAGES += " initramfs-module-httprootfs initramfs-module-ramnfsrootfs"
SUMMARY_initramfs-module-httprootfs = "initramfs support for locating and mounting the root partition via http iso"
RDEPENDS_initramfs-module-httprootfs = "${PN}-base"
FILES_initramfs-module-httprootfs = "/init.d/85-httprootfs"
SUMMARY_initramfs-module-ramnfsrootfs = "initramfs support for locating and mount the root partition as a ramboot from nfs"
RDEPENDS_initramfs-module-ramnfsrootfs = "${PN}-base"
FILES_initramfs-module-ramnfsrootfs = "/init.d/85-ramnfsrootfs"
Then I create a minimal-ramfs image with the bbappend to core-image-minimal-initramfs.bbappend:
# Add i915 graphics firmware
PACKAGE_INSTALL_append_intel-x86-common = " linux-firmware-i915"
IMAGE_INSTALL_append = "wget"
PACKAGE_INSTALL += "initramfs-module-debug base-passwd ${ROOTFS_BOOTSTRAP_INSTALL}"
INITRAMFS_SCRIPTS += "\
initramfs-module-nfsrootfs \
initramfs-module-httprootfs \
initramfs-module-ramnfsrootfs \
"
Note: tar seems to be preinstalled anyway but not wget.
Nothings finished yet, you still need to tell yocto that your system needs a initramfs, add this to your local.conf:
IMAGE_FSTYPES += " iso"
INITRAMFS_FSTYPES += " ext4 cpio.gz"
IMAGE_BOOT_FILES:intel-corei7-64 = "bzImage"
this will create, most importantly a cpio.gz, I use ext4 simply so I can mount it and inspect the files as a preliminary debugging setup, same for iso. cpio.gz is the important stuff, so, now I have that, it’s capable of booting my NFS in a RAM way, how do I tell my kernel to do it?
LABEL ramboot-yocto
MENU LABEL ramboot-yocto
KERNEL ramboot/bzImage
append initrd=ramboot/httpinitramfs.cpio.gz root=/dev/ramnfs ip=dhcp init_fatal_sh=true ramnfs=x.x.x.x:/tftpboot/ramboot/root debug=y vga=788
So here is what I placed in my NFS servers load file for menu. I tell it the kernel is ramboot/bzImage which it loads first. Then it runs and passes the arguments after append. The first thing is initrd, the kernel then goes “Hey, NFS, I need this”, so the NFS server passes it what is in the ramboot directory named httpinitramfs.cpio.gz. The kernel then unpacks this as it’s first filesystem and goes looking for a init program in /bin, /etc, /sbin. It finally finds my setup and then passes init the commands. Init then loads its scripts and passes each one all these parameters.
All of these then go, nope, until they hit mine, which goes, oh yeah, I’m looking for a ramnfs, it then tar pipes that rootfs and then loads that into memory and loads that as root using the command switch root. And that, is how it was done.