Raspios 64-bit recently went GA and I recently found new joy in my Raspberry Pi 3B+ and Raspberry Pi Zero 2 W devices.
Now feels like a great time to revisit diskless booting.
This is what I'm using:
- UniFi for DHCP
- QNAP NAS for TFTP and ISCSI
- Raspberry Pi 3B+
Install RaspiOS 64-bit
Use rpi-imager and install the GA version of 64-bit RaspiOS!
Enable USB Boot Mode
The devices that I have are 4-6 years old. I'm not sure if new ones will have this set already or not.
Run this command, the "OTP dump", to see if usb_boot_mode is enabled:
vcgencmd otp_dump | grep 17:
This is what my OTP dump looked like before usb_boot_mode enabled.
17:1020000a
If the output looks like this, get the latest firmware:
sudo rpi-update
I was a little nervous about the rpi-update, but I'm also following my own notes.
*** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
*** Performing self-update
*** Relaunching after update
*** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
*** We're running for the first time
*** Backing up files (this will take a few minutes)
*** Backing up firmware
*** Backing up modules 5.10.63-v8+
#############################################################
WARNING: This update bumps to rpi-5.15.y linux tree
See: https://forums.raspberrypi.com/viewtopic.php?t=322879
'rpi-update' should only be used if there is a specific
reason to do so - for example, a request by a Raspberry Pi
engineer or if you want to help the testing effort
and are comfortable with restoring if there are regressions.
DO NOT use 'rpi-update' as part of a regular update process.
##############################################################
Would you like to proceed? (y/N)
*** Downloading specific firmware revision (this will take a few minutes)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 173 100 173 0 0 678 0 --:--:-- --:--:-- --:--:-- 678
100 123M 0 123M 0 0 4052k 0 --:--:-- 0:00:31 --:--:-- 3014k
*** Updating firmware
*** Updating kernel modules
*** depmod 5.15.21-v8+
*** depmod 5.15.21-v7+
*** depmod 5.15.21-v7l+
*** depmod 5.15.21+
*** Updating VideoCore libraries
*** Using SoftFP libraries
*** Updating SDK
*** Running ldconfig
*** Storing current firmware revision
*** Deleting downloaded files
*** Syncing changes to disk
*** If no errors appeared, your firmware was successfully updated to fbcab73d5a20f592aa9f6b0e364757ef131dae27
*** A reboot is needed to activate the new firmware
After rebooting from the rpi-update we still need to enable usb_boot_mode, just needed the latest firmware to do so.
Add this line to /boot/config.txt it doesn't matter where, but needs its own line. I put it at the top.
program_usb_boot_mode=1
Then reboot the device again. Now the OTP dump should look like this:
17:3020000a
Now that the OTP dump is set, it is safe to remove the program_usb_boot_mode=1 line from /boot/config.txt file. There is no need to reboot, but if you remove the line, and reboot, you will see that the OTP dump still shows usb_boot_mode enabled.
Enable ISCSI module with initramfs tool
These are the steps I took:
# Update repositories
sudo apt update
# Install available upgrades
sudo apt upgrade -y
# Install open-iscsi
sudo apt install open-iscsi
# Start open-iscsi to create the configs that we will use later
sudo systemctl start open-iscsi
# Create initramfs flag file for the iscsi module
sudo touch /etc/iscsi/iscsi.initramfs
# Create a new initramfs image that includes the iscsi module from the current kernel
sudo update-initramfs -v -k `uname -r` -c
You should now have an initrd.img in your /boot directory.
ls -latr /boot/initrd.img*
The output should look like this:
-rwxr-xr-x 1 root root 10487750 Feb 11 10:59 /boot/initrd.img-5.15.21-v8+
We will refer to this new initrd.img file later!
Add TFTP options to DHCP server
At this point, the bootloader on our device will accept a network boot, similar to PXE, but not exactly the same.
Setup your DHCP server to point to your TFTP server. The DHCP protocol has Option 66 for TFTP Server. Basically, you want to tell your DHCP client (the Raspberry Pi 3B+) which server to connect to next, to get further instructions.
Setup TFTP Server
The network boot process for the Raspberry Pi 3B+ is PXE-like. So it requires a slightly custom setup. It requires the serial number of the device, to be known in advance.
Grab the serial number of the device:
grep Serial /proc/cpuinfo
In my case it looks like this:
Serial : 000000002edd5668
On the TFTP server, create a directory for the device. In the case of my QNAP server, the TFTP root is at /share/tftp but I'll use the variable TFTP_ROOT.
# Set the implementation specific TFTP_ROOT
export TFTP_ROOT=/share/tftp
# Create directory using the device serial number, without leading 0s
sudo mkdir $TFTP_ROOT/2edd5668
Now, from the TFTP server, copy over the /boot partition, of the Raspberry Pi, that includes our new kernel.
sudo scp -r pi@2edd5668:/boot/* $TFTP_ROOT/2edd5668/
If you want to check your work, the directory should look something like this:
COPYING.linux* bcm2708-rpi-cm.dtb* bcm2710-rpi-3-b-plus.dtb* bcm2711-rpi-4-b.dtb* cmdline.txt* fixup4db.dat* initrd.img-5.15.21-v8+* kernel8.img* start4db.elf*
LICENCE.broadcom* bcm2708-rpi-zero-w.dtb* bcm2710-rpi-3-b.dtb* bcm2711-rpi-400.dtb* config.txt* fixup4x.dat* issue.txt* overlays/ start4x.elf*
bcm2708-rpi-b-plus.dtb* bcm2708-rpi-zero.dtb* bcm2710-rpi-cm3.dtb* bcm2711-rpi-cm4.dtb* fixup.dat* fixup_cd.dat* kernel.img* start.elf* start_cd.elf*
bcm2708-rpi-b-rev1.dtb* bcm2709-rpi-2-b.dtb* bcm2710-rpi-zero-2-w.dtb* bcm2711-rpi-cm4s.dtb* fixup4.dat* fixup_db.dat* kernel7.img* start4.elf* start_db.elf*
bcm2708-rpi-b.dtb* bcm2710-rpi-2-b.dtb* bcm2710-rpi-zero-2.dtb* bootcode.bin* fixup4cd.dat* fixup_x.dat* kernel7l.img* start4cd.elf* start_x.elf*
Now, because this is PXE-like, we need to copy the bootcode.bin to the TFTP_ROOT
sudo cp $TFTP_ROOT/2edd5668/bootcode.bin $TFTP_ROOT/
At this point, without an SD Card installed, the Raspberry Pi 3B+ will:
- Ask the DHCP server for an IP address
- Our DHCP server will provide an IP address AND tell the device to visit the TFTP server for more info
- Our device will ask the TFTP server for $TFTP_ROOT/bootcode.bin
- Our device will try to boot from $TFTP_ROOT/2edd5668/
At this point though, our $TFTP_ROOT/2edd5668/ exactly matches what is on the SD Card in our device. We need to tell the device to use our new ISCSI enabled kernel.
I added the following line to the bottom of $TFTP_ROOT/2edd5668/config.txt, but you should use your specific initrd.img version.
initramfs initrd.img-5.15.21-v8+ followkernel
Create ISCSI LUN for the device
For simplicity, I'm creating a 32GB LUN, with "thick" provisioning on my NAS, and mapping that to an ISCSI Target. I chose 32GB simply because that is the size of SDcard that I have been using. I also used the serial number, of the Raspberry Pi, in the naming of the LUN and ISCI target.
To verify that your new ISCSI LUN is accessible we will connect to it from the Raspberry Pi 3B+.
# set the IP address of the ISCSI server
export ISCSI_IP=192.168.1.248
#
sudo iscsiadm -m discovery -t sendtargets -p $ISCSI_IP | grep 2edd5668
The output should contain the new ISCSI target.
192.168.1.248:3260,1 iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c
From above the TARGET_IQN is iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c and I create an enviroment varialbe to store it.
export TARGET_IQN=iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c
Mount the ISCSI LUN to the Raspberry Pi
Login to the ISCSI LUN, this example is not using CHAP authentication.
sudo iscsiadm -m node -l -T $TARGET_IQN -p $ISCSI_IP
Success should look similar to this:
Logging in to [iface: default, target: iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c, portal: 192.168.1.248,3260]
Login to [iface: default, target: iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c, portal: 192.168.1.248,3260] successful.
I validate the new ISCSI connection using sudo fdisk -l from the Raspberry Pi 3B+ and look for something like this, showing the 32GB LUN attached via ISCSI
Disk /dev/sda: 32 GiB, 34359738368 bytes, 67108864 sectors
Disk model: iSCSI Storage
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 1048576 bytes / 1048576 bytes
The LUN is showing up as /dev/sda in this case.
# Format the LUN attached as /dev/sda to ext4
sudo mkfs.ext4 /dev/sda
mke2fs 1.46.2 (28-Feb-2021)
Creating filesystem with 8388608 4k blocks and 2097152 inodes
Filesystem UUID: f581fa52-f7ea-4c87-91cc-e236d64f63a6
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624
Allocating group tables: done
Writing inode tables: done
Creating journal (65536 blocks): done
Writing superblocks and filesystem accounting information: done
# Get the UUID of the LUN
sudo blkid /dev/sda
/dev/sda: UUID="f581fa52-f7ea-4c87-91cc-e236d64f63a6" BLOCK_SIZE="4096" TYPE="ext4"
# Mount the new disk
sudo mount /dev/sda /mnt
Copy the existing installation to the mounted LUN
This step takes some time. We are copying all of the files from the / partition of the SD card on the Raspberry Pi 3B+ to the network attached LUN. We are excluding the directories that are mounted during boot.
# Copy / partition to /mnt
sudo rsync -avhP --exclude /boot --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/
# Create directories that are mounted during boot
sudo mkdir /mnt/{dev,proc,sys,boot,mnt}
Remove SD Card details from /etc/fstab
cat /mnt/etc/fstab
Before:
proc /proc proc defaults 0 0
PARTUUID=9633a3a2-01 /boot vfat defaults 0 2
PARTUUID=9633a3a2-02 / ext4 defaults,noatime 0 1
# a swapfile is not a swap partition, no line here
# use dphys-swapfile swap[on|off] for that
Remove the lines containing PARTUUID which are referring to the SDCard.
After:
proc /proc proc defaults 0 0
# a swapfile is not a swap partition, no line here
# use dphys-swapfile swap[on|off] for that
Update the kernel configs on TFTP server
Now that we have updated our filesystem. We need to revisit kernel that is getting loaded over the network. We need to tell it how to mount the / filesystem via ISCSI.
The ISCSI Initiator, which identifies the Raspberry Pi 3B+ device, can be found at /etc/iscsi/initiatorname.iscsi
sudo cat /etc/iscsi/initiatorname.iscsi
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator. The InitiatorName must be unique
## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.
InitiatorName=iqn.1993-08.org.debian:01:2ccb20a316
On the TFTP server, I'm modifying the $TFTP_ROOT/2edd5668/cmdline.txt file.
I'm adding/updating these details for mounting the ISCSI drive:
- ip=::::2edd5668:eth0:dhcp
- hostname of the device
- root=UUID=f581fa52-f7ea-4c87-91cc-e236d64f63a6
- From the blkid command against the LUN
- ISCSI_INITIATOR=iqn.1993-08.org.debian:01:2ccb20a316
- ISCSI_TARGET_NAME=iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668..
- ISCSI_TARGET_IP=192.168.1.248
- ISCSI_TARGET_PORT=3260
- rw
Before:
console=serial0,115200 console=tty1 root=PARTUUID=9633a3a2-02 rootfstype=ext4 fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_enable=memory cgroup_enable=pids cgroup_memory=1 systemd.unified_cgroup_hierarchy=0
After:
console=serial0,115200 console=tty1 root=PARTUUID=9633a3a2-02 rootfstype=ext4 fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_enable=memory cgroup_enable=pids cgroup_memory=1 systemd.unified_cgroup_hierarchy=0 ISCSI_INITIATOR=iqn.1993-08.org.debian:01:2ccb20a316 ISCSI_TARGET_NAME=iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c ISCSI_TARGET_IP=192.168.1.248 ISCSI_TARGET_PORT=3260 rw
This all needs to be on a single line.
Ready to reboot!
From the Raspberry Pi 3B+ device:
# Unmount the drive
sudo umount /mnt
# Disconnect from ISCSI
sudo iscsiadm -m node --logout all
# Shutdown
sudo shutdown now
Remove the SD card and restart the device!
Summary
If you were paying close attention, you may have notice that the cmdline.txt also had some non-default settings in it. Those are there to allow k3s to run as well.
This article is very specific to the Raspberry Pi 3B+ and using ISCSI. Look for a follow-up example using the Raspberry Pi 4.
Soon, I'll document a diskless-pipeline for setting up Raspberry Pi 3B+ and Raspberry Pi 4, that includes using arkade and k3sup to configure the kubernetes abstraction layer across all of these homelab devices.