Creating the Debian Distribution for the Linksys WRT3200ACM
The previous article published and documented the files needed to get a stock Debian GNU/Linux distribution running on the Linksys WRT3200ACM router.
This article documents how I made those files, and where to find the configs needed to recreate them.
The OpenWrt/LEDE "Bootloader"
I use OpenWrt/LEDE as a mere bootloader for the Debian GNU/Linux system. The reaosn is that this is an excellent base for experiments, as you can always unplug the disk and boot into a standard OpenWrt environment, and also use OpenWrt's failsafe mode in case things go badly wrong. It also deals with this pesky flashing which I normally don't want to touch myself.
There is not much to be said about about creating the OpenWrt/LEDE firmware images - I cloned and prepared the LEDE git repository (see the LEDE documentation for any open question on what I do here):
git clone -b v17.01.4 https://git.lede-project.org/source.git lede-wrt3200 cd lede-wrt3200 ./scripts/feeds update -a ./scripts/feeds install -a
Then you need is a copy of my LEDE configuration:
wget -O .config http://dist.schmorp.de/wrt3200acm/lede/lede.config make oldconfig
Unfortunately, we also need some custom kernel options to make kexec work. I don't know how to do this officially, but the following works for me. First, prepare the kernel:
make kernel_menuconfig # wait, then immediately < Exit >
Then override the kernel config with mine:
wget -O ./build_dir/target-arm_cortex-a9+vfpv3_musl-1.1.16_eabi/linux-mvebu/linux-4.4.92/.config \ http://dist.schmorp.de/wrt3200acm/lede/linux-4.4.92.config make kernel_menuconfig # again, < Exit >
If you know a better or more reliable way of providing a custom kernel config, please speak up!
In any case, this is all that is needed. You can then proceed to build the firmware image:
make -j7
The resulting files can then be found in bin/targets/mvebu/generic.
How to get kexec working
Strictly speaking, all that is needed is an OpenWrt image with working kexec (both kernel support and userspace). Unfortunately, getting both was hard and took me a while.
OpenWrt has an option to enable kexec in its build options. Unfortunately, on ARM, the kernel will simply ignore this option unless either a) SMP support is disabled or b) CONFIG_PM_SLEEP is enabled. It doesn't help that by default, this makes the kexec option completely invisible in the configuration.
In addition to enabling the required options for kexec support, I also enable btrfs and f2fs to be able to boot without an initrd image and a lot of other options to make the kernel more "desktop-like". As a result you should be able to "boot" Debian within a container inside the OpenWrt image, if you should be so inclined.
This is not strictly necessary to merely make a bootloader for Debian, but it's an added feature that might be useful to debug things.
Anyways, the above takes care of the kernel side. The user space side of kexec is, unfortunately, also broken on OpenWrt for some years now. Fortunately, the Debian kexec binaries work fine in a chroot on OpenWrt, so currently the whole system uses the Debian kexec binaries instead of the OpenWrt ones.
The Debian Root Filesystem
The next big part is the Debian GNU/Linux root filesystem. A first step is to get a "virgin" Debian by using debootstrap (I used the one provided by OpenWrt):
mount /dev/sda2 /mnt debootstrap --arch=armhf stretch /mnt http://deb.debian.org/debian
This creates a rather minimal, unconfigured debian in /mnt. Since my OpenWrt bootloader can support debian binaries, I could then enter this with chroot, configure the image with debian tools and add some other important packages:
chroot /mnt bash vi /etc/fstab vi /etc/hostname vi /etc/apt/sources.list apt update apt install kexec-tools gcc make libncurses5-dev \ openssh-server openssh-client ...
I added a lot more packages, but the above ones are, AFAICR, the ones needed to compile the kernel and boot the system (right, no kernel, no boot).
I certainly installed more packages (such as hostapd) and probbaly added a few more files needed specifically for the WRT3200ACM, but I didn't modify any files that should clash with future upgrades or modified the image in some magic way - it should still be a pretty vasnilla debian system.
Extra Package: swconfig
Unfortunately, while debian has a very big pool of packaged software, swconfig is not packaged for debian. The reason is doubtlessly that swconfig onyl works with OpenWrt kernels, while vanilla kernels use something called DSA.
Fortunately, the DebWrt project has a debian package for swconfig, but despite its strong claim of supporting "any port/architecture supported by both Debian and OpenWrt. This includes MIPS, MIPS64 and MIPSEL", in practice they really only do support MIPS and x86 - there are no files for arm, supported by both OpenWrt and Debian.
But hey, the source package works fine (there are better ways of downloading these but it seems the DebWrt release key is hard to download at the moment). The following commands are all executed in the Debian chroot running on the OpenWrt image created earlier:
apt install packaging-dev wget http://packages.debwrt.net/debwrt/pool/main/libn/libnl-tiny/libnl-tiny_0.1.6~debwrtSTRETCH+4.dsc wget http://packages.debwrt.net/debwrt/pool/main/libn/libnl-tiny/libnl-tiny_0.1.6~debwrtSTRETCH+4.tar.xz dpkg-source -x libnl-tiny_0.1.6~debwrtSTRETCH+4.dsc ( cd libnl-tiny-0.1.6~debwrtSTRETCH+4/ && debian/rules binary ) wget http://packages.debwrt.net/debwrt/pool/main/s/swconfig11/swconfig11_0.0-1~debwrtSTRETCH+3.dsc wget http://packages.debwrt.net/debwrt/pool/main/s/swconfig11/swconfig11_0.0-1~debwrtSTRETCH+3.debian.tar.xz wget http://packages.debwrt.net/debwrt/pool/main/s/swconfig11/swconfig11_0.0.orig.tar.gz dpkg-source -x swconfig11_0.0-1~debwrtSTRETCH+3.dsc ( cd swconfig11-0.0/ && debian/rules binary )
If the above goes well, you can find a number of .deb packages in the current directory - you need to install these (and exqactly these are also preinstalled in the provided root filesysstem):
libnl-tiny0_0.1.6~debwrtSTRETCH+4_armhf.deb libsw11_0.0-1~debwrtSTRETCH+3_armhf.deb swconfig11_0.0-1~debwrtSTRETCH+3_armhf.deb
(PS: there are probably a gazllion cooler ways to build debian packages from source. I don't want to know about any of those, use them yourself and get happy!)
The Linux Kernel
At last, we need some good old linux kernel image (and possibly some modules) to get really happy, I mean, actually boot the system.
For some reason (cough, DSA, cough) I really wanted use the OpenWrt kernel. Also, the OpenWrt kernel is known to work on my device and contains, like, a hundred patches to improve stability or so.
But wanting the kernel is one thing, actually geting the kernel sources is another. Here is what I found out.
First, get a development snapshot of LEDE (I checked out whatever was current, specifically, I got commit 5cd48280fd78b3cacdab14ca89fc17e58f642e68).
git clone -b v17.01.4 https://git.lede-project.org/source.git lede-debiankernel cd lede-debiankernel wget -O .config http://dist.schmorp.de/wrt3200acm/lede/lede.config make oldconfig
Remember, all this is just needed to get the kernel sources. The reason I made a separate git clone is that the first clone was to build a stable LEDE firmware (kernel 4.4.92), while the second clone really is just needed for the most current kernel (4.9.58). You could check this out in the first source tree, but keeping them separated makes development easier.
You also don't need my .config - all you need to do is make menuconfig and select the Marvell Armada 37x/38x/XP Target System, and the Linksys WRT3200ACM (Rango) Target Profile.
To get the actual kernel sources, you need to execute these magic lines:
make target/linux/clean target/linux/prepare V=s QUILT=1 cd build_dir/target-*/linux-*/linux-*/ quilt push -a
The last line applies all OpenWrt patches. Funny anecdote: at first, I forgot the quilt push -a and as a result got a standard linux kernel. It actually kind of booted as well at first try, and I was mightily impressed by how well the OpenWrt kernel was maintained - only much later, when debugging why the switch config doesn't work, did I realise my mistake :)
I any case, you should now be in the (quilted) 4.9.58 source tree with all OpenWrt patches applied. Quickly make a copy of that directory and keep it :)
Now that we have the kernel sources, we don't need the git clone anymore and can delete it (again, after saving or moving away the generated kernel source tree).
To build the kernel, you should start with my kernel config (because it works), but otherwise feel free to experiment, of course:
wget -O .config http://dist.schmorp.de/wrt3200acm/debian/linux-4.9.58.config make oldconfig
The kernel config should also be available on the root filesystem as /boot/config-4.9.58.
As with debian source packages, there is a gazillion ways to do it properly, none of which work for me. Or rather, the only proper way to get a debian package ouit of a kernel source tree for me was to use the kernel Makefile:
make -j2 bindeb-pkg
This takes about 45 minutes to build the kernel packages on the router, which I think is quite fast. The router also gets quite hot - at 104°C I decided to add a little fan action to cool it down.
The result were various .deb files in the parent directory (the names can differ slightly):
linux-firmware-image-4.9.58_4.9.58-5_armhf.deb linux-headers-4.9.58_4.9.58-5_armhf.deb linux-image-4.9.58_4.9.58-5_armhf.deb linux-libc-dev_4.9.58-5_armhf.deb
The linux-firmware-image package can be useful, but is not urgently needed as Debian has firmware packages in it's own repository.
The linux-libc-dev package is also provided by Debian (and is newer in Debian), but can be used. It is needed to compile userspace progams that need kernel headers.
The linux-headers package is needed to compile kernel modules, such as the mwlwifi driver for the main WiFi chips in the router.
And finally, the linux-image package contains the kernel and corresponding modules. This is the only package required to actually boot.
In the root filesystem I provide, both the linux-image and the linux-headers package are installed already.
The configuration for the kernel compiles in all options for hardware on board, except the wifi drivers, compiles in btrfs and f2fs to be able to boot these filesystems, and tries to enable everything else that could even be remotely useful (USB VGA display anybody?) as a module. It also conifugred the OpenWrt kernel to include all "desktop" features that system requires, and is not a stripped down kernel config as usually provided by OpenWrt. It should look and feel pretty much like the kernel you use on your amd64 desktrop.
The only notable exception is the DMA engine support (Marvell XOR Engine, mv_xor). If that is enabled, the kernel doesn't boot, and since my serial adapter is still broken, I can't tell why. But I have a hunch, as the standard OpenWrt/LEDE kernel configuration enables it and the kernel complains about memory corruption directly after the mv_xor selftest fails, but still manages to boot. Disabling mv_xor in the OpenWrt/LEDE kernel gets rid of the memory corruption complaints.
And That's It!
And that indeed is it - if I have forgotten some detail you are interested in or you have some comments, please drop me a mail. Likewise, if you used these instructions and want to tell me about it, drop me a mail as well.