THE RANT / THE SCHPLOG
Schmorp's POD Blog a.k.a. THE RANT
a.k.a. the blog that cannot decide on a name

This document was first published 2017-11-11 20:20:32, and last modified 2017-11-24 07:30:25.

GNU/Debian on the Linksys WRT3200ACM

The "Why?"

There are so many things on my to-blog-about list, but the following wasn't. Basically, I got a new toy, a Linksys. The main appeal was that it had copious amounts of RAM (512MB, although in practice "only" 300MB usable), a programmable ethernet switch, 802.11ac wireless and, most importantly, promimently features "OpenWrt-capable" on the box, at a time where more and more vendors try to lock down their devices against custom firmware.

But OpenWrt (and its temporary replacement, LEDE) was always irksome to me - updates (even security updates) were manual and the package selection was limited (no working development environment for example).

Unfortunately, with the cheap plastic routers many people (including me) use, you can't just put a USB stick and boot Debian GNU/Linux - making a distribution for these small SOHO routers is difficult work, as you don't have a keyboard or screen, but extremely limited "disk" (=flash) space and RAM.

OpenWrt does an absolutley splendid job at that! So splendid, that I thought to combine them and use OpenWrt as a mere boot loader for a stock Debian GNU/Linux system on disk (the WRT3200ACM also has a 5V eSATA-p port, but also supports USB-SATA adapters). It also has a great web interface to administrate the router, something I already missed before I started.

There are already Debian GNU/Linux distributions for routers in general (DebWrt) and the Linksys WRT3200ACM (and earlier models) in particular (McDebian).

However, DebWrt, despite claiming to support all OpenWRT architectures, in practise only binaries for MIPS, and McDebian looked very cumbersome and inflexible to me, specifically, there is very little deocumentation on how to build the kernel. Moreso, it doesn't use the well-maintained OpenWrt kernel but a standard kernel, which uses DSA instead of swconfig, and DSA sucks.

So I set out to do my own, with as little hackery as possible, producing both some ready-to-use files (that might save you days of experimenting) and the background information on how to replicate this yourself.

The Design

Early Beginnings

OpenWrt, as I already mentioned, does a splendid job, making sure things such as flashing and failsafe mode work, which is especially useful when, as happened to me, your serial-to-ttl converter failed, and a new one takes a month to ship (it still hasn't arrived, so I effectively had to do all this blind).

So at first, I thought about running DebWrt in a chroot environment. In fact, I did exactly that on my older TP-Link WDR3600. From its /etc/rc.local):

ROOT=/debwrt
cd "$ROOT" || exit 1
[ "$(chroot . unshare echo hi)" = hi ] || exit 1
killall sshd
mount --bind . .
cd "$ROOT"
mkdir -m 0 -p openwrt
mount --rbind / openwrt
exec chroot . unshare -p -f /sbin/init

This kills the OpenWrt sshd and then boots systemd in a Linux container. I let OpenWrt do the network stuff, but Debian GNU/Linux gets the ssh server, so when I ssh to it, I end up in a Debian environment with systemd.

That meant I had the OpenWrt GUI to administrate the system, but a Debian system to play around.

And also two systems I had to provide security updates for.

The Current Design

So I decided to boot Debian GNU/Linux directly with its own kernel, using OpenWrt as a mere bootloader. That seems to combine most advantages - OpenWrt for stability and recovery, Debian for, well, Debian.

So I built a custom OpenWrt (rather, LEDE) image with kexec support which I can use for recovery or even to run the router fully, whose only real purpose is to mount an external disk and boot the Debian GNU/Linux on it.

The Files

After these lengthy introductions, let's look at the the actual files that you can use to run Debian on your WRT3200ACM with minimal danger. With possibly some minor work, these should also work on WRT1900AC and WRT1200AC models, although I have no way to test it.

Note also that I haven't tested the following explanations exactly, either, but the OpenWrt/LEDE sysupgrade image, the kernel and the Debian root filesystem have been tested.

So, what do you need?

When Things go Wrong

First, though, you need to know what to do when things go wrong. The WRT3200ACM has numerous failsafe/recovery features. Apparently (not tested!) you can even recover from a completely empty flash (or broken U-Boot!), and unless you really do erase your flash, you can recover even from half-flashed OpenWrt images easily.

So here are the ways you can recover your router, in increasing order of desperateness:

Disable Debian Autoboot

That one is easy - when something goes wrong with Debian, unplug the disk and OpenWrt will boot normally. Alternatively, you can mount the disk and make the autoboot file non-executable, e.g.:

# from within OpenWrt/LEDE
mount /dev/sda2 /mnt
chmod 0 /mnt/autoboot

# from a booted Debian GNU/Linux
chmod 0 /autoboot

A chmod 755 will make it bootable again.

OpenWrt Failsafe Mode

When your OpenWrt "bootloader" doesn't work, you can boot into the OpenWrt failsafe mode. To do this, switch the router off and on again, and start toggling the small blue button (the on/off switch is the rocker swtch at the back on the left (seen from the front), the small blue button is the, well, small blue button the right). Just press and let go the blue button about once a second.

First, the router will blink slowly, then faster, and at least, very fast (10 Hz). When it blinks that fast, youc na let go and 10-20s later, you should be able to connect to 192.168.1.1 from another computer:

ip addr add 192.168.1.2/24 dev eth0
ssh 192.168.1.1

Read the login message for more instructions - basically, this will give you an unconfigured default OpenWrt installation.

Second Flash Area

The WRT3200ACM has a second flash area which usually contains either the original Linksys firmware, or another virgin OpenWrt image. If OpenWrt is not in failsafe mode but all your config files are gone, you probably switched to this second flash area, and switching back will restore your settings.

To toggle between the two flash areas, use this method:

Switch off the router. Switch on the router, and, when the Power (leftmost) LED lights up, instantly switch off again. Repeat this three times. When you now switch the router on again it should boot from the "other" flash area.

This method toggles the flash area used, so you can go back and forth by doing it.

Recovering from a Dead U-Boot

To recover from a dead U-Boot (the bootloader), you need a serial console and some extra software, see these instructions.

Finally... The Files!

Ok, so here is how you get Debian GNU/Linux booting on your WRT3200ACM.

First acquaint yourself with the files I made available

The lede/ subdirectory contains OpenWrt/LEDE firmware images and extra kernel modules (kmod-ipk/) in case you need to add more modules to your OpenWrt bootloader. The LEDE image is based on the 17.01.4 release - since I only use this as bootloader for Debian, security updates are not very pressing to me so don't expect many updates.

The debian/ subdirectory contains the root filesystem (rootfs.tar.xz) which contains a Debian slightly customised to boot out of the box, swconfig and a working kernel. Some extra Debian packages that you don't really need are in deb/, maybe they are useful.

Installing the "Bootloader"

Next you need to install the OpenWrt "bootlaoder", which is really just a standard OpenWrt/LEDE image configured to support kexec.

You can either download a firmware image for uploading via the stock firmware (untested) or you can download sysupgrade image for use in your existing OpenWrt/LEDE environment .

Choose the one you need. For example, you can either upload the sysupgrade image via LuCi, or somehow copy it to your router and use the sysupgrade command (both of these will reset your settings unless you protect them, as with every OpenWrt upgrade):

wget -O /tmp/sysupgrade.bin http://dist.schmorp.de/wrt3200acm/lede/lede-mvebu-linksys-wrt3200acm-squashfs-sysupgrade.bin
sysupgrade /tmp/sysupgrade.bin

Other than coming with a lot of packages I feel useful (such as OpenSSH), these are fairly standard OpenWrt/LEDE images, and could be used instead of a standard 17.01.4 release - apart from the kernel modules, it should be completely compatible to that OpenWrt/LEDE release.

After the upgrade, OpenWrt will be available as 192.168.1.1, just as if it were in failsafe mode. You can follow the instructions in the First Login page in the OpenWrt wiki to further configure your image. The only hard requirement, though, is to login via SSH, set a password and reboot.

To make it into a Debian bootloader, you then need to modify the /etc/rc.local file to look like this (by logging in via SSH and using vi /etc/rc.local for example):

# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

DEB_DEVICE=/dev/sda2
DEB_FSTYPE=btrfs
#DEB_OPTION=subvol=/debian

# mount and execute /autoboot if it exists
mount -n -r -o "$DEB_OPTION" -t "$DEB_FSTYPE" "$DEB_DEVICE" /mnt &&
   cd /mnt &&
   [ -x ./autoboot ] &&
   mtd resetbc s_env &&
   . ./autoboot

# otherwise leave it mounted

exit 0

All that this does is mount /dev/sda2 and, if that was successful and there is an executable /autoboot script on it, run it. Ok, before running it it resets the boot counter, so that U-Boot doesn't boot into the other partition after three reboots, but that really is it.

You can change the device with DEB_DEVICE, the filesystem type with DEB_FSTYPE and any mount options with DEB_OPTION (which is commented out above, as no special options are used. The example can be used to boot a different btrfs subvolume, e.g. for testing).

Even with this minor modification, the system will still behave like a normal OpenWrt installation unless a disk is attached with something bootable.

That's all that's needed on the OpenWrt side to boot other our external Debain GUN/Linux. However, the not-so-high-quality marvell driver for the wireless cards will need more work: If OpenWrt loads the driver (and the firmware) then trying to load it again under debian will just make the module hang.

The fix for this is to disable the mwlwifi module on the OpenWrt side, by editing /etc/modules.d/50-mwlwifi and commenting out the mwlwifi line (or actually all lines). You should give the same treatment to /etc/modules.d/mwifixex_sdio.

This will make OpenWrt skip loading the modules, which not only makes the driver(s) work later when in debian, but also reduces boot time, as loading the mwlwifi driver takes a long time.

Preparing the Debian Disk

After this works and you had fun with your new custom OpenWrt installation, you can set about preparing the Debian system disk.

At the moment, you have to use a SATA disk, and you have to use either btrfs (for any device) or f2fs (for slow flash) as filesystem. To use usb-storage or another filesystem you would need to use an initramfs as the provided kernel only comes with SATA, btrfs and f2fs built-in. Or you can build your own kernel. Or you can pretty please ask me to recompile the kernel with usb-storage or support an initramfs out of the box.

By default, you need a disk with a swap partition, followed by a btrfs partition. To partition the disk, you need a 2.5" disk (or a 3.5" disk with external power supply) attached to the eSATA port of the router. Actually, you can do it on any other computer, but to boot from it it needs to be attached to the sATA port.

First, absolutely make sure the disk you want to use has no important or unbacked-up data on it. Then you can wipe, partition and format it like this (this is only your input, see below to see a full typescript, and yes, empty lines mean you enter an empty line):

wipefs -a /dev/sda
gdisk /dev/sda
n


+2G
8200
n




p
w
y
mkswap -L SWAP /dev/sda1
mkfs.btrfs -L WRT3200ROOT /dev/sda2

Here is what it looks like when you enter the above commands in an ssh session on the router, i.e. including the command output. Or at least, here is what it should look like:

root@wrt:~# wipefs -a /dev/sda
/dev/sda: 2 bytes were erased at offset 0x000001fe (dos): 55 aa
/dev/sda: calling ioctl to re-read partition table: No error information
root@wrt:~# gdisk /dev/sda
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
  MBR: not present
  BSD: not present
  APM: not present
  GPT: not present

Creating new GPT entries.

Command (? for help): n
Partition number (1-128, default 1): 
First sector (34-312581774, default = 2048) or {+-}size{KMGTP}: 
Last sector (2048-312581774, default = 312581774) or {+-}size{KMGTP}: +2G
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (2-128, default 2): 
First sector (34-312581774, default = 4196352) or {+-}size{KMGTP}: 
Last sector (4196352-312581774, default = 312581774) or {+-}size{KMGTP}: 
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): 
Changed type of partition to 'Linux filesystem'

Command (? for help): p
Disk /dev/sda: 312581808 sectors, 149.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): C770FCCF-88D8-4832-8D0F-1E37E1F3C5A3
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 312581774
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         4196351   2.0 GiB     8200  Linux swap
   2         4196352       312581774   147.0 GiB   8300  Linux filesystem

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sda.
The operation has completed successfully.
root@wrt:~# mkswap -L SWAP /dev/sda1
Setting up swapspace version 1, size = 2147479552 bytes
root@wrt:~# mkfs.btrfs -L WRT3200ROOT /dev/sda2
btrfs-progs v4.7.2
See http://btrfs.wiki.kernel.org for more information.

Label:              WRT3200ROOT
UUID:               44936f7e-2e79-45b4-9d1f-c8c034699761
Node size:          16384
Sector size:        4096
Filesystem size:    147.05GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1   147.05GiB  /dev/sda2

root@wrt:~# 

You don't have to this exactly as shown, but by default my instructions require a swap partition on /dev/sda1 and the Debian root btrfs on /dev/sda2.

Preparing the Root Filesystem

What's left is preparing the root filesystem. This took me ages, so count yourself fortunate that you can just download the rootfs.tar.xz from me.

All the commands in this section require the new root partition to be mounted as /mnt):

mount /dev/sda2 /mnt

You can download and unpack it in one go (it won't fit into either the WRT3200ACM RAM or flash):

curl http://dist.schmorp.de/wrt3200acm/debian/rootfs.tar.xz | xz -d | tar xC /mnt

Or you can do it in two steps:

wget -O /mnt/rootfs.tar.xz http://dist.schmorp.de/wrt3200acm/debian/rootfs.tar.xz
</mnt/rootfs.tar.xz xz -d | tar xC /mnt

Or you can download the file on another computer and so on and so on.

In the end, you should have a /dev/sda2 partition with about 500MB of files on it that resemble a Unix root filesystem.

The next, recommended but not required (your security is not my security), step is to regenerate the SSH host keys:

rm /mnt/etc/ssh/ssh_host_*_key*
ssh-keygen  -t ed25519 -N "" -f /mnt/etc/ssh/ssh_host_ed25519_key
ssh-keygen  -t ecdsa -N "" -f /mnt/etc/ssh/ssh_host_ecdsa_key
ssh-keygen  -t rsa -N "" -f /mnt/etc/ssh/ssh_host_rsa_key

To be able to login (no password login is enabled), you need to edit the file /mnt/root/.ssh/authorized_keys and replace the public key inside with your public key. It must be an RSA, ECDSA or ED25519 key.

There already is a key provided which you could copy out (cp /mnt/root/.ssh/id_ed25519) and use (ssh -i id_ed25519 ...), but obviously, this key is public and thus compromised, so really do provide your own key.

Last, you can edit the network configuration (/mnt/etc/systemd/network/eth0.network, /mnt/etc/resolv.conf) and/or the hostname (/mnt/etc/hostname), but you could also just boot it and configure it afterwards.

At this point, you can unmount the partition, attach it to your router (unless it's already attached of course) and reboot:

umount /mnt
reboot

Booting/Using

If all goes well, the router should first boot OpenWrt (about 40-50s), followed by booting Debian GNU/Linux (another 20-30s).

You should then be able to login by ssh'ing to 192.168.1.1 (same address as OpenWrt by default) or to whatever address you configured.

The switch is configured the same way as the factory-default (I think), that is, you can only connect to it by using one of the numbered ethernet ports, not the WAN port.

When that is successful, you should (if you haven't already) configure the real IP address a default gateway and your nameserver. Being able to reach the internet is important as the real time clock does not work (probably due to a driver bug) and NTP is needed to get the correct time, which in turn is required to successfully compile a kernel for example.

Regardless, you should now have a pretty standard Debian installation. The package unattended-upgrades is installed and will do regular security updates. The mwlwifi driver is installed and ready to use (but I haven't tested it yet).

Updating the MWLWIFI driver

To update the driver, you need to install git, and then:

cd /usr/src/mwlwifi
make -C /lib/modules/`uname -r`/build M=$PWD modules modules_install clean
depmod -a
cp bin/firmware/*.bin /lib/firmware/mwlwifi/

Compiling a new kernel (if needed) is described later.

Chosing which kernel to boot/Using an initrd

The /autoboot script started by OpenWrt chooses a kernel and optionally an initrd. To see how that works, let's look at the script:

# DEB_DEVICE, DEB_FSTYPE d DEB_OPTION inherited from openwrt's rc.local

DEB_KERNEL=vmlinuz
DEB_INITRD=initrd.img
DEB_ARGS="console=ttyS0,115200 root=$DEB_DEVICE rootfstype=$DEB_FSTYPE rootflags=$DEB_OPTION ro init=/preinit"

if [ -e "$DEB_KERNEL" ] && [ -x ./preinit ]; then
   # make /dev /proc /sys available in chroot
   mount --rbind /dev  dev
   mount --rbind /proc proc
   mount --rbind /sys  sys

   # load kernel
   if [ -e "$DEB_INITRD" ]; then
         chroot . kexec -l "$DEB_KERNEL" -a "$DEB_ARGS" -i "$DEB_INITRD"
   else
         chroot . kexec -l "$DEB_KERNEL" -a "$DEB_ARGS"
   fi

   # go!
   chroot . kexec -e
fi
 

As you can see, it is is relatively straightforward. It needs the DEB_DEVICE, DEB_FSTPE> and DEB_OPTION environment variables from the /etc/rc.local script and defines two more, DEB_KERNEL, the path from the root to the kernel image and DEB_INITRD, the path to the matching initrd.

In the default root filesystem, /vmlinuz is a symlink to /boot/vmlinuz-4.9.58 and there is no initrd image.

The script checks that the kernel really exists, and also that there is an executable called /preinit. This is started instead of the normal init/systemd binary to prepare the environment and work around some bugs in systemd.

If everything looks ok, it loads the kernel into memory (kexec -l) and then runs it (kexec -e).

Preinit, Systemd bugs and the Switch Configuration

The /preinit script is run before the actual init. The default script looks like this:

#!/bin/sh

mount -noremount,rw /

# systemd goes into endless loop on certain future dates
date -s "2017-01-01 00:00:00"

# (re-)create the default(openwrt)  switch config
# ports 0-3 (labelled 4 to 1) mapped to port 5 (eth0)
# port 5 (labelled wan) mapped to port 6 (eth1)
swconfig dev switch0 set reset 1
swconfig dev switch0 set enable_vlan 1
swconfig dev switch0 vlan 1 set vid 1
swconfig dev switch0 vlan 1 set ports '0 1 2 3 4 5'
swconfig dev switch0 port 0 set qmode 3
swconfig dev switch0 port 1 set qmode 3
swconfig dev switch0 port 2 set qmode 3
swconfig dev switch0 port 3 set qmode 3
swconfig dev switch0 port 4 set qmode 0
swconfig dev switch0 port 5 set qmode 3
swconfig dev switch0 port 6 set qmode 0
swconfig dev switch0 port 0 set pvid 1
swconfig dev switch0 port 1 set pvid 1
swconfig dev switch0 port 2 set pvid 1
swconfig dev switch0 port 3 set pvid 1
swconfig dev switch0 port 4 set pvid 0
swconfig dev switch0 port 5 set pvid 1
swconfig dev switch0 port 6 set pvid 0
swconfig dev switch0 set apply 1

exec /sbin/init

First it re-mounts / read-write. The it sets the system date to beginning to 2017 - this is because the RTC doesn't work and Systemd will end up in an infinite loop for some dates, as you would expect for quality software. I suspect that, while the chipset has an RTC, the router simply doesn't have a battery so it's essentially random at boot (and in a future release, I might disable it in the kernel).

The script ends by replacing itself with the real init.

Which leaves the big swconfig block in the middle. This configures the switch so that ports 1, 2, 3 and 4 are switched to the eth0 interface and the WAN port is connected to eth1.

Enjoy... or read on!

And this concludes using the provided files. The next article deals with re-creating those files, so you can make your own, find the sources and modify them, build a custom kernel based on my config and so on.

If you have some unanswered questions or simply want to tell me that you tried it out, just drop me a mail! Likewqise if you have some suggestions on what else to include in the kernel or root filesystem.

I vaguely plan to make a working hostapd and possibly dhcp config, so the thing works like a wifi router out of the box, but at the moment, this is simply about getting a standard debian up and running for you to customize.