Up to Main Index Up to Journal for January, 2026
JOURNAL FOR SUNDAY 18th January, 2026
______________________________________________________________________________
SUBJECT: …and back again
DATE: Sun 18 Jan 21:25:27 GMT 2026
Oops! I toasted my server :(
On new year’s eve, before publishing my yearly round-up, I performed my usual
maintenance and installed some updated software packages. It did not go well.
Before I knew it my server was toast with no hope for recovery. The road back
was a very long one indeed…
For decades I have run the next version of Debian, known as “testing”. I find
it very solid and much more up to date than the latest (Bookworm, now Trixie)
distribution.
Raspberry Pi OS is built on Debian and I use the 32-bit testing distribution
on my Raspberry Pi as well. As usual, due to the Bookworm to Trixie transition
the testing packages were frozen. Trixie 13.0, 13.1, 13.2 and then 13.3 were
released. The testing packages were still frozen. This meant I was effectively
running Trixie 13.0 still. Not good for an internet facing server. The testing
freeze would last until this weekend…
For Raspberry Pi OS there are usually two package repositories of note. The
Foundation specific packages and Debian packages. For the 32-bit OS the
Foundation recompiles the Debian packages so that they work across all models
of Raspberry Pi. For the 64-bit OS the Debian packages come directly from
Debian. This means 32-bit moves at the pace of the Foundation and 64-bit moves
at the pace of the Debian project.
Usually you install packages from the same distribution for the Foundation
specific packages and Debian packages. When using testing I have to use the
last stable version of the Foundation specific packages. This meant I was
mixing Bookworm Foundation packages and testing Debian packages. I have done
this for years now and it has not been an issue — I only use a minimal set of
the Foundation specific packages.
On my local Raspberry Pi I had upgraded the Foundation specific packages from
Bookworm to Trixie. There were a few issues I had to sort out. The structure
of the /boot partition had changed. The kernel packages had changed. All of
the overlays disappeared — you need to install the raspi-firmware package.
Then I made my mistake. I switched the server over to the Trixie Foundation
specific packages. I made my tweaks to fix things up. Rebooted. The server
went into a boot loop and never came back. It turns out that my provider does
not officially support Trixie yet on Raspberry Pi. The issue appears to be
related to the layout of the /boot partition changing and their NFS booting
does not look in the alternative locations for files yet. It did not matter
that, except for the kernel, I had been running Trixie via testing for ages.
I spent a long time with support, but in the end it wasn’t going to work.
I finally had my server back and an OS reinstalled by 5th January. By now I
was back at work and with little free time. I also wasn’t working from my
study as I still hadn’t had any heating since before Christmas…
Didn’t I have backups? Couldn’t I just restore the server? Yes… and no.
I did have backups. However, I decided I would rather rely on Debian for my
updates rather than the Foundation. This was prompted by the extended testing
freeze — which also happened with the previous Bullseye and Bookworm releases.
So, this time I switched from a 32-bit operating system and installed the
64-bit version. This meant a simple restore wasn’t possible. I had to
carefully rebuild the server from scratch again. That work is now completed
and everything is back again :)
64-bit software typically takes up 20-30% more disk space and uses 10-20% more
memory. I usually only run 64-bit Raspberry Pi OS on Raspberry Pi with 8Gb RAM
because of this. The current server is a 4GB Raspberry Pi 4B, but seems to be
doing okay. After this experience the same model Raspberry Pi that runs my
homelab will be getting re-installed as well.
I just want to finish by thanking everyone who reached out to me, after the
server went off-line, to make sure I was okay. It was much appreciated.
--
Diddymus
Up to Main Index Up to Journal for January, 2026