Up to Main Index                               Up to Journal for May, 2026

                      JOURNAL FOR SUNDAY 31ST MAY, 2026
______________________________________________________________________________

SUBJECT: Transitioning from Apache to Watchman
   DATE: Sun 31 May 21:41:16 BST 2026

If you are reading this then it means I have successfully transitioned from
using Apache to my own stack I’m calling “Watchman”.

After testing everything thoroughly for weeks, it still all went tits up :(

Currently my server is stuck on Debian Bookworm instead of Trixie. This caused
a few issues with socat and stunnel. So I did what any good engineer would do.
I went and grabbed the sources and compiled socat and stunnel myself. You do
compile stuff yourself when the need arises, or is it just me? This is how the
new setup looks:


                             CHROOT Jail + localhost
                            .--------------------------------------,
   internet -+-> stunnel -->| Go reverse -+-> busybox httpd (www)  |
             |              |   proxy     |-> busybox httpd (www)  |
             `-> socat      |             |-> busybox httpd (code) |
                 (redirect) |             `-> busybox httpd (code) |
                            `--------------------------------------'


Traffic comes in from the internet. If it’s HTTP / port 80 a simple socat
redirect sends the visitor over to HTTPS / port 443. Traffic coming in on port
443 goes via stunnel which handles all of the heavy TLS cryptography. The
traffic is then handed off to the Go reverse proxy as plain HTTP traffic.

The custom built Go reverse proxy will then divert traffic to one of four
busybox httpd servers. This takes into account multiple domains and round
robin between the httpd servers.

Currently there are two domains: www.wolfmud.org and code.wolfmud.org. Each
domain has two httpd servers for performance and failover resilience.

The www.wolfmud.org domain is the main domain for this website.

The code.wolfmud.org domain is where I host git repositories. Projects like
WolfMUD can be cloned from this domain using git.

The response is then sent from the httpd servers back to the router. Depending
on the domain, security and cache headers are injected before the response is
sent back to stunnel.

Finally stunnel sends the response back out onto the internet.

The router also collects some metrics, but no logging. I respect people’s
privacy and want to avoid the GDPR. I can’t hand over data I don’t have.

Currently the raw statistics look like this:


    # WolfMUD Infrastructure Metrics
    server_start_time_seconds 1780254320
    server_current_time_seconds 1780257213
    requests_total 52
    bytes_received_total 4919
    bytes_transmitted_total 2582582

    # HTTP Responses by Status Class
    http_responses_1xx 0
    http_responses_2xx 45
    http_responses_3xx 6
    http_responses_4xx 1
    http_responses_5xx 0


Keeping a close eye on everything is my orchestration infrastructure. A Bash
script called “Watchman”. It runs via cron when the server boots and every 15
minutes — just in case. stunnel, socat, router and httpd are very stable.

Now, if I need to go into maintenance mode I can use “watchman start-maint”.
This swaps out my whole stack and replaces it with a single socat process that
responds with a “503 Service Unavailable” and the message “Site is currently
down for maintenance. Please come back later”.

I can then perform my maintenance and a quick “watchman stop-maint” swaps the
whole of my stack back into service again.

Of course, renewing SSL certificates should not be maintenance. A quick
“watchman reload” starts using the new certificates without dropping a single
connection :)

Why have I built “Watchman”? I now own the complete stack end-to-end. I’ve
compiled it myself from sources. I know how it works end-to-end. It’s very
simple and easy to understand. It’s just stunnel, socat, router, busybox,
watchman script and four configuration files. It’s very secure. The router and
busybox run in bare distro-less read-only chroot jails and only make use of
loopback networking.

My stack in total is probably smaller than the configuration files for your
kubernetes cluster…

So far I have this stack running on my desktop i9, Raspberry Pi 4 (4Gb) and a
Raspberry Pi Zero W. Full details as a how-to coming soon…

--
Diddymus


  Up to Main Index                               Up to Journal for May, 2026