Title: Journal for Monday 8th April, 2019
Abstract: Another lazy Sunday afternoon in front of the keyboard
    Tags: plain text, site generator, footnotes

                      JOURNAL FOR MONDAY 8TH APRIL, 2019
______________________________________________________________________________

SUBJECT: Another lazy Sunday afternoon in front of the keyboard
   DATE: Tue  9 Apr 03:35:29 BST 2019

It seems I never got around to pushing out the new text.Unfold function to the
public dev branch :(

I was also working on using text.Unfold to fix the config and zones packages.
The config package needs to unfold the server greeting and the zones package
needs to unfold the free text sections. These fixes are due to the changes
made to the recordjar package which no longer unfolds text automatically when
reading files. I had everything ready to go, but never finished committing the
fixes and pushing everything out…

Things have now been remedied, commits have been made and changes pushed out
to the public dev branch :)

It seems that rewriting the static site generator, used for this site, has
been distracting me more than I thought. That and being made redundant[1].

All of this site is written using plain text. Really, just plain text. No
markup, markdown or other silliness. Just good old ASCII and a little Unicode.
See for yourself! The source text file for this page is here: file://8.txt

It’s important to me, and probably only me, that the source text for the site
be able to stand on it’s own. I should be able to give the text file to anyone
and they should be able to just read it. They should be able to see the
footnotes, links and email addresses. No surprises, no funny formatting, no
weird ‘## heading’, ‘*bold*’ or ‘_italics_’ and definitely no markup tags such
as those used for XML or HTML. This also implies that anyone should be able to
write content for the site, with no or minimal instruction. So far I seem to
have achieved this somewhat arbitrary, ambitious and lofty goal I’ve set for
myself.

Progress on the site generator has been going very well and I’ll soon be back
to coding on WolfMUD proper “real soon” now.

The generator takes the text files, builds the index pages, automatically
identifies links, generates the HTML pages and the compressed versions.

The generator is at the stage where it can generate the current site exactly.
I’m now adding a few bells and whistles. One specific feature I’ve had on my
wish list for a long time, but the old code was too crufty to add it. What is
this feature? Automatic identification and linking of footnote references like
this one → [2]. Actually that footnote won’t be linked yet if I’m using the
old site generator still :P

As everything is plain text identifying footnote references should be easy,
right? Just look for the opening character ‘[’, followed by one or more
digits, and a closing ‘]’ character. What about the following code snippet?


  for x := range data {
    data.values[0] = data.values[1]
  }


We don’t want either of the indexes to be turned into footnotes.

So I spent a quite Sunday afternoon finally implementing such a feature. There
are a few rules and restrictions, but most of them are sensible ones you’d
follow anyway when writing. If there is a problem with the footnotes it’s not
the end of the world, the text is left ‘as is’ with the footnotes and their
references being just plain text.

For footnote references:

  - Footnote references in the body of the text may have the form [n] or ⟦n⟧
  - Footnote references of the form [n] are not identified within indented
    body text unless at the start of the line, preceded by only white-space
  - Footnote references of the form ⟦n⟧ will be identified in indented text
  - Footnote references in the body of the text must appear sequentially
  - Footnote references must be unique, you cannot use the reference [1]
    in the text body multiple times for the same footnote
  - Footnote references must have a matching footnote

For footnotes:

  - Footnotes at the end of text must have the form [n] only
  - Footnotes at the end of text may have white-space only before them
  - Footnote numbering must start at 1, incrementing by 1 for each new
    footnote
  - Footnote numbering must be sequential
  - Footnotes must have a matching footnote reference

As I said, most of these rules are common practise anyway: write footnotes in
order, use them in order, no reference without a footnote, no dangling unused
footnotes. The only ‘special’ requirement is the use of ‘⟦’ and ‘⟧’ for
footnote references in indented text. Does this mess up my arbitrary,
ambitious and lofty goals? Maybe, just a little. I deliberately used the
Unicode mathematical left/right white square brackets so that the text still
looked normal to a reader. It’s usage is so infrequent that it shouldn’t cause
an issue. I’m already using Unicode which could potentially cause issues. For
example if you view one of the text files using ‘LESSCHARSET=ascii less’ you
might see something like:


  Progress on the site generator has been going very well and I’ll soon be
  back to coding on WolfMUD proper <E2><80><9C>real soon<E2><80><9D> now.


Where <E2><80><9C> and <E2><80><9D> are the UTF-8 bytes for “ and ”. You could
use the following command to convert a UTF-8 encoded Unicode file and view it
as ASCII:


  iconv -futf8 -tascii//TRANSLIT 8.txt | less


This would convert Unicode characters unavailable in ASCII to similar looking
equivalents or to a question mark if that is not possible. For example curly
quotes would become straight quotes. Interestingly ‘⟦’ and ‘⟧’ become ‘?’ :(

Which brings up the question of what do we actually mean by ‘plain text’? Is
it only these ASCII characters plus space, line feed and carriage return:

        ABCDEFGHIJKLMNOPQRSTUVWXYZ    0123456789
        abcdefghijklmnopqrstuvwxyz    !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~

In my personal opinion, writing using Unicode is quite acceptable. In fact,
due to the vast number of languages Unicode covers, I would say Unicode is
more inclusive than US ASCII. The biggest problem with Unicode is finding a
good looking font that also has good coverage of all of the glyphs available.

Now I’ve gone off on a tangent, where was I?

The generator will complain when it cannot assemble the footnotes for one
reason or another. For example:


  "journal/2016/6/8.txt" broken footnote for "[4]"
  "journal/2019/4/7.txt" confused, 2nd footnote found for "[1]"


When this happens all of the footnotes in the text file are left untouched as
plain text. I’m now having to go through the site fixing broken footnotes.
I’ve found references with no actual footnote, accidentally reused references
and places where I’ve juggled text around and messed up the footnote ordering.

I’ve also made a usability change. From any page or index there will be a link
directly back to the main index. This means users will no longer have to drill
upwards when deep in the site ;)

Once I’ve finished tidying the site I hope to deploy an updated version using
the new generator. After that it’s back to WolfMUD coding again.

For now it's getting early and time I tried to get some sleep… (-_-) ZZZzzz…

--
Diddymus

  [1] Gory details, still job hunting: file://../3/22.html

  [2] Sample footnote :)