My photo

Mildred's Website

Tags:
My avatar

GoogleTalk, Jabber, XMPP address:
mildred@jabber.fr


GPG Public Key
(Fingerprint 197C A7E6 645B 4299 6D37 684B 6F9D A8D6 9A7D 2E2B)

wwwgen: imagined architecture

Thu 13 Sep 2012, 03:30 PM by Mildred Ki'Lya comp dev en idea wwwgen

I started writing wwwgen: a website generator that uses redo for its dependency tracking. Unfortunately, I might not have taken the correct approach to the problem. I reuse the webgen concepts, and that might be a bad idea.

Specifically, webgen (and my first version of wwwgen) are bottom-up systems (you take the sources and build everything they can generate). The problem is that redo itself is top-down (you take the target and build it, building sources as they are needed), and I tried to make the two match. It's very difficult, and impossible to do with a clean design.

What I'd like to have is a simple wwwgen binary that generates a HTML file from a source file. Let's imagine how it could work:

So, we have a problem here ...

Now, we have another problem. Specifically, my source file is called title.page and I want my output file to be title/index.html. In webgen, this is implemented by a configuration in title.page telling it to build in title/index.html.

There is a solution to solve both problems at once. the wwwgen command creates an archive (the formats needs to be defined, it could be tar, or different yaml documents in the same file for example). Then, the build process would be:

all.do:

 find src -name "*.src" \
   | sed 's/src$/gen/' \
   | xargs -d '\n' redo-ifchange
 find src -name "*.gen" \
   | xargs -d '\n' wwwgen unpack

default.gen.do:

 redo-ifchange "$2.src"
 wwwgen --redo-dependencies -o "$3" generate "$2.src"

wwwgen generate would parse the source file and generate an archive, that will be unpacked later by wwwgen unpack. Let's see how it can work:

How an we improve the design to be able to create source files dynamically.

There are different views to the problem:

How can we generate source files on the fly:

I prefer the second solution which is more flexible, but we need a way to avoid building things twice. For example, it's not necessary to build a page if on the next phase the page source is going to be regenerated.

Very simply, the generated page can contain a link to the index source file that generated it, and when generating the page, redo-ifchange is run on the index file.

Next question: what if a tag is deleted. The corresponding index page is going to stay around until the next clean. The tag page should keep around a list of index files it generated and delete them when a tag is no longer detected. And deleting the index should not be done using rm because the index will need to delete the pages it generated. The best solution would be to integrate to redo to detect these files.

The build scripts now are:

all.do:

 oldsrclist=
 srclist="$(find src -name "*.src")"
 while [ "$oldsrclist" != "$srclist" ]; do
   echo "$srclist" \
     | sed 's/src$/gen/' \
     | xargs -d '\n' redo-ifchange
   oldsrclist="$srclist"
   srclist="$(find src -name "*.src")"
 done

 find src -name "*.gen" \
   | xargs -d '\n' wwwgen unpack

default.gen.do:

 redo-ifchange "$2.src"
 wwwgen --redo-dependencies -o "$3" generate "$2.src"