Discovery

Being findable is a module

A publication nobody can find is a diary. Discovery — feeds, sitemaps, canonical URLs, the instructions given to crawlers — is not an afterthought bolted on at the end; it is a module with real requirements and at least one genuinely surprising failure in its history.

The sitemap, written by hand

The standard approach is a plugin that enumerates routes at build time and writes a sitemap from them.

That approach produced, in this system, a sitemap with no articles in it at all. Every page here is resolved at request time rather than baked out ahead of time, so a build-time enumerator could only see the handful of fixed routes. The entire point of the site — the writing — was invisible to it. And nothing reported an error; the sitemap was generated, it was valid, it was just empty of anything that mattered.

So the sitemap is generated from the content instead of from the routes. It enumerates every published article, book, chapter, deck, series, category and topic, each with a genuine last-modified date derived from the content itself. A category page’s date is the date of its newest article, because that is when the page actually changed.

The hosting platform generates its own sitemap too, with the same blind spot, plus it lists the error page. Ours is the one to submit.

Canonical URLs

Every page declares its own canonical URL — the address it considers definitive.

This is the tag that quietly decides whether a move goes well or badly. Point it at a dead host and you have told every search engine that your real pages are duplicates of something that no longer exists. That is not a hypothetical: the chapter on moving the address is about exactly this.

Every absolute URL in the system — canonical tags, social card image URLs, the sitemap, every share link — derives from one origin value. There is no second place where the site’s own address is written down.

The feed

A full feed of the writing, in the standard format, linked from every page.

Feeds are unfashionable and I would not remove this one. It is the only distribution channel in the entire system where nothing sits between what I publish and someone who asked to receive it — no ranking, no algorithm, no account.

The crawler instructions

This is where the platform pushes back. The hosting layer intercepts the standard crawler-instructions file at its edge and serves its own, pointing at its own sitemap.

There is a route in the site that serves the correct file, and where the platform lets it through, it is served. Where the platform insists on its own, two things cover the gap: the same complete sitemap is also served at the address the platform advertises, so a crawler following the platform’s instructions still arrives somewhere useful; and every page carries a link element pointing at the real sitemap, which is independent of the file entirely.

Belt and braces, because the alternative is a publication that is thoroughly crawlable in theory.

Verification

There is a script that checks a live origin from the outside: that key routes answer, that canonical and social tags name the right host, that the sitemap contains what it should, that no draft is reachable.

It runs after every deploy. Discovery is the one part of a publication where a mistake is silent, slow and expensive — nothing breaks, traffic simply never arrives — so it is the part that gets checked from the outside rather than assumed from the inside.