Make your docs discoverable#

Goal: let search engines crawl the docs — a sitemap.xml listing every page, a robots.txt that points at it, and a clear line on what mdjango does not own.

You need: a served Content tree. This is the human-crawler counterpart of Publish the LLM artifacts; the routes are in the URLs reference.

The key fact shapes everything below: mdjango has no domain. It mounts under a prefix you choose and emits root-relative URLs, so it stays portable across mounts and static exports. A sitemap needs absolute URLs, and robots.txt/.well-known are only honoured at your site root — which your project owns, not the docs app. So mdjango ships the one part it alone can (the page list) and leaves the root to you.

Add a sitemap#

mdjango ships a Sitemap class; you mount the standard Django sitemap view at your site root. It needs no extra package — django.contrib.sitemaps is part of Django, and it does not require django.contrib.sites (it reads the domain from the request).

# settings.py
INSTALLED_APPS += ["django.contrib.sitemaps"]

# urls.py
from django.contrib.sitemaps.views import sitemap
from django.urls import include, path

from mdjango.sitemaps import DocsSitemap

sitemaps = {"docs": DocsSitemap}

urlpatterns = [
    path("sitemap.xml", sitemap, {"sitemaps": sitemaps}),  # at the root, not under docs/
    path("docs/", include("mdjango.urls")),
]

GET /sitemap.xml now lists every HTML page — the Index page and every Page in navigation order — as absolute URLs, with the scheme and domain taken from the request. The machine artifacts (llms.txt, llms-full.txt, search-index.json, the .md alternates) are deliberately excluded: they are for agents, not a search index. Drafts are excluded too.

Behind a TLS-terminating proxy, set SECURE_PROXY_SSL_HEADER so Django builds https:// URLs. This is ordinary Django deployment config, not an mdjango setting.

Date your pages#

A page can declare when it last changed, which becomes its sitemap <lastmod>:

---
title: Roll back a deploy
updated: 2024-03-15
---

The key is optional and per-page: pages without it simply carry no <lastmod>. Use an ISO YYYY-MM-DD date; a malformed value is ignored with a build warning. mdjango does not fall back to the file's modification time — a git clone or CI checkout stamps every file with the checkout time, which would make <lastmod> identical and wrong across the whole site.

Add a sitemap to a static export#

A static export has no request to read a domain from, so pass one:

python manage.py mdjango_build dist --base-url https://docs.example.com

This writes dist/sitemap.xml — at the export root, so it is served from your domain root — with absolute <loc>s and <lastmod> from any updated dates. Without --base-url no sitemap is written. The domain is a build-time argument, never a setting: mdjango stays domain-agnostic everywhere else.

Add a robots.txt#

robots.txt is only read at your domain root, so mdjango can't serve it from under the docs mount — you add it. A permissive file that points crawlers at the sitemap:

User-agent: *
Allow: /

Sitemap: https://docs.example.com/sitemap.xml

Serve it however you serve other root files — a static file, or a small view:

# urls.py
from django.http import HttpResponse
from django.urls import path
from django.views import View


class RobotsTxt(View):
    def get(self, request):
        sitemap_url = request.build_absolute_uri("/sitemap.xml")
        body = f"User-agent: *\nAllow: /\n\nSitemap: {sitemap_url}\n"
        return HttpResponse(body, content_type="text/plain")


urlpatterns += [path("robots.txt", RobotsTxt.as_view())]

Leave the LLM artifacts and search index crawlable. Duplicate-content dilution is not a real concern for docs, and the llms.txt convention wants those files reachable.

What mdjango leaves to you#

/.well-known/ (ACME challenges, security.txt, and the like) is a site-root concern with nothing documentation-shaped in it — mdjango neither serves nor documents it beyond this line. The llms.txt convention puts llms.txt at the site root, not under .well-known, so there is nothing there to point at the docs.