llms.txt and crawler policy for an agent data site
If AI engines are going to cite your data, they need a machine-readable path to it. This is the exact set of files AgentBurn publishes, and the evidence for what each one does.
Published · 9 min read · by AgentBurn Editorial · dataset snapshot 2026-10-27
1. What llms.txt is, and what it is not
llms.txt is a proposed convention, not a standard. The spec — first published in September 2024 and now in a
v2 revision — asks sites to put a markdown file at /llms.txt that gives a model a short, curated
map of the site instead of an HTML page full of navigation. The format is deliberately small: an H1 with the
site name (the only required section), a blockquote summary, optional detail, then H2 sections whose bodies
are markdown link lists of the form [name](url): note, plus an ## Optional section
for material a model can skip when context is tight. In the spec's words, “The file itself stays small enough
to fit in context. The detail lives behind the links.”
The boundary matters more than the format. The document calls itself “a proposal to standardise”, and nothing obliges a crawler or an assistant to fetch it. Compare robots.txt, which has a published RFC (RFC 9309) and near-universal crawler support. llms.txt has neither: it is a request for good behaviour addressed to systems that may never read it. It is a discovery aid, not a control surface. The spec also expects the file to serve inference rather than training, which is the split our crawler policy is built around.
2. What AgentBurn publishes
Six machine-readable files, each with one job. None is generated at request time, so what a crawler gets is what a reviewer approved.
| Path | What it contains | What it cannot do |
|---|---|---|
| /llms.txt | H1, blockquote summary, then Key pages, Data, API, Citation, Contact, Optional | Guarantee that any model reads, follows or cites it |
| /llms-full.txt | Long-form context: definitions, the four provenance labels, dataset summary, a worked citation sentence | Replace the methodology; it is a summary |
| /ai.txt | Twelve lines: User-Agent: *, Allow: /, the content signal, links to both context files, contact, canonical origin | Enforce anything — same advisory status as robots.txt |
| /robots.txt | Crawl rules, content signal, per-crawler allows, sitemap location | Act as access control; the admin API requires a bearer token |
| /sitemap.xml | Canonical URLs with lastmod dates | Say which pages matter and which are secondary |
| /blog/feed.xml | RSS 2.0, one dated entry per post | Carry the dataset; the JSON files do that |
Two details in llms.txt are worth copying. First, the dataset description states its own limits inside the machine file: 30 production agents, 22 priced models, 3 public benchmarks, “20 of 30 projects have no public per-task metrics and show none”, and “daily run volume is a model estimate everywhere, never measured data.” An assistant summarizing our coverage summarizes the gaps too, which is the point. Second, the Citation section gives a literal template — snapshot 2026-10-27, access date — and one instruction that matters more: keep each field's provenance label attached, and do not cite a “not published” field, because it has no number.
We also have one markdown page twin, the homepage at /index.md, served as
text/markdown by a headers rule. The v2 spec asks for a .md version of every page an
agent might need, plus rel="alternate" type="text/markdown" and rel="describedby"
link relations. That is a bigger change than adding a text file, and we have not finished it: the blog posts
have no markdown twins yet.
3. The content-signal choice: retrieval yes, training no
Our robots.txt starts from open access — Allow: / —
excludes the noisy paths (/archive/, /qa/, /admin,
/cdn-cgi/), and then states one line that carries the editorial position:
Content-Signal: search=yes, ai-input=yes, ai-train=no. That split comes from the Content Signals proposal, a convention layered on robots.txt rather than an RFC. Read
plainly: indexing and retrieval are fine, using our pages to answer a user's question is fine, and training a
model on them is not. Below the signal, ten AI crawlers are named explicitly — GPTBot, OAI-SearchBot,
ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and
Bytespider — each with Allow: /.
Allow: / and nothing else — do
not inherit the * disallow list. That is a known inconsistency in the file, not a trick. Private
paths are excluded to keep noise out of indexes; the real access control is a bearer token on the admin API.
4. How to verify a crawler can read your files
Verification has two halves: the fetch, which you can test, and the interpretation, which you cannot.
# 1. status and content type
curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' https://getagentburn.com/llms.txt
# 2. is it the markdown map?
curl -sS https://getagentburn.com/llms.txt | head -n 12
# 3. does a named crawler get a different response?
curl -sS -A 'GPTBot/1.2 (+https://openai.com/gptbot)' https://getagentburn.com/llms.txt | head -n 5
# 4. is the signal and sitemap declared?
curl -sS https://getagentburn.com/robots.txt | grep -Ei 'content-signal|sitemap'
# 5. can it reach the dataset itself?
curl -sS -A 'ClaudeBot/1.0' -o /dev/null -w '%{http_code}\n' https://getagentburn.com/data/projects.jsonWhat these prove: the route returns 200, the body is parseable markdown, a named user agent is not being challenged by a WAF rule, and the JSON behind the map is reachable. What they do not prove: that any crawler fetched it, parsed it, or that an assistant used a figure from it. Those questions live in server logs, and logs record a self-reported user agent — trivially spoofed, frequently absent. A hit count is an upper bound on reads, not evidence of citation. The spec suggests a cheaper test: give a model only your llms.txt and ask it a question your site should answer. If the answer is wrong, the map is missing a page or a definition.
5. A copyable checklist for a tool site
| Step | Do this | Common failure |
|---|---|---|
| 1 | Write /llms.txt as a curated map: H1, one-paragraph summary, 5–12 key pages with one-line notes, data and API sections, a citation line | Dumping every URL; it becomes a worse sitemap |
| 2 | Put definitions and provenance rules in a longer companion file, linked from llms.txt | Leaving a model to infer what your numbers exclude |
| 3 | State your gaps in the machine file, not only in prose pages | Summaries that read as full coverage when half the fields are empty |
| 4 | Declare a content signal in robots.txt and name the retrieval crawlers you want | Assuming the signal is enforced, or that consent is implied |
| 5 | Keep /sitemap.xml and a dated feed in sync with canonical URLs | Stale lastmod, duplicate canonical paths |
| 6 | Serve markdown twins and link them with rel="alternate" type="text/markdown" and rel="describedby" | Publishing a .md file that nothing can discover |
| 7 | Re-run the curl checks after every deploy | A bot-challenge rule quietly blocking the crawlers you invited |
Steps 1–3 change what a model can say about you. Steps 4–7 keep it true after the next deploy.
6. What we cannot claim
We publish no AI-citation rate, no visibility score and no “llms.txt lifts citations by X%” figure, because we have no controlled measurement that would support one. Bot hits in edge logs measure requests, not citations; a user agent can be forged or missing; and we do not run assistant answer panels to check whether a number was quoted correctly. Any such percentage attributed to this site is not ours.
The honest claim is narrow and testable: these files exist, they are reachable, they are parseable, they say what the dataset contains and what it does not, and they ask for retrieval without training. Whether a given model reads them is outside our control — which is more or less the lesson of publishing for AI crawlers in 2026. The part you control is the file; the part you do not is the reader.
The dataset those files describe is at /projects, with the provenance rules in the methodology. If you are building this for your own data, the companion piece on what 20 of 30 projects do not publish is the argument for stating gaps in public rather than smoothing them over.
Sources
- The /llms.txt file, v2 — llmstxt.org (format, intent, and the inference-versus-training framing).
- Robots Exclusion Protocol — RFC 9309 (group matching and crawler obligations).
- Content Signals — contentsignals.org (the
search/ai-input/ai-trainproposal). - AgentBurn machine-readable files — /llms.txt, /llms-full.txt, /ai.txt, /robots.txt, /sitemap.xml, /blog/feed.xml.
- Dataset snapshot 2026-10-27 — projects.json; coverage and provenance labels as stated in /docs/methodology.