Add llms.txt and llms-full.txt to documentation builds #42

Merged

coding-agent-marvin8 merged 0 commits from refs/pull/42/head into main

2026-06-14 08:08:27 +00:00

coding-agent-marvin8 commented

2026-06-14 07:47:58 +00:00

(Migrated from codeberg.org)

Adds a MkDocs hook that generates llms.txt and llms-full.txt in the site output on every mkdocs build / mike deploy.

llms.txt — brief index following the llmstxt.org spec (title, description, link per page)
llms-full.txt — all pages concatenated as markdown with source URLs

No new dependencies. The hook uses on_pre_build to reset state, on_page_markdown to collect pages, and on_post_build to write both files to site/.

Closes #41

Adds a MkDocs hook that generates `llms.txt` and `llms-full.txt` in the site output on every `mkdocs build` / `mike deploy`. - `llms.txt` — brief index following the llmstxt.org spec (title, description, link per page) - `llms-full.txt` — all pages concatenated as markdown with source URLs No new dependencies. The hook uses `on_pre_build` to reset state, `on_page_markdown` to collect pages, and `on_post_build` to write both files to `site/`. Closes #41

coding-agent-marvin8 commented

2026-06-14 07:59:40 +00:00

(Migrated from codeberg.org)

packages/fenliu/hooks/llms_txt.py line 34 — @marvin8

In llms-full.txt, pages without a title will produce an empty heading (e.g., '# \n\n'). Consider filtering out such pages or providing a fallback title.

✅ Fixed in commit 9ae1492 — added if title guard to the sections list comprehension, matching the filter already applied to llms.txt on the line above.

[`packages/fenliu/hooks/llms_txt.py` line 34](https://codeberg.org/marvinsmastodontools/dujiangyan/pulls/42#issuecomment-17431499) — @marvin8 > In llms-full.txt, pages without a title will produce an empty heading (e.g., '# \n\n'). Consider filtering out such pages or providing a fallback title. ✅ Fixed in commit `9ae1492` — added `if title` guard to the `sections` list comprehension, matching the filter already applied to `llms.txt` on the line above.

coding-agent-marvin8 commented

2026-06-14 08:00:23 +00:00

(Migrated from codeberg.org)

packages/fenliu/hooks/llms_txt.py line 18 — @marvin8

A10: SSRF - The page.canonical_url is collected and later written to llms.txt/llms-full.txt files without validation. If an attacker can control the canonical URL (e.g., via page metadata injection in markdown), they could inject malicious URLs that might be used by automated tools (e.g., LLM scrapers) to perform SSRF attacks.

🔴 Not actioning. canonical_url is constructed by MkDocs itself from site_url in mkdocs.yml concatenated with the page path — it is not derived from page content or metadata and cannot be controlled by a document author. Writing a URL into a static text file is also not SSRF; SSRF requires the server to make an outbound HTTP request based on attacker-supplied input, which does not happen here.

[`packages/fenliu/hooks/llms_txt.py` line 18](https://codeberg.org/marvinsmastodontools/dujiangyan/pulls/42#issuecomment-17431502) — @marvin8 > A10: SSRF - The `page.canonical_url` is collected and later written to llms.txt/llms-full.txt files without validation. If an attacker can control the canonical URL (e.g., via page metadata injection in markdown), they could inject malicious URLs that might be used by automated tools (e.g., LLM scrapers) to perform SSRF attacks. 🔴 Not actioning. `canonical_url` is constructed by MkDocs itself from `site_url` in `mkdocs.yml` concatenated with the page path — it is not derived from page content or metadata and cannot be controlled by a document author. Writing a URL into a static text file is also not SSRF; SSRF requires the server to make an outbound HTTP request based on attacker-supplied input, which does not happen here.

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

marvin8/dujiangyan!42

No description provided.

Rows
Columns