About this tool

An llms.txt is a simple markdown file you place at the root of your site. It gives AI crawlers a structured summary of your key pages, with a title, a short description of your site, and your content grouped into sections. Think of it as a curated table of contents written for AI, alongside the sitemap you already provide for search engines.

This tool does the heavy lifting for you. Enter your domain and it crawls your public pages, groups them into sections based on your site structure, and drafts readable titles from your URLs. Add your site name and a one-line summary, and you have a valid llms.txt ready to review. Everything is editable, so you can tidy titles, trim pages, and shape it into a genuinely useful, curated file before you publish.

Honest note: llms.txt is still an emerging standard and adoption across AI tools is uneven. We recommend it as a low-effort, sensible addition to your AI visibility setup, not a guaranteed ranking lever.

Use cases

• Give AI crawlers clean context. A structured summary of your key pages, in a format built for AI rather than search engine indexing.

• Start from your real site, not a blank file. The crawl builds the structure for you, so you curate rather than write from scratch.

• Curate what matters. Exclude low-value URLs, keep the pages you want AI tools to understand, and edit the titles to read well.

• Pair it with your other AI-crawler work. Use it alongside your robots.txt and sitemap as part of a wider AI visibility setup.

How to use

1. Enter your website. Just the domain, no https:// needed.

2. Add your site name and summary (optional). These become the title and summary line at the top of your file. You can edit them afterwards.

3. Set the crawl scope (optional). Choose a maximum number of pages and exclude any URL patterns you do not want included, such as tags or filters.

4. Generate. The tool crawls your site and builds the file, grouped into sections.

5. Review and edit. Tidy any titles derived from your URLs, remove pages you do not need, and refine the summary.

6. Copy or download and upload the file to your site root so it is available at yourdomain.com/llms.txt.

FAQs

What is llms.txt?

It is a plain markdown file at the root of your site that gives AI crawlers a clean, curated overview of your content: a title, a short summary, and your key pages grouped into sections with links. It complements your robots.txt and sitemap rather than replacing them.

How does the tool build it?

It crawls your public pages server-side, groups the URLs into sections based on your site structure (main pages, blog, case studies and so on), and drafts a title for each page from its URL. You then edit the result into a curated file.

Why should I edit the result?

llms.txt works best as a hand-curated file. Titles are derived from your URLs, so some may read awkwardly, and not every crawled page belongs in your final file. A few minutes of tidying makes it far more useful.

Does llms.txt actually help with AI visibility?

It is a sensible, low-effort step, but it is an emerging standard and support across AI tools varies. Treat it as one part of a broader AI visibility strategy rather than a guaranteed result on its own.

Why did some of my pages not appear?

The crawler follows internal links and skips assets and any patterns you exclude. Pages that are not linked from your site, or that only render through JavaScript, may not be found. It also focuses on a capped number of pages, since llms.txt is best kept as a focused list of your most important content.

Where does the file go?

Upload it to the root of your domain so it is available at yourdomain.com/llms.txt.

Is it free?

Yes, free with no signup. If you would like help being found and cited across AI search, from content and structure to authority, contact [email protected] or book a call with Growthack.