llms.txt generator and AI crawler rules builder
This free tool creates two files: an llms.txt that lists your key pages for AI systems, and robots.txt rules that allow or block the main AI crawlers. Fill in your pages, choose which crawlers to allow, and copy the results.
1. llms.txt
Save it as llms.txt and upload it to the root of your site, so it opens at yoursite.com/llms.txt.
2. AI crawler rules for robots.txt
Tick the crawlers you want to allow. Unticked crawlers are blocked.
Add these lines to your existing robots.txt. Do not replace the file. Google-Extended does not control Google Search or AI Overviews; those follow Googlebot.
How to use it
Two files, both optional, both quick.
List your key pages
Pick the ten to twenty pages that best explain your business: home, services, pricing, locations and your strongest guides.
Decide on crawlers
Allow the crawlers that power AI search if you want to appear in AI answers. Blocking a crawler keeps your content out and also removes you from what it powers.
Upload and check
Upload llms.txt to the root of your site and add the rules to robots.txt. Open both addresses in a browser to confirm they load.
Common questions
Short, direct answers.
What is llms.txt?
llms.txt is a proposed plain-text file that gives AI systems a short, curated list of a site's most useful pages. It sits at the root of a website, like robots.txt.
Do AI assistants actually use llms.txt?
The major assistants have not confirmed that they rely on it. It is cheap to add and harmless, so treat it as a small extra and not as a strategy.
Should I block AI crawlers?
Only if you do not want to appear in AI answers or be used in training. Blocking the crawlers behind AI search removes you from those results.
Does robots.txt force crawlers to obey?
No. It is a request that well-behaved crawlers follow. The major AI companies state that their crawlers respect it.
What is the difference between GPTBot and OAI-SearchBot?
OpenAI describes GPTBot as the crawler for training data and OAI-SearchBot as the crawler for ChatGPT's search results. You can allow one and block the other.
Want this done for you?
The services behind this tool.
Check your AI visibility, free
Find out whether ChatGPT and Gemini name your business when customers ask. Run the live check in about a minute with your email, or use the free self-test (no sign-up) for Copilot, Perplexity and Google too.
More free tools: review link and QR code · schema generator · llms.txt generator · ROAS and CPC calculator
Request your free growth audit
Tell us about your business. We review your search presence, AI visibility and ads, then send you what we find.
Prefer to talk? WhatsApp +91 95992 84951 or email info@frislerdigital.com.