AI crawlers and robots.txt: what to allow or block
The main AI crawlers as of November 2026, which ones train models and which ones power search answers, plus a robots.txt snippet you can copy.
Plain guides on how AI answer tools find, read and quote websites, and what you can change on yours.
This is our library of guides on being found by AI answer tools. Each one opens with a short answer, shows when it was last checked and lists its sources.
The main AI crawlers as of November 2026, which ones train models and which ones power search answers, plus a robots.txt snippet you can copy.
GEO and SEO share the same foundation: crawlable pages, an index and useful content. Here is what is new with AI answers and what you can keep doing.
Which search index and crawlers ChatGPT, Gemini, Perplexity, Claude and Copilot rely on, and what that means for getting your website cited.
A hands-on 15-minute checklist to see if ChatGPT, Gemini and other AI tools can read and quote your website, plus what an automated scan adds.
llms.txt is a proposed text file that gives AI tools a short guide to your website. What it is, how to write one, and why its impact is still unproven.
What schema.org structured data (JSON-LD) is, what it does and does not do for AI answers, with copy-ready examples for a local business, product and article.
GEO means making your website easy for AI answer engines to find, read and cite. A plain guide to what it is and how those engines pick sources.
Should your business spend time on GEO yet? An honest guide by business type: online shops, law firms, local services, SaaS companies and publishers.
Each guide starts with a direct answer in the first 2 sentences. We link to the source for every fact, and we show the date a guide was last checked. When AI tools change, we update the guide and the date.
Prefer a feed reader? Follow the RSS feed.