AI crawler

What types of AI crawlers are there?

AI crawlers fall into three broad types based on what they do: crawlers that collect material to train AI models, search crawlers that gather pages in advance for AI search answers, and user-requested visits that come to read a page the moment a customer asks a question.

Even within the same company, each name has a different role, so when you say "block AI crawlers" you need to choose exactly what you're blocking. If you block the training crawler but leave the search crawler open, you can keep showing up in AI search answers.

OpenAI and Perplexity say user-requested visits may not follow robots.txt, since the user directly asked for them. Anthropic, on the other hand, says Claude-User can also be controlled with robots.txt. Policies differ by company and can change, so it's a good idea to check the official docs now and then.

Does blocking Google-Extended remove you from Google Search?

No, Google says Google-Extended doesn't affect your visibility or rankings in Google Search. Google-Extended isn't a separate visiting robot; it's a token you put in robots.txt to say whether content Google has collected may be used for Gemini model training and grounding (attaching search content as the basis for an answer).

AI Overviews and AI Mode in Google Search follow the same conditions as regular search. A page has to be indexed in regular search and eligible to show with a snippet, so be especially careful: blocking Googlebot, the search crawler, can remove you from both regular search and the AI features.

In short, you can split it up: keep Googlebot open, and if you don't want your content used for Gemini training, block only Google-Extended. Google says your visibility in Google Search stays the same if you do this.

How do you set up AI crawlers in robots.txt?

You set them up by listing each crawler's name (User-agent) together with the paths it may or may not visit (Allow and Disallow) in robots.txt, the access-guide file at the top level of your website (yourbusiness.kr/robots.txt).

robots.txt is a request to be respected, not a lock. Official crawlers say they follow it, but programs pretending to be them and some user-requested visits may not. Information that truly must not be public belongs behind a login.

1. 01Open yourbusiness.kr/robots.txt in your browser to see what it currently says.
2. 02Check that there's no line blocking everything, like "Disallow: /" under "User-agent: *". Test settings from before launch are sometimes left in place.
3. 03Decide separately whether to allow or block the training crawlers (GPTBot, ClaudeBot, Google-Extended) and the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot).
4. 04Write the names of only the crawlers you want to block, with "Disallow: /", and either leave the rest out or open them with "Allow: /".
5. 05Block only the paths search doesn't need, like the admin panel or the payment-complete screen.
6. 06If you have a subdomain (e.g. shop.yourbusiness.kr), put a separate robots.txt on that address too.
7. 07After making changes, double-check with the robots.txt report in Google Search Console and the robots.txt validator in Naver Search Advisor.

Can you block training crawlers but allow search crawlers?

Yes, you can set them separately. OpenAI says blocking GPTBot signals that your site's content shouldn't be used to train its foundation models, and that sites blocking OAI-SearchBot won't be shown in ChatGPT search answers.

Anthropic likewise explains that blocking ClaudeBot excludes your future content from training data, and blocking Claude-SearchBot removes you from the index used for search answers. A business website exists to get the word out to customers, so keep the search crawlers open and decide on training crawlers based on your own judgment.

If the content itself is your product, such as columns, recipes or educational material you wrote, blocking training crawlers is a perfectly reasonable choice. On the other hand, if most of your information should be widely known, like hours, location and prices, you don't gain much by blocking them.

When do AI crawlers get blocked by mistake?

Crawlers can get blocked even if you never touched robots.txt. Some security features in hosting or CDNs (services that store copies of your website around the world) include a setting that blocks AI crawlers all at once, and when it's on, crawlers can't get in regardless of robots.txt.

Asking your web agency two questions usually settles it: "Which crawlers did you allow and block in robots.txt?" and "Is AI bot blocking turned on in the hosting or security settings?" If visits are so frequent they become a burden, some crawlers, like ClaudeBot, say they support a visit interval setting (Crawl-delay).

• A "block search engines" setting turned on before launch was never turned off after launch
• A security service's "AI bot blocking" feature is on by default
• Whole IP addresses were blocked because of heavy traffic: Anthropic says IP blocking may not work reliably and recommends robots.txt instead
• Content only appears after JavaScript runs, so some crawlers may read just a blank page