robots.txt

What does a robots.txt file look like?

robots.txt is a plain text file with one rule per line: "for which robot (User-agent)," "where not to go (Disallow)" and "where it may go (Allow)." In September 2022, the Internet Engineering Task Force (IETF) published it as a standard, RFC 9309.

The file must be named robots.txt, sit in the top-level folder of the site and be a UTF-8 text file. Its rules apply only to the domain, protocol and port where the file lives, so a subdomain like "shop.yourbusiness.kr" needs its own file.

Field names (User-agent, Disallow) aren't case-sensitive, but paths are. "/Admin" and "/admin" count as different paths.

• User-agent: * → applies to all robots.
• Disallow: /admin/ → please don't read anything under /admin/.
• Allow: / → everything else is fine to read.
• Sitemap: https://yourbusiness.kr/sitemap.xml → here's where the sitemap is.

What's the difference between robots.txt and noindex?

robots.txt says "don't come read this," while noindex says "you can read it, but don't put it in search results." Google clearly states that robots.txt is not a way to keep a page out of Google Search.

Even a page blocked by robots.txt can appear in search results as a bare URL with no description if it's linked from elsewhere. To reliably keep it out of search, use noindex, and don't block that page in robots.txt so robots can read the noindex.

In practice, split it like this: noindex for confirmation pages customers don't need to see, robots.txt for search and filter URLs where robots could wander endlessly, and a login for anything with personal information.

How do I create and check a robots.txt file?

Most website builders and frameworks create robots.txt for you automatically. Open it once before launch and check it in the order below.

Naver Search Advisor has tools to quickly generate and download a robots.txt, request a crawl of an updated file right away and test whether a specific page is blocked. Google reads a new file on its own, with no need to submit it separately.

If you want Google to know about your robots.txt changes quickly, you can request a recrawl from the robots.txt report in Search Console. Otherwise, Google may keep using the old cached file for a while.

1. 01Open "yourbusiness.kr/robots.txt" in an incognito window and check that the file shows as text.
2. 02Check whether there's just a single "Disallow: /" line under "User-agent: *". If so, your entire site is blocked.
3. 03Make sure only paths robots don't need to read, like the admin panel, are listed under Disallow.
4. 04Check that the Sitemap line at the bottom has your full sitemap URL.
5. 05Check for errors in the robots.txt report under Settings in Google Search Console.
6. 06In the Verification menu of Naver Search Advisor's webmaster tools, request a crawl of robots.txt and test that your key pages show as crawlable.

What happens if robots.txt is missing or returns an error?

If there's no file, search robots assume they can read every page. Both Naver and Google treat a robots.txt that returns a "not found" response like 404 as having no restrictions.

A server error (5xx), on the other hand, is risky. Naver treats it as "disallow everything," and Google stops crawling the site for the first 12 hours, then uses the last rules it read for up to 30 days. If your server is unstable, search robots may not be able to read your website at all.

There are a few other rules worth knowing. Google usually caches robots.txt for up to 24 hours, so changes may not take effect right away, and it ignores anything beyond 500KiB. Naver says that if robots.txt comes back as an HTML document, it may ignore the rules inside and treat the file as missing.

How do I handle AI crawlers with robots.txt?

You can allow or block AI crawlers by name (User-agent) too. For example, OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search separately, and Google lets you decide separately, under the name Google-Extended, whether your content is used for Gemini training and as grounding for its answers.

AI Overviews and AI Mode in Google Search follow the settings for Googlebot, the regular search robot. Google states that Google-Extended doesn't affect inclusion or ranking in Search. Conversely, blocking Googlebot can remove you from both regular search and AI features.

For a small business website, it usually pays to have accurate information come up when customers ask AI. Think of training and search crawlers separately, and start by considering keeping at least the AI search crawlers open. Keep in mind, though, that robots.txt is a request, so there's no guarantee every crawler will follow it.

What are common robots.txt mistakes?

Listing paths you want to hide in robots.txt effectively tells them to anyone who opens the file. The international standard itself says robots.txt rules are not a form of access control and can't replace security measures.

Blocking style files (CSS) or script (JS) paths is another common mistake. Search robots need to read these files to understand the page properly, so Naver also asks that these resources follow the same rules as the page or be allowed.

Right before launch, check robots.txt and noindex as a pair. Lifting the build-time block is a one-line change for a developer, but if it's missed, your site can vanish from search for weeks.

• Leaving the "Disallow: /" added during the build in place after launch
• Also blocking noindex pages in robots.txt so the noindex can't be read
• Blocking CSS, JS or favicon paths
• Putting robots.txt in a subfolder instead of the site's top level
• Writing paths with different capitalization than the actual URLs