robots.txt

robots.txt Generator and Checker

Choose how search engines and other crawlers may visit your site and create a robots.txt file. Upload it to the top folder of your site so it opens at https://your-domain/robots.txt. The tool also checks for common mistakes such as blocking the whole site.

Basic setting
Rules per crawler

Paths start with /. * matches any characters and $ marks the end of a URL (for example /*.pdf$).

AI crawlers

Blocking AI crawlers keeps your content out of future AI training, but it may also make your site less likely to be introduced or cited in AI search and AI answers. Blocking the search crawlers (lower in the list) has a bigger effect.

Sitemap

robots.txt

Check

    Notes on robots.txt

    robots.txt is a request to crawlers, not access control. Pages listed in Disallow can still appear in search results if other sites link to them. To keep a page out of search results, use noindex; to keep it private, use a password.

    Do not list secret URLs in robots.txt: anyone can read the file.

    • Rules for the same user-agent are combined into one group. When several rules match a URL, Google uses the most specific (longest) one.
    • Changes are picked up the next time crawlers read robots.txt (usually within a day). Everything is created in your browser and nothing is sent.

    About this tool

    Create a robots.txt file that tells search engines and other crawlers which parts of your site they may visit. Choose to allow or block all crawlers, add Allow and Disallow rules per user agent, exclude the WordPress admin, block the main AI crawlers such as GPTBot, ClaudeBot and Google-Extended, and add Sitemap lines. The tool also checks for common mistakes such as blocking the whole site or writing paths incorrectly.

    How to use

    1. Under Basic setting, choose whether to allow or block all crawlers.
    2. If needed, add user agents and paths under Rules per crawler. Exclude the WordPress admin adds a common rule in one click.
    3. To keep your content out of AI training, tick Block the main AI crawlers and choose which crawlers to block.
    4. Enter your sitemap URL to add a Sitemap line.
    5. Review the Check results, then copy or download the file and upload it so it opens at https://your-domain/robots.txt.

    Useful for

    • Preparing a robots.txt file when you launch a new website
    • Keeping crawlers away from a site that is not public yet or a test site
    • Making it less likely that your content is used to train AI
    • Excluding pages crawlers do not need, such as admin pages

    FAQ

    Where should robots.txt go?

    In the top folder of your site, so that it opens at https://your-domain/robots.txt. A robots.txt file in a subfolder is ignored.

    Does Disallow keep a page out of search results?

    Not always. robots.txt is a request not to crawl, and a page linked from other sites can still appear in results. Use noindex for pages that must not appear.

    What are the downsides of blocking AI crawlers?

    Your content is less likely to be used for AI training, but your site may also be introduced or cited less often in AI search and answers. The tool selects the training crawlers by default and blocks search crawlers only if you choose them.

    Does blocking Google-Extended affect Google Search?

    According to Google, Google-Extended controls use in Gemini training and answers and does not affect Google Search.

    Can I check my existing robots.txt?

    The check covers the file you build with the form. There is no option to paste an existing file, so recreate its rules in the form to check them.

    Want more visitors from search and AI?

    We review crawl settings, sitemaps, structured data and content so both search engines and AI tools understand your business.

    Ask about SEO and AI search
    Contact