PageYak

Robots.txt generator

Choose what crawlers may visit, block AI training bots if you like, and test a page before you upload the file. Everything runs in your browser.

All crawlers

One path per line. Use * for any text and $ for the end, as in /*.pdf$.

AI crawlers to block

One per line, written in full.

Bing and Yandex wait this long between requests. Google ignores it.

robots.txt


        
Upload it to the top folder of your site, so it opens at https://your-site.com/robots.txt.

      

      
    
      
    
        
    Test a page
    
        
    
          
    
          
    
        
    
        
    
      
    
    
    
  
    

    

    
  
    

    

    
  
    
    

    
  
    What robots.txt does
    
  
    robots.txt is a small text file at the top of a website. It tells crawlers, such as Google's and those that collect text for AI, which pages they may visit. Well-behaved crawlers follow it. It is not a lock: a blocked page can still be opened by anyone with the link.
    
  
    
    
      
    User-agentWhich crawler the lines below it are for. * means every crawler.
    
      
    DisallowA path the crawler should not visit. Disallow: / keeps it out of the whole site.
    
      
    AllowAn exception inside a blocked folder. The longest matching line wins.
    
      
    SitemapWhere to find the list of your pages.
    
        
  
    

    

    
  
    Blocking AI crawlers
    
  
    AI companies name their crawlers so sites can choose. Training crawlers collect pages to teach AI models. Search crawlers let AI answers link to your pages, so blocking them can cost you visitors. Crawlers that fetch a page because a person asked an AI about it act more like a browser.
    
  
    Blocking training crawlers does not remove your pages from Google or Bing search.
    

    

    
  
    Privacy: how this tool handles your settings
    
  
    When you use Robots.txt generator, your browser reads your settings on your device and the tool's code does the work in this tab. PageYak's server only sends the page and its code. It never receives your settings, so there is nothing to delete afterwards.
    
  
    How PageYak works explains this in more detail.
    

    


    
  
    When to edit robots.txt
    
  
    robots.txt asks crawlers to stay out; it does not hide or protect pages. Use a noindex tag or a password for anything private.
    
  
      
    
    • Keeping crawlers out of admin areas, search result pages and test folders.
      • 
    
    • Telling search engines where your sitemap is.
      • 
    
    • Deciding whether AI crawlers such as GPTBot may read your site.
      • 
  
    

    

    
  
    Questions
    
  
    Will Disallow remove a page from Google?
    Not always. Google can still list a blocked page if other sites link to it, without a description. To keep a page out of search, let Google crawl it and add a noindex robots meta tag instead.
    
  
    Where does the file go?
    In the top folder of your domain, named exactly robots.txt. Each subdomain needs its own file.
    
  
    How fast do changes work?
    Crawlers read the file again from time to time. Google usually picks up changes within a day.
    
  
    Does every website need a robots.txt?
    No, but it is good practice to have one that at least lists your sitemap.
    
  
    How do I test my robots.txt?
    Open yoursite.com/robots.txt in a browser, and check the robots.txt report in Google Search Console.
    

    

    
    
  
  

    

  
    
  
    Last updated 12 November 2026.