Robots.txt Security & Privacy Generator

Control Search Engine Indexing • 100% Local

robots.txt is not a security tool. It's a public, voluntary request that only well-behaved search engines respect — anyone can read your robots.txt file, and listing a path here does not hide, block, or protect it in any way. Use real access control (a password, authentication, or firewall rule) to actually protect sensitive pages.
Quick Add Common Paths

Toggle any of these to add them as Disallow rules for all crawlers (User-agent: *). You can also add custom paths below.

Custom Path
Sitemap (optional)
Must be a full absolute URL, not a relative path — a relative path here won't be honored by search engines.
Generated robots.txt

                Save this as robots.txt in your site's root directory (e.g. https://yoursite.com/robots.txt) — it won't work anywhere else.
            

                    
        

    


    
    

        

            
 About

            
 Contact

        

        

            
About Robots.txt Security & Privacy Generator

            
What robots.txt actually does: it's a plain text file at the root of your site that politely asks search engine crawlers not to index certain paths, so they don't show up in search results. That's genuinely useful — for example, keeping a staging environment or an internal admin dashboard out of Google's index. What it does not do: prevent anyone from visiting the page directly if they know or guess the URL, stop crawlers that choose to ignore it (robots.txt is entirely voluntary — well-known crawlers like Googlebot respect it, but nothing forces any bot to), or remove a page that's already been indexed (removing an indexed page requires either a noindex meta tag plus waiting for re-crawl, or a manual removal request in Search Console). Since robots.txt itself is served as a plain, publicly readable file at /robots.txt, listing a sensitive path in it is a bit like putting a sign on a locked door that says exactly what's behind it — it doesn't unlock the door, but it does tell everyone where to look.

            
Paths in Disallow are matched as prefixes, not exact paths or regular expressions — Disallow: /admin blocks /admin, /admin/, /admin/dashboard.html, but also /administrator-notes.html, since that path also happens to start with the same characters. If you only want to block a specific folder and everything inside it, use a trailing slash (/admin/) to avoid accidentally catching unrelated pages with similar names.

            
A Sitemap directive must point to a full, absolute URL — a relative path won't be recognized. Crawl-delay is supported by some search engines (like Bing) but is explicitly ignored by Google, so it's not included by default here.

            
For anything that actually needs to be kept private, use real access control — HTTP authentication, a login system, or a firewall/IP restriction — rather than relying on robots.txt.