FREE SEO TOOL
Robots.txt generator.
Generate a robots.txt file with a user-agent group, crawl rules and optional sitemap references.
No sign-up or API key. Your text and files are processed in this browser and are not uploaded by this tool. Use Reset to clear your inputs.
Your result.
Browser-onlyAdd your inputs to get started.
Controls crawler access requests, not privacy or guaranteed indexing. Disallow does not reliably remove a URL from search results.
How to use it.
Use care with a full-site block. Robots.txt is public and should never be used to protect private information.
- Select a starting mode and crawler group.
- Add any path rules and sitemap URLs.
- Review the output before publishing it at the domain root.
A LITTLE BACKGROUND
Separate crawl rules from indexing choices.
Robots.txt tells cooperating crawlers which paths they may request. This generator prepares a single user-agent group and optional sitemap references. It helps you review a proposed crawl policy before uploading a plain-text file at the domain root. A root-level file is different from a robots meta tag on an individual page.
Put it to work.
A custom group could disallow an internal search route while allowing a public subpath. A full-site block uses Disallow: / and can prevent the selected crawler from accessing all paths. Read the result line by line before applying it to a live site.
What to check next.
A disallowed URL can sometimes remain known to a search engine, and a crawler that cannot fetch a page cannot read its noindex tag. Private documents need authentication or other access controls. Do not place sensitive paths in this public file and assume they are protected.
Questions about
this tool.
Where should robots.txt be published?
It normally belongs at the root of the relevant protocol and host, such as https://example.com/robots.txt. Confirm how your host or SEO plugin manages that file before replacing it.
What are the limits of this robots.txt generator?
Controls crawler access requests, not privacy or guaranteed indexing. Disallow does not reliably remove a URL from search results.
Can I use the output on a commercial project?
You can use the generated output in your own workflow, subject to the rights in the material you provide. Review the file or markup before publishing it. These tools do not verify ownership, factual claims or suitability for your particular project.
Need a closer look
at the website?
Discuss the wider issue with WTfont. We can help connect the technical details with the content, design and marketing brief.
Explore the service ↗