SEO
Robots.txt Generator
Build robots.txt rules with allow/disallow paths, optional crawl-delay, and sitemap lines.
What is Robots.txt Generator?
robots.txt tells compliant crawlers which paths they may fetch. It is not a security boundary—sensitive areas still need authentication. This generator creates a User-agent: * block, either allowing the site while disallowing listed paths or disallowing everything for staging. You can attach an optional Crawl-delay and a Sitemap URL. Output is plain text you can download as robots.txt. Rules are assembled in your browser from your form choices. Always verify the live file at https://yourdomain.com/robots.txt and test important bots in Search Console after changes.
Use cases
- You want to disallow /admin and /private while allowing the rest of a marketing site.
- A staging site should Disallow: / until launch day.
- You need a Sitemap: line pointing at https://example.com/sitemap.xml.
- You want a starting robots.txt before refining per-bot rules in Search Console.
How to use Robots.txt Generator
- Open the Robots.txt Generator on www.it3.it.com/tools/robots-txt-generator/.
- Choose Allow all (then list paths to block) or Disallow all.
- Optionally add a sitemap URL and crawl-delay, then Generate.
- Download robots.txt and upload it to your site root; confirm it loads over HTTPS.
Example
Input: Allow all + disallow /admin and /private + sitemap https://example.com/sitemap.xml
Output: User-agent: * Disallow: /admin Disallow: /private Sitemap: https://example.com/sitemap.xml
Exact text follows your radio selection and path list.
Why use it3’s Robots.txt Generator?
Free robots.txt drafting with no account. The file is built in your browser so you can download and deploy on your host. Pair it with the XML Sitemap Generator when you need both discovery files.
Tips
Common mistake: leaving Disallow: / on production after staging. That can stop Google from crawling the live site—remove it before launch.
Key features
- Allow or disallow-all modes
- Custom disallow paths
- Optional crawl-delay
- Sitemap URL line
- Download robots.txt
- No signup required
Frequently asked questions
Does robots.txt hide private pages from hackers?
No. It only guides crawlers. Protect private areas with login, permissions, and server rules.
Does Google honor Crawl-delay?
Google ignores non-standard crawl-delay. Some other bots may honor itu2014know which crawlers you care about.
Can I list multiple sitemaps?
Yes in robots.txt generally, though this UI adds one Sitemap lineu2014add more manually if needed.
Why is a page still indexed after Disallow?
Disallow blocks crawling, not always indexing if Google finds links elsewhere. Use noindex where appropriate.
Is my draft uploaded to it3?
No. The text is generated locally in your browser for download.
Explore more free tools
More free SEO tools on it3.
View all tools →