Where should robots.txt be placed?

It must be placed at the root of your domain, e.g. https://example.com/robots.txt.

Can I add multiple user-agent blocks?

Yes, add as many rule groups as you need, each targeting a different user-agent.

Does robots.txt guarantee pages won't be indexed?

No, it only discourages crawling. Use a noindex meta tag to prevent indexing.

Is my data uploaded to a server?

No. This tool runs entirely in your browser using standard web APIs, so your input never leaves your device and nothing is logged or stored on our servers.

Does it work offline and on mobile?

Yes. Once the page has loaded it keeps working without a network connection, and the layout is responsive so it stays usable on phones and tablets.

Robots.txt Generator

Create a robots.txt file with custom crawl rules.

Support

Works Offline

Privacy First

No Login

No API

Rules

Group 1

User-agent

Crawl-delay (seconds)

Allow (one path per line)

Disallow (one path per line)

Sitemap URLs (one per line)

robots.txt

YAML

User-agent: *

Sitemap: https://example.com/sitemap.xml

About the Robots.txt Generator

Build a robots.txt file by adding rules per user-agent, disallow/allow paths, crawl-delay, and sitemap references. Download or copy the finished file ready to drop at your site root.

Popular uses

robots.txt
robots generator
crawler rules
disallow
sitemap

Examples

Block all crawlers from /admin

YAMLUser-agent: * | Disallow: /admin

Output

YAMLUser-agent: *
Disallow: /admin

Keyboard shortcuts

Copy the main outputCtrl / ⌘ + Shift + C
Download the resultCtrl / ⌘ + S
Share this toolCtrl / ⌘ + Shift + S
Reset the inputsAlt + R
Open the tool search paletteCtrl / ⌘ + K

Related tools

Broken Link Checker

SEO

Scan a web page for broken, redirected and working links.

Meta Tag Analyzer

SEO

Analyze HTML for missing or problematic meta tags.

Sitemap Validator

SEO

Validate sitemap XML and check whether its URLs are reachable.

Web Performance Analyzer

SEO

Measure TTFB, page size, compression, caching and resource counts for any URL.

XML Sitemap Generator

SEO

Generate an XML sitemap from a list of URLs.

Bulk URL Status Checker

Network & IP

Check HTTP status codes and redirects for many URLs at once.

Works well with

Next steps from other categories.

URL Redirect Chain Analyzer