Where should robots.txt be placed?

It must be placed at the root of your domain, e.g. https://example.com/robots.txt.

Can I add multiple user-agent blocks?

Yes, add as many rule groups as you need, each targeting a different user-agent.

Does robots.txt guarantee pages won't be indexed?

No, it only discourages crawling. Use a noindex meta tag to prevent indexing.

Is my data uploaded to a server?

No. This tool runs entirely in your browser using standard web APIs, so your input never leaves your device and nothing is logged or stored on our servers.

Does it work offline and on mobile?

Yes. Once the page has loaded it keeps working without a network connection, and the layout is responsive so it stays usable on phones and tablets.

Robots.txt Generator

Create a robots.txt file with custom crawl rules.

Support

Works Offline

Privacy First

No Login

No API

Rules

Group 1

User-agent

Crawl-delay (seconds)

Allow (one path per line)

Disallow (one path per line)

Sitemap URLs (one per line)

robots.txt

User-agent: *

Sitemap: https://example.com/sitemap.xml

About the Robots.txt Generator

Build a robots.txt file by adding rules per user-agent, disallow/allow paths, crawl-delay, and sitemap references. Download or copy the finished file ready to drop at your site root.

Examples

Block all crawlers from /admin

User-agent: * | Disallow: /admin

Output

User-agent: *
Disallow: /admin

Keyboard shortcuts

Copy the main outputCtrl / ⌘ + Shift + C
Download the resultCtrl / ⌘ + S
Share this toolCtrl / ⌘ + Shift + S
Reset the inputsAlt + R
Open the tool search paletteCtrl / ⌘ + K

Related tools

Breadcrumb Schema Generator

SEO

Generate JSON-LD BreadcrumbList structured data.

Canonical Tag Generator

SEO

Generate canonical link tags to avoid duplicate content issues.

FAQ Schema Generator

SEO

Build JSON-LD FAQPage structured data from Q&A pairs.

Keyword Density Checker

SEO

Analyze keyword frequency and density in your text.

Meta Tag Analyzer

SEO

Analyze HTML for missing or problematic meta tags.

Open Graph Generator

SEO

Generate Open Graph and Twitter Card meta tags.

Frequently asked questions

The technical SEO checklist developers actually own
Titles, canonicals, structured data, robots.txt and sitemaps — the parts of SEO that live in your codebase.

Version 1.0.0 · Updated 2026-08-06 · Runs entirely in your browser