Where should robots.txt be placed?

It must be placed at the root of your domain, e.g. https://example.com/robots.txt.

Can I add multiple user-agent blocks?

Yes, add as many rule groups as you need, each targeting a different user-agent.

Does robots.txt guarantee pages won't be indexed?

No, it only discourages crawling. Use a noindex meta tag to prevent indexing.

Is my data uploaded to a server?

No. This tool runs entirely in your browser using standard web APIs, so your input never leaves your device and nothing is logged or stored on our servers.

Does it work offline and on mobile?

Yes. Once the page has loaded it keeps working without a network connection, and the layout is responsive so it stays usable on phones and tablets.

Robots.txt Generator

Create a robots.txt file with custom crawl rules.

Support

Works Offline

Privacy First

No Login

No API

Rules

Group 1

User-agent

Crawl-delay (seconds)

Allow (one path per line)

Disallow (one path per line)

Sitemap URLs (one per line)

robots.txt

YAML

User-agent: *

Sitemap: https://example.com/sitemap.xml

About the Robots.txt Generator

Build a robots.txt file by adding rules per user-agent, disallow/allow paths, crawl-delay, and sitemap references. Download or copy the finished file ready to drop at your site root.

Examples

Block all crawlers from /admin

YAMLUser-agent: * | Disallow: /admin

Output

YAMLUser-agent: *
Disallow: /admin

Keyboard shortcuts

Copy the main outputCtrl / ⌘ + Shift + C
Download the resultCtrl / ⌘ + S
Share this toolCtrl / ⌘ + Shift + S
Reset the inputsAlt + R
Open the tool search paletteCtrl / ⌘ + K

Related tools

Breadcrumb Schema Generator

SEO

Generate JSON-LD BreadcrumbList structured data.

Broken Link Checker

SEO

Scan a web page for broken, redirected and working links.

Canonical Tag Generator

SEO

Generate canonical link tags to avoid duplicate content issues.

FAQ Schema Generator

SEO

Build JSON-LD FAQPage structured data from Q&A pairs.

Keyword Density Checker

SEO

Analyze keyword frequency and density in your text.

Meta Tag Analyzer

SEO

Analyze HTML for missing or problematic meta tags.

Frequently asked questions

The technical SEO checklist developers actually own
Titles, canonicals, structured data, robots.txt and sitemaps — the parts of SEO that live in your codebase.
How to find and fix broken links before they hurt your site
A practical workflow for finding 404s, reviewing redirects and validating sitemap URLs without removing useful links by mistake.

Version 1.0.0 · Updated 2026-08-06 · Runs entirely in your browser