Where should robots.txt be placed?

It must be placed at the root of your domain, e.g. https://example.com/robots.txt.

Can I add multiple user-agent blocks?

Yes, add as many rule groups as you need, each targeting a different user-agent.

Does robots.txt guarantee pages won't be indexed?

No, it only discourages crawling. Use a noindex meta tag to prevent indexing.

Robots.txt Generator

Create a robots.txt file with custom crawl rules.

Works Offline

Privacy First

No Login

No API

Rules

Group 1

User-agent

Crawl-delay (seconds)

Allow (one path per line)

Disallow (one path per line)

Sitemap URLs (one per line)

robots.txt

User-agent: *

Sitemap: https://example.com/sitemap.xml

About the Robots.txt Generator

Build a robots.txt file by adding rules per user-agent, disallow/allow paths, crawl-delay, and sitemap references. Download or copy the finished file ready to drop at your site root.

Examples

Block all crawlers from /admin

User-agent: * | Disallow: /admin

Output

User-agent: *
Disallow: /admin

Related tools

Breadcrumb Schema Generator

SEO

Generate JSON-LD BreadcrumbList structured data.

Canonical Tag Generator

SEO

Generate canonical link tags to avoid duplicate content issues.

FAQ Schema Generator

SEO

Build JSON-LD FAQPage structured data from Q&A pairs.

Keyword Density Checker

SEO

Analyze keyword frequency and density in your text.

Meta Tag Analyzer

SEO

Analyze HTML for missing or problematic meta tags.

Open Graph Generator

SEO

Generate Open Graph and Twitter Card meta tags.

Frequently asked questions

The technical SEO checklist developers actually own
Titles, canonicals, structured data, robots.txt and sitemaps — the parts of SEO that live in your codebase.

Version 1.0.0 · Updated 2026-08-06 · Runs entirely in your browser