100% FREE • RFC 9309 • CMS PRESETS • AI SCRAPER BLOCKING

Robots.txt Generator

Create a valid robots.txt file to opt your content out of AI training crawlers, control how Google, Bing and other search bots crawl your site, and make better use of your crawl budget.

🛡️ Block AI training (8 crawlers):
|
Generated robots.txt:
RFC 9309 Compliant

RFC 9309 and protecting your content from AI scrapers

In September 2022 the IETF published RFC 9309 (Robots Exclusion Protocol), formally standardizing robots.txt. The robots.txt file is the first resource that Google, Bing, Yandex and AI data-collection bots request before crawling a domain. Keep in mind that robots.txt is advisory: well-behaved crawlers follow it, but it does not technically block access.

🤖 Protect content from AI

Disallowing GPTBot, ClaudeBot and CCBot tells these crawlers not to collect your articles, source code and data for training language models.

⏱️ Save crawl budget

Disallowing heavy technical folders (/api/, /_next/, /wp-admin/) focuses crawlers on your valuable landing pages.

⚡ Local export

Generation and download happen 100% client-side via a Blob. Drop the file straight into your site root (public/robots.txt).

🛠️

Related Developer & Office Tools

All tools →
💾 SQL & DB

SQL Formatter & Validator

Beautify and format SQL queries, indent clauses, minify to 1 line for PostgreSQL, MySQL, and ClickHouse client-side.

Development Open →
👁️ WCAG 2.2 & UX

Color Contrast Checker (WCAG 2.2)

Calculate color contrast ratios for text and backgrounds per WCAG 2.2 AA/AAA accessibility standards with live previews.

Design & UX Open →
🔐 Web Crypto

Secure Password Generator

Cryptographically secure passwords powered by Web Crypto API running 100% locally on your machine.

Security & IT Open →
🔑 Safe & Local

JWT Token Decoder & Inspector

Inspect and decode JSON Web Tokens locally: inspect header, payload claims, and token expiration dates safely.

Security & IT Open →
🧪 Regex & IT

Regular Expression (Regex) Tester

Test and debug regex patterns with live match highlights, capture groups, regex flags, and cheatsheets.

Development Open →
🎨 Design & CSS

CSS & Tailwind Gradient Generator

Visual gradient designer with angle controls, trendy color presets, and 1-click CSS and Tailwind class export.

Design & UI Open →

Was this tool useful?

The site is free and has no paywalls. A small tip in TON or a share helps keep it running and add new tools.

Share in Telegram