Create robots.txt rules from paths you want crawlers to avoid. Add a sitemap URL and review the scope before publishing the file.
How to use
Enter one path beginning with / per line. Add the full sitemap URL if needed. Save the result as robots.txt at your site's root.
Example
/drafts/ creates a rule for that path under User-agent: *. Inspect the output so an accidental / does not block the whole site.
FAQ
Does robots.txt protect private data or remove indexing?
No. Use authentication for privacy. A blocked URL can still appear in search, so crawling rules are not a substitute for noindex.
Related tools
A practical task
Check the effect of the broadest path first. A slash can exclude the whole site, while a prefix can cover many pages beyond one example URL. Keep rules focused and test the deployed file from the site's actual root.
Another useful question
Does a Disallow rule replace a login?
No. Cooperative crawlers can obey it, but ordinary visitors and other clients can still request the path. Enforce access restrictions in the application.
Settings and formats
| Field | Values |
|---|---|
| Sitemap URL | Starting value: https://example.com/sitemap.xml |
Details for interpreting the result
| Detail | Explanation |
|---|---|
| Crawling and indexing | Crawling is fetching a resource; indexing is including information about it in search. A crawl rule does not provide authentication. A blocked address can still be known through links even if its content is not fetched. |
| Robots rule scope | A path rule can match more than the single URL a person had in mind. Review trailing slashes, prefixes, and broad exclusions before deploying robots.txt. Private information needs access control rather than a crawler promise. |
| Sitemap purpose | A sitemap is a list of URLs offered for discovery. It does not repair inaccessible pages, remove duplicate content, or guarantee indexing. Keep the list focused on the versions you want search engines to consider. |
| Canonical signals | A canonical tag indicates a preferred URL among duplicate or similar pages. It is a signal rather than a command. Internal links, redirects, sitemaps, and actual content should agree about the preferred address. |
| URL components | A URL contains structure such as scheme, host, path, query, and fragment. Characters that separate those parts differ from characters inside a value. Encode the right component rather than transforming the full address blindly. |
| Query values | Query separators connect parameter names and values. An ampersand inside a value must not be confused with the separator between parameters. Build the URL with correct component encoding before sharing it. |