Where should I upload the robots.txt file?
The file must be uploaded to the root directory of your website. It should be accessible at https://yourdomain.com/robots.txt. If it is in a subdirectory (e.g., /blog/robots.txt), search engines will not find it.
Do I need a robots.txt file for a small website?
It is not strictly mandatory. If you don't have one, search engines will assume they can crawl everything. However, it is good practice to have one, even if it just links to your sitemap.
Can I use robots.txt to remove a page from Google?
Not exactly. `Disallow` prevents Google from visiting the page, but if it's linked from elsewhere, it can still appear in search results (often with a "description not available" message). To fully remove a page, allow crawling but add a <meta name="robots" content="noindex"> tag to the page header.
What is "Crawl-delay"?
This directive asks bots to wait a certain number of seconds between requests. Note that Googlebot ignores this directive (preferring crawl rate settings in Google Search Console), but Bingbot and Yandex still support it.
Does the file syntax matter?
Yes, it must be exact. It is case-sensitive, and wildcard support (`*`, `
Robots.txt Generator — Allow/Disallow Rules & Sitemaps | SimplyUtils
) is standard for major search engines but not part of the original protocol. Our generator handles these syntax details for you.