Robots.txt File Generator
Default robot access
|Action||Robot||Files or directories|
Your Robots.txt File
Now, copy and paste this text into a blank text file called "robots.txt" (don't forget the "s" on the end of "robots") and put it in your root directory. Like all other files on your server, make sure its permissions are set so that visitors (such as search engines) can read it.
Introduction to Robots.txt
Block all web crawlers from all content
User-agent: * Disallow: /
Block a specific web crawler from a specific folder
User-agent: Googlebot Disallow: /no-google/
Block a specific web crawler from a specific web page
User-agent: Googlebot Disallow: /no-google/blocked-page.html
Allow a specific web crawler to visit a specific web page
Disallow: /no-bots/block-all-bots-except-rogerbot-page.html User-agent: rogerbot Allow: /no-bots/block-all-bots-except-rogerbot-page.html
User-agent: * Disallow: Sitemap: http://www.example.com/none-standard-location/sitemap.xml
Disallow Googlebot from indexing of a folder, except for allowing the indexing of one file in that folder
User-agent: Googlebot Disallow: /folder1/ Allow: /folder1/myfile.html
Crawl Delay -Search engines allow you to set crawl priorities.
Microsoft's information for Bing is located
User-agent: bingbot Crawl-delay: 10
where the 10 is in seconds.