Robots.txt Generator

Default - All Robots are:  
Sitemap: (leave blank if you don't have) 
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo MM
  Yahoo Blogs
  DMOZ Checker
  MSN PicSearch
Restricted Directories: The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

FREE Robots.txt Generator

An instruction file called Robots.txt describes how a website should be crawled. Websites use this standard to tell search engines which parts of their website need indexing. It is also known as robots exclusion protocol. There is also an option to specify which areas of your site should not be crawled; such areas contain duplicate content or are under construction. There is a substantial probability that bots such as malware detectors and email harvesters won't follow this standard and will scan your site for weaknesses in security, and they will begin examining it from places you don't want indexed.

It includes the directive User-agent, below which other directives like "Allow," "Disallow," and "Crawl-Delay" can be written. If written manually, it might take a lot of time, and you can enter multiple lines of commands in one file. The Disallow and Allow attributes must be used in the same manner if you want the bots to ignore a page.

One wrong line in your robots.txt file can exclude your page from indexation. So, don't assume it is easy. Therefore, leave the file creation to our Robots.txt generators and have no worries about it.

Frequently Asked Questions (FAQs)

How do I create a custom robots.txt file?

You can create your own .txt file in any text editor and then upload it to the root folder of your site using FTP. Once you’ve uploaded it, wait a few hours and the changes will be reflected on your site. To make sure everything is working, you can use a free robot text checking tool, such as SEnuke Checker. Once you’ve taken care of the robots.txt file on your site, you can start working on your article marketing campaign.

Is robots.txt mandatory?

No. It is not mandatory, but it’s highly advisable. The more your customers interact with you, the more you can learn about them and their interests. Then you can tailor your marketing messages to your customers based on what they like, so they will be more likely to buy from you.

Is robots.txt safe?

Yes. Robots.txt is very safe. The only information that is sent to the robots.txt server is your phone number and the message you are sending. All of this information is sent in encrypted form so that no one can intercept it. The information is stored on the robots.txt server and is deleted after the message is delivered to the recipient. Do not worry, no one is monitoring your messages, there are no hidden "features" like that. You can message anyone, anytime, even if they have not registered for robots.txt. In fact, anyone can use robots.txt to text you -- but only if you have registered for the service.