Author Topic: Help create a robots.txt  (Read 371 times)

wizzy

  • Newbie
  • *
  • Posts: 5
    • View Profile
Help create a robots.txt
« on: March 03, 2010, 09:08:52 PM »

SEO Tools
Robot Control Code Generation Tool
Free Robots Metatag Generator
http://www.mcanerin.com/en/search-engine/robots-txt.asp
If you are not sure how to build a robots.txt file this program will create a txt file for your site.

meta tag generator includes a simple section for the generation of a robots metatag for your use.

The robots metatag is a simple method of telling robots what you want and don't want indexed or followed on your site. Like the robots.txt file, it is NOT secure, nor is it a requirement. Rogue robots can ignore it if they wish to. Further, a human is under no restrictions at all.

The robots metatag allows you to restrict the spidering behavior of a single page.

Rule of thumb: If you want to restrict robots from entire websites and directories, use the robots.txt file. If you want to restrict robots from a single page, use the robots metatag. If you are looking to restrict the spidering of a single link, you would use the link "nofollow" attribute.