Robot / Crawler

Robot / Crawler

Search engines will look in your root domain for a special file named "robots.txt" (http://www.mydomain.com/robots.txt). The file tells the robot (spider) which files it may spider (download). This system is called, The Robots Exclusion Standard.
Read here for more information

This one keeps our robot out.

User-agent: Eurosoft-Bot
Disallow: /


This one bans keeps our bot from getting at the directory file:

User-agent: Eurosoft-Bot
Disallow: /directory_1/
Disallow: /directory_2/