Robot / Crawler
Robot / CrawlerSearch engines will look in your root domain for a special file named "robots.txt" (http://www.mydomain.com/robots.txt). The file tells the robot (spider) which files it may spider (download). This system is called, The Robots Exclusion Standard.Read here for more information This one keeps our robot out. User-agent: Eurosoft-Bot |
| This one bans keeps our bot from getting at the directory file: User-agent: Eurosoft-Bot |