# http://www.robotstxt.org/wc/norobots.html # { Disallow these User-agent: MSIECrawler Disallow: / User-agent: IRLbot Disallow: / # } # Limited access User-agent: MJ12bot Crawl-delay: 15 Disallow: # Allow rest User-agent: * Crawl-delay: 4 Disallow: