# robots.txt # Stupid DOSing bots # Gaisbot/3.0 (robot05@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) User-agent: Gaisbot Disallow: / # Accoona-AI-Agent/1.1.1 (crawler at accoona dot com) User-agent: Accoona-AI-Agent Disallow: / # ichiro/2.0 (ichiro@nttr.co.jp) User-agent: ichiro Disallow: / # Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com) User-agent: ZyBorg Disallow: / ### With Crawl-delay # Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) User-agent: Slurp Crawl-delay: 20 # msnbot/1.0 (+http://search.msn.com/msnbot.htm) User-agent: msnbot Crawl-delay: 20 # OmniExplorer_Bot/4.75 (+http://www.omni-explorer.com) WorldIndexer User-agent: OmniExplorer_Bot Crawl-delay: 30 ### Default User-agent: * Disallow: /trac/pub/timeline Disallow: /trac/pub/log Disallow: /trac/z3lab/timeline Disallow: /trac/z3lab/log Disallow: /trac/pub/browser/funkload/trunk/funkload/demo/cmf/passwords.txt Disallow: /pub/funkload/trunk/funkload/demo/cmf/passwords.txt