Web Hosting - robots.txt
What is a robots.txt? https://en.wikipedia.org/wiki/Robots_exclusion_standard
The Web Hosting service by default implements a robots.txt file with a crawl-delay function within the /httpdocs folder for each Shared Hosting domain.
Example:
User-agent: *
Crawl-delay: 10
This robots.txt is in place to help stagger the load on individual web sites who are indexed by Google, Archive.org and the like. It can be especially helpful when multiple bots maybe indexing a web site at the same time, which can make sites less responsive. Additionally it will help increase the overall capacity on the server(s) as most sites are indexed frequently and staggering this traffic helps to that end.
We recommend that you keep this file in place containing the crawl-delay but but feel free to edit it to suite your needs.
If you have further questions, email webhosting@doit.wisc.edu