Let me know if you start having problems with this site. I have been upgrading the security. About 2 or 3 PM each day I see the site slammed by remote sites sucking gigabytes of content out of my site. I’m not sure why they would want it. Nor do I mind that they take it. I do mind that they overload the web server I use and make the site way slow. So, I come in each day and disconnect those that are behaving like bots… continuous dozens-of-requests per second.
I’ve been changing robots.txt and adding ‘slow down’ instructions for search engines. That helps with the big well mannered engines. But, not the problem bots that have been hitting.
I’ve been blocking the networks the most abusive bots use. But, doing that without restricting legitimate users is complicated. So, it isn’t a good solution. Neither is blocking individual IP addresses. The bad guys just change IP.
So, now I am trying some new software. It is in learning mode. So, I’m ‘completely’ sure what all it will decide to do.