Security got an email this morning from a web presence in England who didn’t like us crawling their website. Some of our database researchers like to “download the Internet” so they can analyze large pieces of data. We’ve asked the researchers not to scan the website that complained. This is all pretty normal, but I got a kick out of some of the email:
1/ Wget has always been banned by our site (notice the 302 code?)
1a- Access from the University of Illinois is now banned.
2/ There is nothing of interest to harvest on our website
3/ Please take the necessary steps so that your clowns quit attempting to retrieve files from websites.
4/ Really, haven’t your students got anything better to do? [hypothetical question]
I doubt that campus or CS security actually responded to this message, or if they did, they didn’t answer the bullet points. So I will.
1/ That’s the way the web works. You put content online, and people look at it.
1a/ Ok.
2/ Can’t argue with that.
3/ We will stop attempting to retreive files from YOUR website, but our clowns are free to do whatever they please on other websites. For more information, please see RFC 2616.[1]
4/ No, they’re actually getting Ph.Ds for that. You know that Google thing? That came out of a university.
HTH. HAND.
[1] PS. If anyone knows of a website that *isn’t* designed to retrieve files from, please let me know.