Over the years most of us who have "been around the block" in IT operations have always said, based on experience, that one of the biggest looming threats to IT operations is always a well intended employee who makes an honest mistake. I can recall discussing this topic almost daily with my honorable USAF colleagues, including Col. (R) Glenn Watt, Col. (R) David Gruber, and Maj. Gen. (R) Bill Donahue, all experts in IT operations and IT operational risk.
Even before those memorable days consulting for the USAF, I can recall one of my first UNIX projects with General Electric Information Services (GEIS), where I wrote programs that logged every keystroke by any developer making emergency changes in their operational environment. We always were concerned about external threats, but internal accidents were a much higher operational priority.
One final story, while at US Sprint working on the (then new) commercial Internet backbone back in 1993, I recall one day when one of our star employees, well intended, uploaded a new version of the CISCO IOS to all our commercial routers. There was a bug in the OS that caused every router to go down. No hacker could have done the vast damage that one well-intended, trusted employee can do.
So, it comes as no surprise that a Google employee, working on updating their malware notification service uploaded a simple little "/" as a malware site a few days ago (January 31, 2009), effectively declaring the entire Internet to be malware for nearly 55 minutes. This will probally end up being the single largest denial-of-serivce attack of 2009, self-inflicted; where for around 55 minutes, Google declared the entire Internet "/" as malware.
Being on-line much of the time, either at my desk or by mobile phone, I was on-line (Googleing of course) when this happened and posted Google Error: This site may harm your computer at the UNIX and Linux Forums. It was interesting to read many security professionals blogging (on other sites) how this might be some elaborate hacker attack. Folks tend to overreact and love the sensational stories, as we all know.
The truth is much more mundane than fantasy. Google lost a lot of money in ad revenue during those 55 minutes. In addition, Google suffered reputational losses. Google's self-inflicted denial-of-service attack is a stark reminder to all IT security professionals about what is the greatest threat and risk to operational security.
