I’m currently reviewing server logs for times which Homeland Stupidity has been unavailable to determine if denial of service attacks have been launched against this site. About an hour ago, the load average of the system shot up to over 40 (under normal service, it’s less than 0.1). This means, in non-technical terms, that someone or something hit the site and hit it very hard. Because I happened to be watching at the time, I was able to stop the service briefly and thereby stop the attack.
The surprising part is who caused this evening’s brief outage. It wasn’t terrorists; it was a major U.S. defense contractor.
At 5:47 p.m. Central time I received a single web browser request from two IP addresses, 192.31.106.34 and 192.35.35.35, which traced back to proxy servers run by Lockheed Martin Corporation. A few seconds later, and for the next few minutes, the same two IP addresses began making dozens of requests for different pages on the site, causing the Web server and database to go insane looking up very infrequently accessed information. This drove up the system load average to unusually high levels and brought the system to a crawl.
I shut down the Web server at 5:52 p.m. and brought it back up at 5:54 p.m. The misbehaving proxy servers had given up at that point and gone off to bring down someone else’s Web site.
After a short investigation I determined that the proxy server Lockheed Martin is using, known as Blue Coat, performs prefetching of Web sites. Prefetching means that the corporate proxy server requests a copy of a Web page in anticipation of the user clicking on a link, whether the user wants it or not. By prefetching Web pages one of its internal users might click on, the proxy can serve the page to its user more quickly. However, by prefetching pages too quickly, the proxy server can generate denial of service conditions at some Web sites, and this appears to be what happened here.
From reviewing available Blue Coat documentation and Google searches, there doesn’t seem to be any way to ask the proxy server not to do this. If you’re aware of one, please let me know. In the meantime, I’ll continue tuning the Web site to avoid this situation in the future. I’ve made a couple of more tweaks to Apache which should reduce or eliminate this specific denial of service effect and also give faster performance to everyone else. And I’ll continue watching closely for other bad behavior from large U.S. defense contractors.
Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works license.
Video published by this site is licensed under a Creative Commons Sampling+ license.
Bad Behavior has blocked 3475 access attempts in the last 7 days.
Facebook
Digg
del.icio.us
reddit.com
Newsvine
ba
Feb 15, 2006
isn’t stuff like d.o.s. attacks illegal?
considerd hacker like activities?
Michael Hampton
Feb 16, 2006
From reviewing the logs, I’m of the opinion that the server should have been able to handle the load of the number of requests received without shooting up to a load average of 40. So I’m getting down and dirty with apache (and might even replace it with lighttpd or something) right now, and I’m starting to see some improvements.
lemon
Feb 16, 2006
hmm interesting…Defence Contractor VS Homeland Stupidity LOL
Anonymous
Feb 18, 2006
I use to work for the government and Lockheed was one of our contractors. They have a multitude of services and do many other things besides making aircraft for the government. One happens to be “spook” activities. Not only do they have contracts with DoD, their main constituent, but also the DIA, the Justice Dept., NSA, Homeland Security, among others. They also field the largest contract lobbiest group in Washington. This company is so ingrained in politics, neither Washington or Lockheed can live without one another. I can almost assure you they were directed to attack your site by people within the Justice Department, as they have a number of contracts with Lockheed out for survellience activities under information technologies and C4 systems. You probably fielded an article that irritated them enough to give you problems on line. Even though you apparently did nothing illegal for them to come and arrest you, the ping was to rattle you, and to see how well your capabilities were to followup. It should not surprise anyone that all mail coming to your site is being monitored as well and IP addresses collected to do profile studies and further surrvelliance, and I will bet anything that they have attached spy ware to your system way before this latest incident ever happened. So, yes it was Lockheed that pinged you, but neither Lockheed has no real interest in your website other than what it was contracted to do. It obviously came from a government agency that has a keen interest in your website, and it wouldn’t surprise me if it is either the FBI or a directive straight from Alberto Gonzales to keep tabs on you. The request for the Justice Department intervention was probably insitagated by the top echelon within Homeland Security itself. My question is, what was the offending article you ran before the ping?
Jeremy
Feb 19, 2006
Wow Mr. Anonymous, that has to be one of the most rediculous things I’ve ever seen posted. Lockheed Martin intentionally “attacking” this site. Michael already said he found out what happened.
BTW, DIA and NSA are part of the DoD. Don’t be a consiparcy theorist and pretend to be a “former government ewmployee” at the same time.
Anonymous
Feb 27, 2006
To Jeremy and Anonymous: I think you are right Anonymous. You can easily find this on: http://www.bluecoat.com/ Here is the text below.
Blue Coatâ„¢ enables government organizations to keep “good” employees from doing “bad” things on the Internet. Blue Coat proxy appliances provide visibility and control of Web communications to address today’s new risks-such as inappropriate Web surfing, viruses brought in via back door channels such as instant messaging and Web-based email, and network resource abuse due to peer-to-peer (P2P) file sharing and video streaming. Trusted by the US Air Force, SEC, NASA and more, Blue Coat has shipped more than 20,000 proxy appliances worldwide.
Blue Coat has significant experience in providing government agencies with powerful control and security over mission-critical communications and information. The Blue Coat Proxy SGâ„¢ family of proxy appliances provides total visibility and control of Web communications with wire-speed performance. Based on Blue Coat SGOSâ„¢, a custom, object-based operating system with integrated caching, these proxy appliances leverage existing authentication systems to enable flexible policy enforcement down to the individual user. Proxy SG provides comprehensive proxy support of all Web protocols with integrated content filtering, instant messaging control, peer-to-peer (P2P) control, streaming control, pop-up ad blocking and virus scanning. Blue Coat’s end-to-end product portfolio includes powerful reporting, policy and configuration management software – delivering a scalable proxy solution for centralized or distributed environments. Click here for more information on Blue Coat solutions.
Anonymous
Feb 27, 2006
Jeremy,
As ridiculous as you make think it seems, it happens all the time, particularly to websites that poses a name contrary to Homeland Security (like Homeland Stupidity). It is like waving a red flag at them. Ever ask why Lockheed uses Blue Coat for prefetching websites? It is normally used to protect against or block viruses, invasive websurfing, spyware, and stuff from unsecure webservers. However, it is also used for viewing web activity for security reasons. It is scripted pretty well so you should not have any problems prefetching pages fast. But it can be programmed to interrupt traffic, or if someone is intentionally watching traffic via the proxy server, then yes, it will give you denial messages as a result if they choose to interrupt you. So that brings you to asking why Lockheed’s proxy servers are even interferring with e-traffic outside its own corporate confines at all unless they are intentionally watching traffic. Also you are not privy to the government contracts held by Lockheed to know what they are doing, but I can tell you that most all of their contracts are with the government and they do not run public proxy servers for the benefit of non-government internet users.
You are not quite right about DIA and NSA as part of DoD. NSA is a separate governmental entity only answerable to the president, not DoD. Yes DIA subagency under DoD, but you excluded mentioning the Justice Dept which houses the FBI and they are not part of DoD at all, and neither is the Treasury Dept which houses the Secret Service guys.
I suggest you get to know your government a little better.
Michael Hampton
Feb 27, 2006
Oh, I completely forgot Anonymous #4′s original claims. (Will you people please pick names for yourselves? They don’t have to be real.) That LMCO was testing my capabilities. That the government is monitoring my server. That the government has planted spyware in my computer.
Bullshit.
Occam’s Razor, people. I don’t now think the prefetching was any sort of attempt to “test” my capabilities, not after actually reviewing the logs.
And speaking of the logs, I had, about two weeks before this took place, actually moved to a completely different server under my complete control, with a fresh copy of the operating system, and all services hand-configured by yours truly. (Which is why this problem came about in the first place.)
Finally, my computer is also new, purchased at about the same time at random from a stock of laptops at a major electronics store which I don’t normally visit. It hasn’t been out of my sight since.
Oh, and with a little help from, er, someone with firsthand knowledge of the techniques involved, have been able to do some dry cleaning and ensure I am not currently under any sort of physical surveillance.
Though, honestly, from what I’ve been able to determine, even the government bureaucrats who stumble across my site either get a good laugh out of it, or find themselves grudgingly agreeing — or wholeheartedly agreeing — with what I have to say. They even share it with their friends and co-workers.
It probably helps that the Department of Justice can’t access this site at all, but that’s their own fault…
Michael Hampton
Feb 27, 2006
Blue Coat currently classifies this site under “News/Media.” I like that. It means virtually nobody behind one of these proxies will find this site blocked. A few rare places actually do block this category, though since it also blocks sites such as CNN.com, MSNBC, etc., it is extremely rare.
I’d like to keep it that way.
As for the automated prefetching process, as I mentioned previously I had taken steps to tune the server to remain up and responsive during one of these conditions, and so far it has done so admirably. I think I’m ready to be /.ed.