Homeland Stupidity was unavailable for six hours Friday night due to a distributed denial of service attack. Steps are being taken to mitigate the effects of future denial of service attacks. All services have been restored as of early Saturday morning.

At the moment I believe the attack was simply an attempt to compromise site security by breaking in. The attackers used a botnet to flood the server with so many requests that it could not keep up. When I discovered the problem, the server was trying to deal with almost 2,000 simultaneous requests from all over the Internet, and of course failing. Even though the attack had ended at that point, the server software had gotten confused by the sheer number of requests.

The attack began at around 8:30 p.m. and the server was overloaded by 8:32 p.m. Normal service has been restored as of 2:30 a.m.

Based on the available data, the attack appears to have been targeted at this site, probing more or less at random for any security holes in the installed software. A review of the server activity indicates no actual breach of security occurred.

For performance reasons, this server currently uses Apache 2.2 with the worker MPM, as well as several other performance optimizations. However, the worker MPM is still considered unstable, meaning that it may have bugs which would cause it to fail. This is currently what I believe happened. The server has previously handled very large loads without incident, e.g. appearing on the front page of Slashdot.org and digg.com at the same time. In fact, that’s why the performance optimizations are there in the first place.

In the future, the system will be monitored for unusual activity such as this, with real-time countermeasures taken in the event of another attack. In addition, time permitting, I will spend some time debugging the Apache code to determine if any bugs exist in the worker MPM which would cause the server to stop working and to fix them.

Update: The service monitoring is working fine. It pages me whenever anything unusual happens. In fact, it just paged me to let me know I’d logged in to one of the servers being monitored!