WordPress.com DDoS Details

7 Mar

As you may have heard, on March 3rd and into the 4th, 2011, WordPress.com was targeted by a rather large Distributed Denial of Service Attack. I am part of the systems and infrastructure team at Automattic and it is our team’s responsibility to a) mitigate the attack, b) communicate status updates and details of the attack, and c) figure out how to better protect ourselves in the future.  We are still working on the third part, but I wanted to share some details here.

One of our hosting partners, Peer1, provided us these InMon graphs to help illustrate the timeline. What we saw was not one single attack, but 6 separate attacks beginning at 2:10AM PST on March 3rd. All of these attacks were directed at a single site hosted on WordPress.com’s servers. The first graph shows the size of the attack in bits per second (bandwidth), and the second graph shows packets per second. The different colors represent source IP ranges.

The first 5 attacks caused minimal disruption to our infrastructure because they were smaller in size and shorter in duration. The largest attack began at 9:20AM PST and was mostly blocked by 10:20AM PST. The attacks were TCP floods directed at port 80 of our load balancers. These types of attacks try to fill the network links and overwhelm network routers, switches, and servers  with “junk” packets which prevents legitimate requests from getting through.

The last TCP flood (the largest one on the graph) saturated the links of some of our providers and overwhelmed the core network routers in one of our data centers. In order to block the attack effectively, we had to work directly with our hosting partners and their Tier 1 bandwidth providers to filter the attacks upstream. This process took an hour or two.

Once the last attack was mitigated at around 10:20AM PST, we saw a lull in activity.  On March 4th around 3AM PST, the attackers switched tactics. Rather than a TCP flood, they switched to a HTTP resource consumption attack.  Enlisting a bot-net consisting of thousands of compromised PCs, they made many thousands of simultaneous HTTP requests in an attempt to overwhelm our servers.  The source IPs were completely different than the previous attacks, but mostly still from China.  Fortunately for us, the WordPress.com grid harnesses over 3,600 CPU cores in our web tier alone, so we were able to quickly mitigate this attack and identify the target.

We see denial of service attacks every day on WordPress.com and 99.9% of them have no user impact. This type of attack made it difficult to initially determine the target since the incoming DDoS traffic did not have any identifying information contained in the packets.  WordPress.com hosts over 18 million sites, so finding the needle in the haystack is a challenge. This attack was large, in the 4-6Gbit range, but not the largest we have seen.  For example, in 2008, we experienced a DDoS in the 8Gbit/sec range.

While it is true that some attacks are politically motivated, contrary to our initial suspicions, we have no reason to believe this one was.  We are big proponents of free speech and aim to provide a platform that supports that freedom. We even have dedicated infrastructure for sites under active attack.  Some of these attacks last for months, but this allows us to keep these sites online and not put our other users at risk.

We also don’t put all of our eggs in one basket.  WordPress.com alone has 24 load balancers in 3 different data centers that serve production traffic. These load balancers are deployed across different network segments and different IP ranges.  As a result, some sites were only affected for a couple minutes (when our provider’s core network infrastructure failed) throughout the duration of these attacks.  We are working on ways to improve this segmentation even more.

If you have any questions, feel free to leave them in the comments and I will try to answer them.

20 Responses to “WordPress.com DDoS Details”

  1. sillyandrea March 7, 2011 at 11:59 am #

    I’m curious as to how you narrowed down finding out which blog was being targeted. Assuming it was a subdomain, on all the tools I have access to, that part is truncated off. In some cases, specific posts are targeted, so I can work backwards from that.

    But finding a specific virtual subdomain in logs? Yeah, I’m lost. :D

  2. Taylor March 7, 2011 at 1:20 pm #

    Taylor here, I’m a WordPress dev at Forbes. I was interested to read how the attackers switched from TCP flood to HTTP resource consumption late in the game.

    Do you think the attackers would have been any more successful (at least temporarily) with an attack vector like Slowloris (http://ha.ckers.org/slowloris/), which does long term requests at regular intervals? How might you respond to such an attack?

    • Barry March 7, 2011 at 1:27 pm #

      Howdy Taylor. Probably not, since attacks such as Slowloris are just trying to consume connections, not CPU on servers/routers/switches or bandwidth. Modern web servers, such as nginx can handle many thousands of simultaneous connections and have parameters than can be tuned to minimize the impacts of such attacks. How we would respond depends on the size and other details of the attack.

  3. Mark McWilliams March 7, 2011 at 2:03 pm #

    WOW at the graphs, especially the second one!

  4. Destination Infinity March 14, 2011 at 10:14 pm #

    I am not sure how load balancers help in migitaging DDOS attacks… Even if the load balancers are fed with a huge amount of traffic, they would be unavailable for others, right?

    Destination Infinity

    • Barry March 19, 2011 at 11:28 am #

      Yes, true, but we have many load balancers, so we are load balancing across load balancers :) Although there are other factors involved, mitigating DDoS attacks is a lot about capacity – if you have more capacity than the attacker, you win.

Trackbacks/Pingbacks

  1. WordPress.com DDoS Details | Joseph Scott - March 7, 2011

    [...] WordPress.com DDoS Details, Barry Abrahamson (lead sysadmin at Automattic). [...]

  2. Some Turbulence — Blog — WordPress.com - March 8, 2011

    [...] us to work harder. Rest assured that behind-the-scenes here at Automattic we’re learning as much as we can from these incidents so we’re even better prepared and more able to adapt next time something [...]

  3. Turbulência — Blog — WordPress.com - March 8, 2011

    [...] trabalhar com mais afinco. Fique descansado, porque aqui nos bastidores da  Automattic estamos a tirar as lições destes incidentes para melhor nos preparar e adaptar se e quando situações inesperadas voltem a [...]

  4. Un peu de turbulence — Blog — WordPress.com - March 8, 2011

    [...] pousse à travailler encore plus. Soyez assurés que dans les coulisses, ici àAutomattic nous  apprenons tout ce que nous pouvons de ces incidents, pour être encore mieux préparés et plus en mesure de nous adapter, la [...]

  5. Turbulencias — Bitácora — WordPress.com - March 8, 2011

    [...] de la disponibilidad nos lleva a trabajar más duro. Descansad seguros de que en Automattic estamos aprendiendo todo lo que podemos de estos incidentes para, la próxima vez que se nos cruce algo inesperado, estar incluso más [...]

  6. That’s A Lot Of Hardware - March 8, 2011

    [...] I’m sure Automattic has help with regards to managing all that hardware but I wonder how Barry does it and how much of the hardware load he has to worry [...]

  7. いくらかの混乱 — ブログ — WordPress.com - March 9, 2011

    [...] 僕らのチームはサービスの提供状況が100%を少しでも下回った場合、。Automattic の舞台裏では、これらの出来事からできるだけ多くの情報を収集し、もしも次回予期しないことが起こってしまった場合、さらに準備万端でいられるように努力しています (WP.com では毎日 DDoS 攻撃が起こっていますが、そのうち99.9%はユーザーに全く影響を与えていません) 。 [...]

  8. Yesterday’s DoS Attack – Response from CEO | Codero - March 9, 2011

    [...] and the methods of their originators are becoming more and more sophisticated.  Similar attacks hit the free-speech platform WordPress® as recently as last [...]

  9. It’s just Business « AlterNet Views - March 9, 2011

    [...] interesting is that denial of service attacks occurs every day on WordPress.com and 99.9% of them have no user impact. We even have dedicated infrastructure for [...]

  10. Vaultpress and content snapshot of Groundviews « ICT for Peacebuilding (ICT4Peace) - March 20, 2011

    [...] of last week, Groundviews is backed up 24/7 using Vaultpress. In part, this was prompted by the DDoS attack targeted at WordPress.com recently. Not for the first time, this attack demonstrated that even the most robust web companies [...]

  11. WordPress.Com Dashboard After 4 Hours Downtime | Linear Fix - March 22, 2011

    [...] downtime was not caused by a denial of service attempt which caused the last major downtime on WordPress.com earlier this month. Statistics on WordPress.com hosted blogs are now updating and [...]

  12. China attacks Wordpress.com: ‘Largest and most sustained attack’ in WP history | COTO Report - March 23, 2011

    [...] tech-defender, Automattic, said the attack was in the 4-6 Gbit range, but is dwarfed by a 2008 attack of 8 gigabits per [...]

  13. China attacks Wordpress.com: 'Largest and most sustained attack' in WP history - Infowars Ireland - August 3, 2013

    […] tech-defender, Automattic, said the attack was in the 4-6 Gbit range, but is dwarfed by a 2008 attack of 8 gigabits per […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 856 other followers

%d bloggers like this: