Anatomy of a Denial of Service Attack

27 Oct

Running one of the largest websites on the internet with about 5 million unique sites hosted exposes you to all sorts of issues.  There are constant events to deal with, some internal, some external.  This morning, one of the more common external events, a Distributed Denial of Service Attack occurred.  We experience these types of attacks rather frequently, but most are easily mitigated and have no user impact.  One this morning, however, was rather large and thus impacted some users.

Here is a timeline and description of this morning’s events:

9:40 AM EST — Our internal monitoring systems alerted us to unusual activity in one of the four geographically diverse datacenters which serve WordPress.com traffic.  Here is what that anomaly looks like in graphical terms:

10:00 AM EST — The target of the attack was identified and removed from our network.  The attack, however continued.  This is because the attacker had hijacked tens of thousands of computers (probably by installing a virus which was spread via email) and these computers had no idea the site was no longer there.  A small log sample shows over 8 million requests for this one site from over 10,000 unique IP addresses.

10:20 AM EST — Since we have servers in multiple data centers throughout the United States which serve traffic for WordPress.com all the time, we were able to route all legitimate traffic out of the affected data center, and let the single affected data center deal with the attack.   

11:30 AM EST — The IPs targeted in the attack were null routed at this point which allowed us to bring all datacenters back online to serve normal traffic.

We keep hourly traffic metrics and based on those numbers, it looks like during the attack there was about a 5% decrease in overall pageviews during the 40 minutes before traffic was re-routed.  All things considered, not a bad outcome for an attack this size.  Looking at bandwidth graphs, this attack was in the 500Mbit – 750Mbit/sec range.  

27 Responses to “Anatomy of a Denial of Service Attack”

  1. raincoaster October 27, 2008 at 9:46 pm #

    I hope it wasn’t just me suddenly becoming popular. Sorry, guys!

  2. Chris October 27, 2008 at 10:00 pm #

    Nice job, what were they attacking?

  3. Leon Poole October 28, 2008 at 3:57 am #

    Nice – I’ve never seen it graphically like that before but looks like it was handled well.

  4. Joe Jacobs October 28, 2008 at 8:38 am #

    What software do you use to monitor the traffic and get the graphs, etc?

  5. Barry October 28, 2008 at 8:44 am #

    Munin – http://munin.projects.linpro.no/

  6. Lee Newton October 28, 2008 at 11:33 am #

    Well handled Barry :) Crap I’m always humbled by the amount of traffic that goes through wordpress.com!

  7. Noel Jackson October 28, 2008 at 11:57 pm #

    Barry is my hero. The 5% decrease is pretty amazing (that it was that little). Dang.

  8. Kevin Blalock October 29, 2008 at 11:25 pm #

    Excellent work Barry! That is a pretty sizable attack for sure! It always impresses me at how fast traffic can be rerouted with a well planned out setup like you guys have in place!

  9. YOU KNOW November 12, 2008 at 6:59 pm #

    I hate the internet.

  10. @magitam December 24, 2008 at 12:33 pm #

    Wow! Thanks for sharing that :) It’s good to be able to see just how ably WordPress keeps itself protected, and how easily you can manouver around DoS attacks! Nothing like this to instill ever more faith in the great service you guys offer! Thanks :)

  11. Vlasi December 25, 2008 at 10:39 am #

    I hate the internet

  12. Gary January 8, 2009 at 9:51 pm #

    Very interesting. Thanks. It’s interesting to see a more detailed ‘timeline’ of events and to see the impact on graphs.

  13. llbbl January 14, 2009 at 1:51 pm #

    wow

  14. блесенка January 27, 2009 at 7:27 am #

    Omg…..
    for example my site Блесенка.ру have 3 mb traff per day +)

  15. Emre Sokullu January 29, 2009 at 3:12 am #

    Thanks for sharing this info! Although smaller in scale obviously, we’re running into similar problems too.

    Could you share in a new post what kind of tools you use to prevent ddos?

    Cheers,

  16. Greg February 18, 2009 at 12:29 pm #

    прикольно, нигде не видал

Trackbacks/Pingbacks

  1. DOS Attack Brought Down WordPress.com Blogs | The Blog Herald - October 29, 2008

    [...] network. The only obvious communication that I could find about this matter was a tweet linking to a blog post about DOS attacks in general, and the actual one in particular, also written by a WordPress.com team representative. [...]

  2. BlogMaster - WordPress.com sotto attacco DDoS - October 29, 2008

    [...] Non ne è stata data nessuna comunicazione ufficiale e quello che si sa, come spesso accade, arriva per vie traverse. I fatti si riferiscono ad appena un paio di giorni fa e l’attacco ha interessato tutti i blog ospitati sulla piattaforma di WordPress.com, inclusi alcuni ospiti vip come il network GigaOM. Questa volta però non si è trattato del solito attacco sporadico, debole e senza particolare impatto sulla piattaforma, ma di un attacco DDoS davvero massiccio e ben architettato, portato avanti con una potenza di fuoco tra i 500 ed i 750 Mbit/s con più di 10000 macchine coinvolte. Gli amanti dei dettagli tecnici e dei grafici di carico del server e traffico, possono trovare maggiori informazioni sul blog di Barry. [...]

  3. WordPress.com blogs DOS attacked | WordPress Philippines - October 29, 2008

    [...] no official word on any of the WordPress/Automattic blogs, only a tweet from @wordpressdotcom and a post entitled “Anatomy of a Denial of Service [...]

  4. Denial of Service Attack Hits WordPress | Network Industry Review - October 30, 2008

    [...] Blog Herald picks up the story, pointing to Barry’s post with the stats. It looks like some hackers had a beef with one of the blogs they were hosting and directed .5/Gig [...]

  5. WordPress.com sufre un ataque DDOS | Ayuda WordPress - December 14, 2008

    [...] comunicación disponible acerca de este asunto es un tweet enlazando al post de un blog acerca de ataques DOS, pero en general, sin precisar, también comentado por un representante del equipo de [...]

  6. 2008 Year-End Wrap-Up « Blog « WordPress.com - January 2, 2009

    [...] is the case with any year, there were a couple rough days in 2008, but we survived a DOS attack or two with very minimal downtime, and learned a lot in the process that will have us better prepared in [...]

  7. 2008 Year-End Wrap-Up « Dunghue’s Blog - January 16, 2009

    [...] is the case with any year, there were a couple rough days in 2008, but we survived a DOS attack or two with very minimal downtime, and learned a lot in the process that will have us better prepared in [...]

  8. Karma! Web™ Web Design » Blog Archive » 2008 Year-End Wrap-Up - February 10, 2009

    [...] is the case with any year, there were a couple rough days in 2008, but we survived a DOS attack or two with very minimal downtime, and learned a lot in the process that will have us better prepared in [...]

  9. New Datacenter for WordPress.com « Barry on WordPress - February 16, 2009

    [...] looks like across about 700 CPU cores.  As you can see there is plenty of idle CPU for those big spikes or in case one of the other 2 data centers fail and we have to route more traffic to this [...]

  10. Linux Readers » Blog Archive » Datacenter for WordPress.com - May 12, 2009

    [...] looks like across about 700 CPU cores.  As you can see there is plenty of idle CPU for those big spikes or in case one of the other 2 data centers fail and we have to route more traffic to this [...]

  11. WordPress.com Targeted by "Extremely Large" DDOS Attack | Liz Gannes | NetworkEffect | AllThingsD - March 3, 2011

    [...] has said it receives DDOS attacks frequently, but is usually able to contain them from affecting users. (The [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 852 other followers

%d bloggers like this: