New Datacenter for WordPress.com

Towards the end of 2008, we brought online a new datacenter to serve the over 5.5 million blogs now hosted on the WordPress.com platform.  Adding the data center in Chicago, IL gives us a total of 3 data centers across the US which serve live content at any given time.  We have decommissioned one of our facilities in the Dallas, TX area.  Our friends at Layered Technologies were kind enough to shoot this footage for us (think The Blair Witch Project) and the always awesome Michael Pick took care of the editing.  Here’s a peak at what a typical WordPress data center installation looks like…

done

For those interested in technical details here is a hardware overview of the installation:

150 HP DL165s dual quad-core AMD 2354 processors 2GB-4GB RAM
50 HP DL365s dual dual-core AMD 2218 processors 4GB-16GB RAM
5 HP DL185s dual quad-core AMD 2354 processors 4GB RAM

And here is a graph of what the current CPU usage looks like across about 700 CPU cores.  As you can see there is plenty of idle CPU for those big spikes or in case one of the other 2 data centers fail and we have to route more traffic to this one.

Anatomy of a Denial of Service Attack

Running one of the largest websites on the internet with about 5 million unique sites hosted exposes you to all sorts of issues.  There are constant events to deal with, some internal, some external.  This morning, one of the more common external events, a Distributed Denial of Service Attack occurred.  We experience these types of attacks rather frequently, but most are easily mitigated and have no user impact.  One this morning, however, was rather large and thus impacted some users.

Here is a timeline and description of this morning’s events:

9:40 AM EST — Our internal monitoring systems alerted us to unusual activity in one of the four geographically diverse datacenters which serve WordPress.com traffic.  Here is what that anomaly looks like in graphical terms:

10:00 AM EST — The target of the attack was identified and removed from our network.  The attack, however continued.  This is because the attacker had hijacked tens of thousands of computers (probably by installing a virus which was spread via email) and these computers had no idea the site was no longer there.  A small log sample shows over 8 million requests for this one site from over 10,000 unique IP addresses.

10:20 AM EST — Since we have servers in multiple data centers throughout the United States which serve traffic for WordPress.com all the time, we were able to route all legitimate traffic out of the affected data center, and let the single affected data center deal with the attack.   

11:30 AM EST — The IPs targeted in the attack were null routed at this point which allowed us to bring all datacenters back online to serve normal traffic.

We keep hourly traffic metrics and based on those numbers, it looks like during the attack there was about a 5% decrease in overall pageviews during the 40 minutes before traffic was re-routed.  All things considered, not a bad outcome for an attack this size.  Looking at bandwidth graphs, this attack was in the 500Mbit – 750Mbit/sec range.  

WordPress.com using S3

Demitrious has a great post explaining how we are using S3, Varnish, and Pound to serve 60 million image requests per day on WordPress.com

UPDATE: Almost forgot, but Matt reminded me, he has a really super duper awesome post about WordPress.com and S3 too!

One milllllliion blogs on WordPress.com

At 10:56:22PM PDT on 5/23/2007, the 1 millionth active blog was registered on WordPress.com. And the winner is…..

claudiacanals.wordpress.com

Not much there right now, but hopefully there will be soon. Maybe head over and leave a comment on their about page to let them know!

Predictions on how long it will take to get to 2 million?

Additional Capacity

So, I haven’t blogged much lately but there is a reason. Over the past month we have been hard at work expanding the infrastructure behind WordPress.com and Akismet. Here are some of the things that we have done over the past month or so:

• Migrated out of San Diego
• Brought online almost 100 new servers in 3 new datacenters — Dallas, TX, San Antonio, TX, and San Francisco, CA
• Tripled the database hardware behind WordPress.com
• Now serving WordPress.com blogs out of 3 datacenters in real-time
• Akismet is now served from 2 datacenters

Here are a couple pictures of some new hardware racked and powered on just before we put it into production last week.

From top to bottom (left):

• 21 x HP DL145
• 4 x HP DL365

From top to bottom (right):

• 18 x HP DL145
• 4 x HP DL365
• 1 x 3U HP Storage Array
• 1 x HP DL385

And the back….

Thanks to Evan League and Brian Maples of Layered Tech for doing the build-out pictured above and sending the photos over.