On WordPress.com we have over 218 million tables and perform tens of thousands queries per second. To scale all of this, we shard our 24 million blogs across more than 550 MySQL servers. This allows us to cope with load bursts and to handle database servers failures.

For those who are unfamiliar, MySQL data replication is asynchronous and works as follows:

1. [Master] Receives a query that modifies database structure or content (INSERT, UPDATE, ALTER etc.)
2. [Master] The query is written to a log file (aka the binlog).
3. [Master] The query is executed on the master.
4. [Slaves] Create a “Slave I/O” thread that connects to the [Master] and requests all new queries from the mater’s binlog.
5. [Master] Creates a “Binlog dump” thread for each connected slave, that reads the requested events from the binlog and sends them to the slave.
6. [Slaves] Start a “Slave SQL” thread which reads queries from the log file written by the “Slave I/O” thread and executes them

There are a number of things to be considered in this scenario, which can lead to a condition known as replication lag where the slaves have older data then the master:

• Since only one thread on the slave executes write queries, and there are many execution threads on the master, there is no guarantee that the slave will be able to execute queries with the same speed as the master.
• Long running SELECTs or explicit locks on the slave, will cause the “Slave SQL” thread to wait, thus slowing it down.
• Long running queries on the master would take at least the same amount of time to run on the slave, causing it to fall behind the master
• I/O (disk or network) issues can prevent or slow down the slave from reading and replaying the binlog events

In order to deal with this, we needed a way to avoid connections to lagged slaves as long as there are slaves that are current. This would allow for the lagged ones to recover faster and avoid returning old data to our users. It also had to be something flexible enough, so we could have different settings for acceptable replication lag per dataset or stop tracking it altogether. Since we use the advanced database class, HyperDB, for all our database connections, it was the obvious place to integrate this.

We implemented it  in the following steps:

• If a connection modifies data in a given table, then all subsequent SELECTs on the same connection for that table are sent to the master. Chances are replication won’t be fast enough to propagate the changes to the slaves on the same page load.  This logic has existed in HyperDB for a while.
• Before we make a connection to a slave, we use a callback, to check if we have information for this slave’s lag in the cache and we skip it based on that, unless all slaves in the dataset are considered lagged.  In case replication breaks on all slaves, we would rather return old data then overload the master with read queries and cause an outage.
• After a successful connection to a slave, if there was nothing in the cache regarding its lag status and not all slaves are considered lagged, we execute a second callback that checks whether this slave is lagged and updates the cache.

A slave is considered lagged when it has a “lag threshold” defined in it’s dataset configuration and the current lag is more than this threshold.

We considered the following options for checking if a slave is lagged.  No MySQL patches are required for any of them:

• Checking the value of Seconds_Behind_Master from the SHOW SLAVE STATUS statement executed on the slave. It shows the difference between the timestamp of the currently executed query and the latest query we have received from the master. Although it is easy to implement and has low overhead, the main problem with using this option is that it is not completely reliable, as it can be tricked by IO latency and/or master connection problems.
• Tracking the “File” and “Position” on SHOW MASTER STATUS executed on the master and comparing it to Relay_Master_Log_File and Exec_Master_Log_Pos of SHOW SLAVE STATUS on the slave. This way we can wait until the slave executes the queries from binlog “file” and position “position” before send certain queries to that slave and thus effectively we wait for the data to be replicated to the point where we need it. While very reliable, this option is more complex, has lots of overhead and doesn’t give us clock time value which we can track and set between servers.
• Tracking the difference between the current time on the slave and the replication of a timestamp update from the master, which runs every second. This is basically what mk-heartbeat does. It requires proper time sync between the master and the slave servers but is otherwise very reliable.

The third option fit our needs best, however the code is flexible enough to easily support any of these. For caching, we decided to go with memcached, since it works well in our distributed, multi-server, multi-datacenter environment, but other methods (APC cache, shared memory, custom daemon etc.) would work just fine.

HyperDB is free, open-source and easy to integrate in your WordPress installation. You can download it here.  We hope you enjoy this new functionality and please let us know if you have any questions in the comments.

One of our hosting partners, Peer1, provided us these InMon graphs to help illustrate the timeline. What we saw was not one single attack, but 6 separate attacks beginning at 2:10AM PST on March 3rd. All of these attacks were directed at a single site hosted on WordPress.com’s servers. The first graph shows the size of the attack in bits per second (bandwidth), and the second graph shows packets per second. The different colors represent source IP ranges.

The first 5 attacks caused minimal disruption to our infrastructure because they were smaller in size and shorter in duration. The largest attack began at 9:20AM PST and was mostly blocked by 10:20AM PST. The attacks were TCP floods directed at port 80 of our load balancers. These types of attacks try to fill the network links and overwhelm network routers, switches, and servers  with “junk” packets which prevents legitimate requests from getting through.

The last TCP flood (the largest one on the graph) saturated the links of some of our providers and overwhelmed the core network routers in one of our data centers. In order to block the attack effectively, we had to work directly with our hosting partners and their Tier 1 bandwidth providers to filter the attacks upstream. This process took an hour or two.

Once the last attack was mitigated at around 10:20AM PST, we saw a lull in activity.  On March 4th around 3AM PST, the attackers switched tactics. Rather than a TCP flood, they switched to a HTTP resource consumption attack.  Enlisting a bot-net consisting of thousands of compromised PCs, they made many thousands of simultaneous HTTP requests in an attempt to overwhelm our servers.  The source IPs were completely different than the previous attacks, but mostly still from China.  Fortunately for us, the WordPress.com grid harnesses over 3,600 CPU cores in our web tier alone, so we were able to quickly mitigate this attack and identify the target.

We see denial of service attacks every day on WordPress.com and 99.9% of them have no user impact. This type of attack made it difficult to initially determine the target since the incoming DDoS traffic did not have any identifying information contained in the packets.  WordPress.com hosts over 18 million sites, so finding the needle in the haystack is a challenge. This attack was large, in the 4-6Gbit range, but not the largest we have seen.  For example, in 2008, we experienced a DDoS in the 8Gbit/sec range.

While it is true that some attacks are politically motivated, contrary to our initial suspicions, we have no reason to believe this one was.  We are big proponents of free speech and aim to provide a platform that supports that freedom. We even have dedicated infrastructure for sites under active attack.  Some of these attacks last for months, but this allows us to keep these sites online and not put our other users at risk.

We also don’t put all of our eggs in one basket.  WordPress.com alone has 24 load balancers in 3 different data centers that serve production traffic. These load balancers are deployed across different network segments and different IP ranges.  As a result, some sites were only affected for a couple minutes (when our provider’s core network infrastructure failed) throughout the duration of these attacks.  We are working on ways to improve this segmentation even more.

If you have any questions, feel free to leave them in the comments and I will try to answer them.

For those interested in technical details here is a hardware overview of the installation:

150 HP DL165s dual quad-core AMD 2354 processors 2GB-4GB RAM
50 HP DL365s dual dual-core AMD 2218 processors 4GB-16GB RAM
5 HP DL185s dual quad-core AMD 2354 processors 4GB RAM

And here is a graph of what the current CPU usage looks like across about 700 CPU cores.  As you can see there is plenty of idle CPU for those big spikes or in case one of the other 2 data centers fail and we have to route more traffic to this one.

Here is a timeline and description of this morning’s events:

9:40 AM EST — Our internal monitoring systems alerted us to unusual activity in one of the four geographically diverse datacenters which serve WordPress.com traffic.  Here is what that anomaly looks like in graphical terms:

10:00 AM EST — The target of the attack was identified and removed from our network.  The attack, however continued.  This is because the attacker had hijacked tens of thousands of computers (probably by installing a virus which was spread via email) and these computers had no idea the site was no longer there.  A small log sample shows over 8 million requests for this one site from over 10,000 unique IP addresses.

10:20 AM EST — Since we have servers in multiple data centers throughout the United States which serve traffic for WordPress.com all the time, we were able to route all legitimate traffic out of the affected data center, and let the single affected data center deal with the attack.   

11:30 AM EST — The IPs targeted in the attack were null routed at this point which allowed us to bring all datacenters back online to serve normal traffic.

We keep hourly traffic metrics and based on those numbers, it looks like during the attack there was about a 5% decrease in overall pageviews during the 40 minutes before traffic was re-routed.  All things considered, not a bad outcome for an attack this size.  Looking at bandwidth graphs, this attack was in the 500Mbit – 750Mbit/sec range.  

UPDATE: Almost forgot, but Matt reminded me, he has a really super duper awesome post about WordPress.com and S3 too!

claudiacanals.wordpress.com

Not much there right now, but hopefully there will be soon. Maybe head over and leave a comment on their about page to let them know!

Predictions on how long it will take to get to 2 million?

• Migrated out of San Diego
• Brought online almost 100 new servers in 3 new datacenters — Dallas, TX, San Antonio, TX, and San Francisco, CA
• Tripled the database hardware behind WordPress.com
• Now serving WordPress.com blogs out of 3 datacenters in real-time
• Akismet is now served from 2 datacenters

Here are a couple pictures of some new hardware racked and powered on just before we put it into production last week.

From top to bottom (left):

• 21 x HP DL145
• 4 x HP DL365

From top to bottom (right):

• 18 x HP DL145
• 4 x HP DL365
• 1 x 3U HP Storage Array
• 1 x HP DL385

And the back….

Thanks to Evan League and Brian Maples of Layered Tech for doing the build-out pictured above and sending the photos over.

Background

The model we have adopted is to use commodity hardware to serve all the functions of the site. We do not rely on SANs or super-expensive multi-processor systems. Our web servers are usually either single or dual processor machines with 1-2GB of RAM and a small, inexpensive hard drive. Our database servers are single or dual processor machines with 4-8GB of RAM and 2-4 fast SCSI drives in a RAID array using a hardware RAID controller. Because there is redundancy built into the architecture that several of these machines can fail at once at the site is unaffected, the individual machines do not need to be extremely robust. Historically, CPU time has been the most precious resource on the web servers and disk I/O on the DBs.

Requirements

• Provide the hardware outlined above
• Support Debian AMD64 or similar
• Provide a Gigabit private backend network for inter-server communication
• Ability to deploy additional servers quickly and painlessly
• Sales and support team that is competent and easy to work with
• US Datacenter and not Dallas or San Diego

Initial Impressions

Dedicated server providers seem to fall into 2 classes:

1) No-frills provider that offers server between $79 – $149/month. No phone support or advanced services. This would be fine, but they usually cannot provide the DB-class machines we need and do not provide Gigabit backend networks.

2) Full-service provider that offers servers beginning at $250/month and up. These providers are usually not the best fit for us because they justify their higher prices by saying they have superior support. We don’t really need “superior” support, just someone that can take care of hardware issues when needed. Seems like a waste to pay for something you are never going to use and don’t really need. Also, in my experience, the more expensive the hosting company, the more painful the sales process is. Sometimes I feel like I am buying a car….

The Finalists

Over the past month, we have narrowed the field from about 10 different possible providers down to the following 3. Each of these fit somewhere between no-frills and full-service as mentioned above, but I definitely get the feeling that they lean one way or the other.

• ServePath
  Based in San Francisco, California (where Automattic is also based) they seem to lean towards the full-service side, offering 24×7 phone support, higher-end servers, load balancers, and firewalls. Their sales process has been pretty agonizing. It is now going on 45 days of back and forth, price changes, configuration changes, conference calls, and about 50 one-on-one calls with our sales guy. I had chance to visit their offices and tour their datacenter earlier this week so I got a feel of how things work there — it appears to be a well-run organization. One thing that struck me as a bit odd was that they do not deploy any rack-mounted servers in their datacenter. All of their servers, which are built in house, are in tower (white-box) chassis like you would see under an office desk. This is something I would expect in the lower-end market, but at $500+ per month and rather large RAID arrays (300GB SCSI x 6) I would expect to see more rack-mounted chassis to take advantage of the superior cooling.

• Server Beach
  Based in San Antonio, Texas and now owned by Peer1, Server Beach is definitely more on the no-frills side. When we were looking at new datacenters about 6 months ago, Server Beach could not provide what we wanted — they did not offer SCSI RAID or Gigabit privatenet, but said it was coming. Well, those things are now available. Kudos to Server Beach on a super-simple and painless sales process. Once I contacted them with the configuration we wanted, they scheduled a conference call to discuss the details. They had members of their support, operations, and sales teams on the line. About 30 minutes later we were all done and a day later they had sent over a proposal. It was right the first time – had everything we asked for and at a very fair price.

• Joyent/TextDrive
  Honestly, we probably will not go with TextDrive for this deployment, as it is a pretty radical departure from our current configuration, and they don’t technically meet the requirements laid out above. They are worth mentioning, however, because they are doing some pretty cool stuff with OpenSolaris, zones, and ZFS, and it sounds like it could be a good fit for our architecture model. Their storage is super-fast, the container model allows you to replicate existing containers with a single command — there is no need to provision and setup a new physical server. ZFS offers all sorts of cool stuff like snapshots, compression, and built-in data consistency checks. Utilizing this architecture, would require that we maintain 2 completely separate environments — Linux and Solaris — and there is a definite time investment in doing so. I think that we will probably look at moving some of our services to Solaris containers in the near future, but I am not sure it will be WordPress.com.

The Verdict

First, let me say that we still have yet to make the final decision, but hope to do so by the end of the week. Both ServePath and Server Beach seem like they will be great companies to work with. ServePath is local, so we can walk over to their offices and meet with them if needed – there is something to be said for working with local vendors. Server Beach has been a pleasure to work with thus far and I have some experience working with them in the past. Peer1′s VP of Marketing also blogs on WordPress.com. Pricing and server configurations are almost identical, so that really isn’t as much of a factor as one would think.

Anyone have experience working with either of these companies? Suggestions? Feedback?

• 80 physical processors
• 139GB of RAM
• 91 hard drives with a combined total of over 15 terabytes of storage space
• 2000 database queries per second spread over 40 MySQL instances
• Over 8 million objects stored in memcached serving over 8000 requests per second

The best part is that this is just the beginning! There are many more exciting things to come. Next up — expansion to 2 additional datacenters in the US.