Load Balancer Update
April 28, 2008A while back, I posted about some testing we were doing of various software load balancers for WordPress.com. We chose to use Pound and have been using it past 2-ish years. We started to run into some issues, however, so we starting looking elsewhere. Some of these problems were:
- Lack of true configuration reload support made managing our 20+ load balancers cumbersome. We had a solution (hack) in place, but it was getting to be a pain.
- When something would break on the backend and cause 20-50k connections to pile up, the thread creation would cause huge load spikes and sometimes render the servers useless.
- As we started to push 700-1000 requests per second per load balancer, it seemed things started to slow down. Hard to get quantitative data on this because page load times are dependent on so many things.
So… A couple weeks ago we finished converting all our load balancers to Nginx. We have been using Nginx for Gravatar for a few months and have been impressed by its performance, so moving WordPress.com over was the obvious next step. Here is a graph that shows CPU usage before and after the switch. Pretty impressive!
Before choosing nginx, we looked at HAProxy, Perlbal, and LVS. Here are some of the reasons we chose Nginx:
- Easy and flexible configuration (true config “reload” support has made my life easier)
- Can also be used as a web server, which allows us to simplify our software stack (we are not using nginx as a web server currently, but may switch at some point).
- Only software we tested which could handle 8000 (live traffic, not benchmark) requests/second on a single server
We are currently using Nginx 0.6.29 with the upstream hash module which gives us the static hashing we need to proxy to varnish. We are regularly serving about 8-9k requests/second and about 1.2Gbit/sec through a few Nginx instances and have plenty of room to grow!
Barry -
One question: How are you guys doing failover with the nginx box? Nginx has been on my list of things to look at, but so far no time in the R&D bank.
Mike
Mike,
We are using Wackamole and Spread which is the same way we did it when we were running Pound. Works great!
Thanks for the rundown Barry. We’re about to go live with nginx in a similar role, it’s a really nice piece of software. We’ve got it behind ipvs / keepalived to handle simple layer 4 load balancing and failover, the combination works well.
Have you seen any issues with ssl or ssl+gzip? This seems to be an area where 0.5 and 0.6 have both had a few bugs recently — and something that seems not too easy to exercise without real traffic. Thanks!
James,
Awesome. We haven’t seen any issues with ssl or ssl+gzip. Our SSL traffic volume is pretty low, however.
[...] глобально в Алексе, и 26-ой сайт в США, что больше LiveJournal) перевел все load-balancer’ы на nginx на прошлой неделе. « Microsoft передумала в плане [...]
We’ve been using nginx for a while now (for about 2-3 years) and the performance is just impressive.
wordpress.com использует nginx…
WordPress.com перешел на использование nginx на своих load-balancer-а…
[...] перевел свои балансировщики нагрузки на nginx - очень серьезный [...]
What was the +/- versus HAProxy? We haven’t used either but am very curious!
Thanks.
[...] After two years of using Pound, WordPress decided to switch to Nginx as software load balancers for WordPress.com. Read the full story on Barry’s Blog. [...]
From everything I’m reading, there’s not many reasons *not* to switch to nginx. I’m building my network with it starting out, so I can use its various capabilities in the future. What kind of load balancing does it do? It has built in round-robin, with a weight measurement, right? It doesn’t have anything to check the upstream servers’ health as far as I know. I’m esp interested in the static gzip module and passing things off to Varnish - can you explain more how those tie together? I assume Varnish is upstream from the nginx load balancer?
Thanks
What’s the max *safe* amount of traffic a single Nginx instance is load balancing for you and how much memory and cpu are being utilized?
[...] Vor kurzem hat auch WordPress mitgeteilt, nun benutzt WP Nginx in eigenen Proxi-Server. Er ist der Einzige, der fähig ist 8000 Anfragen in die Sekunde zu bearbeiten [...]