Load Balancer Update
April 28, 2008A while back, I posted about some testing we were doing of various software load balancers for WordPress.com. We chose to use Pound and have been using it past 2-ish years. We started to run into some issues, however, so we starting looking elsewhere. Some of these problems were:
- Lack of true configuration reload support made managing our 20+ load balancers cumbersome. We had a solution (hack) in place, but it was getting to be a pain.
- When something would break on the backend and cause 20-50k connections to pile up, the thread creation would cause huge load spikes and sometimes render the servers useless.
- As we started to push 700-1000 requests per second per load balancer, it seemed things started to slow down. Hard to get quantitative data on this because page load times are dependent on so many things.
So… A couple weeks ago we finished converting all our load balancers to Nginx. We have been using Nginx for Gravatar for a few months and have been impressed by its performance, so moving WordPress.com over was the obvious next step. Here is a graph that shows CPU usage before and after the switch. Pretty impressive!
Before choosing nginx, we looked at HAProxy, Perlbal, and LVS. Here are some of the reasons we chose Nginx:
- Easy and flexible configuration (true config “reload” support has made my life easier)
- Can also be used as a web server, which allows us to simplify our software stack (we are not using nginx as a web server currently, but may switch at some point).
- Only software we tested which could handle 8000 (live traffic, not benchmark) requests/second on a single server
We are currently using Nginx 0.6.29 with the upstream hash module which gives us the static hashing we need to proxy to varnish. We are regularly serving about 8-9k requests/second and about 1.2Gbit/sec through a few Nginx instances and have plenty of room to grow!
Barry -
One question: How are you guys doing failover with the nginx box? Nginx has been on my list of things to look at, but so far no time in the R&D bank.
Mike
Mike,
We are using Wackamole and Spread which is the same way we did it when we were running Pound. Works great!
Thanks for the rundown Barry. We’re about to go live with nginx in a similar role, it’s a really nice piece of software. We’ve got it behind ipvs / keepalived to handle simple layer 4 load balancing and failover, the combination works well.
Have you seen any issues with ssl or ssl+gzip? This seems to be an area where 0.5 and 0.6 have both had a few bugs recently — and something that seems not too easy to exercise without real traffic. Thanks!
James,
Awesome. We haven’t seen any issues with ssl or ssl+gzip. Our SSL traffic volume is pretty low, however.
[...] глобально в Алексе, и 26-ой сайт в США, что больше LiveJournal) перевел все load-balancer’ы на nginx на прошлой неделе. « Microsoft передумала в плане [...]
We’ve been using nginx for a while now (for about 2-3 years) and the performance is just impressive.
wordpress.com использует nginx…
WordPress.com перешел на использование nginx на своих load-balancer-а…
[...] перевел свои балансировщики нагрузки на nginx – очень серьезный [...]
What was the +/- versus HAProxy? We haven’t used either but am very curious!
Thanks.
[...] After two years of using Pound, WordPress decided to switch to Nginx as software load balancers for WordPress.com. Read the full story on Barry’s Blog. [...]
From everything I’m reading, there’s not many reasons *not* to switch to nginx. I’m building my network with it starting out, so I can use its various capabilities in the future. What kind of load balancing does it do? It has built in round-robin, with a weight measurement, right? It doesn’t have anything to check the upstream servers’ health as far as I know. I’m esp interested in the static gzip module and passing things off to Varnish – can you explain more how those tie together? I assume Varnish is upstream from the nginx load balancer?
Thanks
Nginx offers weighed round robin and IP-hash based sticky sessions by default. It also does health checks.
See: http://wiki.codemongers.com/NginxHttpUpstreamModule
We do not serve text via varnish, just images and video, so I don’t have any input as to how the gzip functionality would work in that case.
What’s the max *safe* amount of traffic a single Nginx instance is load balancing for you and how much memory and cpu are being utilized?
We have tested it up to about 10k req/sec. Memory footprint is minimal, and Nginx doesn’t use much CPU time. Where you end up with problems is in the TCP overhead and the time spent handling software interrupts. It gets much worse with iptables and connection tracking. Performance here is probably better on FreeBSD than Linux (we run Linux), but I haven’t tested it.
[...] Vor kurzem hat auch WordPress mitgeteilt, nun benutzt WP Nginx in eigenen Proxi-Server. Er ist der Einzige, der fähig ist 8000 Anfragen in die Sekunde zu bearbeiten [...]
[...] from WrdPress is explaining how they setup nginx into their 300 servers cluster, for their 2 main sites (wordpress.com and [...]
Hi. Nice article – thanks. Not sure if I understand where varnish fits in. Does it work like this?
Internet Client –> Varnish(s) –> NGINX(s) –> Webservers
Our setup is:
Client –> Nginx –> (Varnish|Webserver) –> [Webserver]
Depending on the request type some requests are then passed to Varnish and others are sent directly to the web servers. We currently use Varnish only to serve on static images and video content (reverse caching proxy to Amazon’s S3).
[...] From 0:00 CET orion.janforman.com is running nginx russian webserver with apache backend
this result in much faster transfers and better memory usage. Some testimonials from wordpress [...]
Barry,
This is a follow up to Mike’s question on 4/28 about the failover configuration of nginx. We are specifically interested in understanding if and how nginx can be configured for a traditional active/active failover pair. We want to know if nginx supports state sharing between the failover pair so as to maintain continuation of service for such features as server affinity.
Any light and/or guidance you can share is greatly appreciated.
Matthew
Hi, you said ..
“Only software we tested which could handle 8000… ”
so you mean LVS kind of kernel level load balancing is even slower than ngnix?
Memory footprint is minimal, and Nginx doesn’t use much CPU time in me too…
Wackamole would be a decent replacement for heartbeat for managing IPs it sounds like.
So in theory you could use Wackamole+nginx for Active/Passive(+more) nginx instances and Wackamole would handle all the IP switching and skip using LVS/ldirectord|keepalived/heartbeat, right?
[...] more than a million sites; more than doubling in numbers. The WordPress blogging system recently converted all of its load balancers to nginx, using the upstream hash module to serve 8-9 thousand requests [...]
Barry – I appreciate your answers, having real-world examples of nginx and varnish give us the answers we need – this is a great resource.
[...] from Jamie’s pointer, in doing the initial research, what got me excited was reading that WordPress.com had switched to nginx for their load balancing (and might eventually switch for their web serving as well), and that Fastmail is using nginx for [...]
[...] more than a million sites; more than doubling in numbers. The WordPress blogging system recently converted all of its load balancers to nginx, using the upstream hash module to serve 8-9 thousand requests [...]
[...] more than a million sites; more than doubling in numbers. The WordPress blogging system recently converted all of its load balancers to nginx, using the upstream hash module to serve 8-9 thousand requests [...]
Ok, here’s a silly question. Are you using nginx on the backend web servers too?
hi,
we plan a website with around 10000-50000 online users.
we plan to use nginx as a loadbalancer and will have the webservers within an internal ip-network.
my question is: if the nginx LB has to route+NAT all the users to the internal webservers, how much load will that make on the nginx server? Is it possible?
Thank u very much for your help!
[...] baca postingan Barry tentang migrasi wordpress ke nginx ( jadi balancernya kl gak salah) saya jadi penasaran untuk coba mencicip engine x (nginx). [...]
[...] nginx, a lightweight HTTP server. This is the softwareWordPress.com uses as a load balancer: We have been using Nginx for Gravatar for a few months and have been impressed by its [...]
Hi,
Really interesting post. I also like to know like Matthew Porter +/- against HAProxy. HAProxy as i know also supports hot-reconfiguration and can take pretty heavy load.
And another thing thats really interesting is how many servers you need to server that kind of traffic. Especially how many you need as proxy servers?
Thanks for sharing!!
I just read the post again and found an answer to my own question…
You have 20+ loadbalancers…
That raises another question.. how to you spread the load over the load balancers
/Mathias
[...] WordPress (eh già credevate che usassero Apache), YouPorn, FastMail.FM, Kongregate, Hulu, provate infatti a vedere le risposte http da questi siti. [...]
@Mathias – reading this document, looks like nginx is only able to do simple round-robin.
.
Nothing fancy yet such as like intelligent request queueing with HAproxy.
.
Also I noticed that HAproxy can handle 34000+ connections per second, as shown in this page. This is well beyond WP.com’s 10000 conn/sec.
.
Could the lack of performance of HAproxy in this post be explained by the request queue bug? Which has since been fixed after the publication of this post. The test on the new version of HAproxy shows it beating nginx, cpu-load wise.
No bench on connection/sec though, so it may be completely irrelevant, but still, it might be of interest.
.
Hopefully we’ll be able to find out even more on these great pieces of software.
.
Thanks.
how to you spread the load over the load balancers
I’m curious about that, too… seems like the only real option at that layer is RR DNS?
I cant believe your comment that nginx was the only solution that could reach 8000 cons/sec. HaProxy (latest) I’ve had doing full cookie inserts at 27,000 cons/sec. A graph here compares connections/sec on the Kemp 1500 and Loadbalancer.org R16 which are both based on LVS here http://www.loadbalancer.org/whyr16.html (we also use Pound & Haproxy). Blatant commercial link but still relevant.
and what are you using to route and switch 1.2Gbit/sec?
[...] which were delivering static assets like CSS, JavaScript and (some) image files. Recently the WordPress.com load balancers were upgraded to nginx and since then nginx has been proving to be a very high performance piece of software, with some [...]
I’m just learning about load balancing WordPress. I was wondering how the load balancing deal with mysql and how data would replicate between the different servers.
The idea is to have 2 data centers, each data center would have 1 load balancers, 2 web servers and 1 mysql server.
Scott.
Very impressive, will nginx work with Joomla??
[...] what I have heard, it’s capable to run quite high load on very low requirements. Regarding to this article, WordPress.com is using Nginx as load balancer, handling over 8000 requests / sec. [...]
industryfinest nginx work with Joomla!
Barry,
Thanks, though I found this post a bit late, it saved my job. We have decided to port our latest word press news site to nginx. We are already getting 10K hits per day, and expect around 50K once new features and channels are added..
I’m definately going to download it now.. I’ve been looking for a small load-balancing solution myself for along time, I just with it was’t all russian documentation…
Also give crossroads a try (crossroads.e-tunity.com). It has a very small footprint but still a lot of powerful features like access control and dos prevention.
[...] what you’re probably thinking of is WordPress.com recently switching to using Nginx as a frontend load-balancing HTTP proxy or to serve static images and files instead of Lighttpd. Those are both excellent use cases for [...]
[...] oh yeah, forgot to mention wordpress uses it: barry.wordpress.com/2008/04/28/load-balancer-update/ [...]
Красивый блог у тебя! Особенно падающий снег!
[...] Load Balancer Update « Barry on WordPress (tags: architecture wordpress varnish nginx) This entry was written by bairos, posted on December 23, 2008 at 1:30 am, filed under delicious-daily. Bookmark the permalink. Follow any comments here with the RSS feed for this post. Post a comment or leave a trackback: Trackback URL. « links for 2008-12-21 [...]
Тут уже столько напостили до меня. Остается только присоединиться.
What was the +/- versus HAProxy? We haven’t used either but am very curious!
nice thanx you
[...] sobre o mesmo assunto, me lembro de ter lido no blog do Barry, engenheiro do wordpress.com, um artigo sobre a atualização do load-balancer que usam para distribuir a carga de acesso ao site entre os diversos [...]
[...] de 2009 ganhou mais 1270000 sites, o nginx é usado por exemplo no excelente site de blogging, o wordpress, do qual faz parte este blog, bem como no FastMailFM, só em Dezembro de 2008 estava a ser usado em [...]
[...] beban yang sesuai, kami ada juga consider tentang LVS, Pound dan seangkatan dengannya. Tetapi pos dari wordpress tentang nginx ini membuatkan kami ingin mencuba nginx. Dah cuba, kami mula menyukainya, hohoho [...]
[...] nginx has been running for more than four years on many heavily loaded Russian sites including Rambler (RamblerMedia.com). In March 2007 about 20% of all Russian virtual hosts were served or proxied by nginx. According to Google Online Security Blog year ago nginx served or proxied about 4% of all Internet virtual hosts, although Netcraft showed much less percent. According to Netcraft in March 2008 nginx served or proxied 1 million virtual hosts. The growing in picture and colour! According to Netcraft in December 2008 nginx served or proxied 3.5 millions virtual hosts. And now it is on 3rd place and ahead of lighttpd. According to Netcraft in March 2009 nginx served or proxied 3.06% busiest sites. 2 of Alexa US Top100 sites use nginx. Here are some of success stories: FastMail.FM, WordPress.com. [...]
Script for measurte cpu usage write by hand or buy?
They are generated using Munin which is open source software.
Yes,
Really interesting poster. I m also like to know liked Matthew Porter +/- against HAProxy. HAProxy as i know also supports hot-reconfiguration and can take pretty heavy loades.
Have you seen any issues with ssl or ssl+gzip? This seems to be an area where 0.5 and 0.6 have both had a few bugs recently?