So I went to the NY MySQL meetup. It was pretty interesting to hear first-hand the thoughts of someone using MySQL in a heavy trafficked environment. They’re essentially running huffpo on a heavily customized movabletype system (php) on 14 front end web servers, 4 memcached servers and 7 mysql servers. They couldn’t move to MySQL on ec2 because the max memory limit for their big servers were too small (they must have some heavy hardware going).
OK, so onto the interesting stuff:
They have a dual master setup, while running 5 slaves using MySQL 5.0.x and apparently almost all tables are on InnoDB. The second master was primarily for a hot backup of the site. They’ve setup memcache to take load off the mysql servers, and interestingly enough, they’ve moved session storage actually into memcache instead of mysql. Their reads come from the slaves, but HuffPO CTO said specifically that it’s only for stuff that can be a few seconds behind the prod server. Any sort of statistical stuff comes off the slaves as well. One interesting thing they noted was that at times of extreme load, they have the ability to turn on and turn off features. Also, they’ve mentioned that they add another level of cache upfront using Akamai to deliver dynamic pages.
They’re currently shifting to using ssd’s for their table storage and a new san for their log storage on a new hardware. They’ve also setup a load balanced vip for clustered access to the slaves. Overall I thought it was interesting to hear how they’ve setup their enviornment.
One thing they did note was that they had tried to use memcached on each of their 14 web servers but that they had synchronization issues across the servers, each cache would lose itself then force a rehash across all the servers. Instead of keeping it on 14 servers, they consolidated to run it on 4 servers, which upped the cache hit rate from the 25% that they were getting.
An interesting thing he did mention was that they give a potential hire a link to an ec2 image that need to use for a test. If they can’t pass that test then they’re most likely not technical enough to go on. Pretty great idea.
