My friend Jason is blogging about Joyents grids and linked to one of their new whitepapers. The cost structures in the whitepaper were interesting to me, to give the executive summary for $2425 a month they got in aggregate 12 GB of RAM, 130GB of storage, and 750GB of public transfer. Some of the storage was block, and some was “distributed NFS mounted.” In systems terms, this came down to 800/mo for Big-IP load balancing, 1000/mo for 8 web servers, and 500/mo for 2 DBs.
Since a lot of startups ask me for hardware advice, I was curious this setup would compare to one of the better dedicated providers, LayeredTech. As disclosure, Automattic currently has 8 servers with Joyent that we got when WP.com first started (and they were called Textdrive) and we have 50+ with Layered Tech. Both are active, we serve blog traffic from Dallas (LT) and the main site/tags traffic from San Diego (Joyent).
However this raises an interesting question: If we had $2425/mo to burn and were just starting out, which would be a better choice?
We have the Joyent costs laid out there in a typographically-correct whitepaper, so I decided to dig into LayeredTech’s slightly-1999 website to see what would be comprable. First I started with the web nodes, their best deal currently seem to be the AMD 275 dual core with 2GB RAM, 250gb HD for 175/mo. Let’s get 10 of those. Let’s get two more but with dual 73gb SCSI mirrored drives for the DBs, at 190/mo. We need a file server, something CPU light, I went for this one and put two mirrored 500gb drives in it for 159/mo. Finally we need some load balancers. Two million hits a day is only 23 requests a second, which my laptop couuld probably do, but let’s get two 3800s at 109/mo each and we can put Pound + Wackamole on them for high availability. Those can easily balance probably about 30-40mbps of traffic each.
So far we’re at $2507/mo, but since we have 15 servers we can get a 10% reseller discount, bringing our final cost to $2256/mo using just the pricing on their website. What are you getting in aggregate for that? 27GB of RAM, 500GB of storage via NFS, 15 CPUs (most dual-core), and 18 TB (!) of public bandwidth. With all that extra RAM you could probably do some nice caching using something like Memcached or using MogileFS you could maybe utilize a good chunk of the 2.4 terabytes of excess storage you have on each of the web nodes. We’ve also saved $169/mo, though that isn’t that big a difference, and honestly both Joyent and LayeredTech could probably do better prices than what they publish.
What did we lose? All of this is probably on much cheaper hardware than the Suns that Joyent uses, which may mean it’ll fail more often, however since we have redundancy and failover at every layer this shouldn’t matter. The load balancers are more than enough for our load, but they’re not at all in the class of Big-IP load balancers that probably cost $100k+ each. Scaling Joyent is presumably paying for more resources on existing hardware, so transparent, while scaling at LT would involve buying and configuring additional hardware. The servers are unmanaged, so you’ll need someone to setup and maintain them. (In the whitepaper the company was paying Joyent $1200/mo for this.) The Joyent processors are faster, but that may be offset by the additional boxes we have in the LayeredTech setup. The only way to know which is faster is to benchmark each, which is beyond the time/money I wanted to invest in this blog post.
This isn’t meant to be an endorsement of one direction or the other. There are a thousand other things outside of what I’ve talked about in this post when considering a hosting provider to spend thousands of dollars a month with, not the least of which is their support, network, datacenter history, size, relationship with you, and much more. However after evaluating a number of “grid” solutions recently, I don’t think they really provide a strong cost benefit for technically advanced companies, it’s mostly convinence and marketing.
What do you guys think? Anyone want to work out what the same would cost on Amazon? Which would you go with?
Update: HostingFu runs the numbers on Amazon.
12 replies on “Economical Grids?”
That must be some laptop.
Not really, but Pound is really light and fast. 🙂
What about differences in management costs? How many people are you going to need to manage the one versus the other?
Recently, the company I work at has grew from a single LAMP server to multiple Windows 2003/LAMP servers in multiple hosting locations.
Software updates, patches, new modules/libraries etc. all take a whole lot more coordination and time these days simply because it needs to be replicated to more than a couple of machines… Sounds like that $1200 a month is worth it unless you feel the need to get your own people in on the (hardware) management side of things.
Of course, we’ve not had hardware failures … yet.
Wow! An impressive post with lots of figures and thought. I hope your brain didn’t set off any smoke detectors!?
Seriously though, good stuff. It’s not often I get to read about the actual day-to-day running costs of such enterprises as WordPress.com. I found it really quite interesting and informative in a geeky way 🙂 Thanks.
“That must be some laptop.”
Nah…. Matt just has HUGE “lap” if you get my drift. wink wink.
Why not perlbal btw? You guys just couldn’t get it compiled?
Pound is thread based right? Ug
[…] Economical Grids? […]
Great article! Submitted to my friends!
[…] Photo Matt Â» Economical Grids? wherein Matt challenges the economics of grids; interesting math from someone who obviously has experience with the subject (tags: Mullenweg grids economics Joyent Amazon virtualization) […]
[…] Economical Grids? – great post from Matt, I’m curious how the costs go up at 10x or 100x the scale. My understanding is that Grids can scale up larger a little easier. I could be wrong though. […]
27Gb of RAM? Man… *Jaw dropped down* Wow… Crazy! I have no more comments, LOL 😀
I agree with Kevin F — at the $2425/mo scale the variation in labor costs is more significant than the variation in hosting costs. And from day one it’s usually hard to know which resources will be bottlenecks (e.g., when you’re pushing 10TB/mo, how much CPU, memory and disk I/O will you need).
I would prioritize manageability (whatever that means for your environment) and flexibility over savings on servers and networks until hosting costs are much larger. Whether this argues for or against using a “grid” is hard to say and varies depending on the app and what in-house expertise you have.