My friend Jason is blogging about Joyents grids and linked to one of their new whitepapers. The cost structures in the whitepaper were interesting to me, to give the executive summary for $2425 a month they got in aggregate 12 GB of RAM, 130GB of storage, and 750GB of public transfer. Some of the storage was block, and some was “distributed NFS mounted.” In systems terms, this came down to 800/mo for Big-IP load balancing, 1000/mo for 8 web servers, and 500/mo for 2 DBs.

Since a lot of startups ask me for hardware advice, I was curious this setup would compare to one of the better dedicated providers, LayeredTech. As disclosure, Automattic currently has 8 servers with Joyent that we got when WP.com first started (and they were called Textdrive) and we have 50+ with Layered Tech. Both are active, we serve blog traffic from Dallas (LT) and the main site/tags traffic from San Diego (Joyent).

However this raises an interesting question: If we had $2425/mo to burn and were just starting out, which would be a better choice?

We have the Joyent costs laid out there in a typographically-correct whitepaper, so I decided to dig into LayeredTech’s slightly-1999 website to see what would be comprable. First I started with the web nodes, their best deal currently seem to be the AMD 275 dual core with 2GB RAM, 250gb HD for 175/mo. Let’s get 10 of those. Let’s get two more but with dual 73gb SCSI mirrored drives for the DBs, at 190/mo. We need a file server, something CPU light, I went for this one and put two mirrored 500gb drives in it for 159/mo. Finally we need some load balancers. Two million hits a day is only 23 requests a second, which my laptop couuld probably do, but let’s get two 3800s at 109/mo each and we can put Pound + Wackamole on them for high availability. Those can easily balance probably about 30-40mbps of traffic each.

So far we’re at $2507/mo, but since we have 15 servers we can get a 10% reseller discount, bringing our final cost to $2256/mo using just the pricing on their website. What are you getting in aggregate for that? 27GB of RAM, 500GB of storage via NFS, 15 CPUs (most dual-core), and 18 TB (!) of public bandwidth. With all that extra RAM you could probably do some nice caching using something like Memcached or using MogileFS you could maybe utilize a good chunk of the 2.4 terabytes of excess storage you have on each of the web nodes. We’ve also saved $169/mo, though that isn’t that big a difference, and honestly both Joyent and LayeredTech could probably do better prices than what they publish.

What did we lose? All of this is probably on much cheaper hardware than the Suns that Joyent uses, which may mean it’ll fail more often, however since we have redundancy and failover at every layer this shouldn’t matter. The load balancers are more than enough for our load, but they’re not at all in the class of Big-IP load balancers that probably cost $100k+ each. Scaling Joyent is presumably paying for more resources on existing hardware, so transparent, while scaling at LT would involve buying and configuring additional hardware. The servers are unmanaged, so you’ll need someone to setup and maintain them. (In the whitepaper the company was paying Joyent $1200/mo for this.) The Joyent processors are faster, but that may be offset by the additional boxes we have in the LayeredTech setup. The only way to know which is faster is to benchmark each, which is beyond the time/money I wanted to invest in this blog post.

This isn’t meant to be an endorsement of one direction or the other. There are a thousand other things outside of what I’ve talked about in this post when considering a hosting provider to spend thousands of dollars a month with, not the least of which is their support, network, datacenter history, size, relationship with you, and much more. However after evaluating a number of “grid” solutions recently, I don’t think they really provide a strong cost benefit for technically advanced companies, it’s mostly convinence and marketing.

What do you guys think? Anyone want to work out what the same would cost on Amazon? Which would you go with?

Update: HostingFu runs the numbers on Amazon.