Category Archives: Automattic

The company behind WordPress.com, Tumblr, WooCommerce, Pocket Casts, Day One, Beeper, and more.

Sun Isn’t Relevant to Startups

Preface: I don’t write critically about a company unless I think they have some glimmer of getting better, however small that may be.

I just watched another Scoble Show interview with Jonathan Schwartz, and it reminded me how frustrated I continue to be with Sun, particularly their Startup Essentials program. He seems to be complaining about the perception of startups that the Sun platform is completely wrong for a new generation of startups, a perception I would strongly agree with.

To illustrate, here is a timeline of my experience with the Startup Essentials program.

November 3rd, 2006: Drinking the kool-aid at Startup Camp, register on behalf of Automattic for the Startup Essentials program.

November 8th, 2006: I receive an email addressed to “Mr Muellenweg” (wtf?) saying “Due to the volume of interest in this exciting new program, it is taking us longer than expected to review your application. We will let you know your status within the next few days — thanks for your patience.”

Late-November: I talk for about an hour to a nice guy named Eric Cosway doing research about what I thought of Startup Camp and Sun. I tell him about the trouble I’ve had registering for Startup Essentials, he says he’ll note that and pass it on.

3 months since: Complete silence, punctuated by frequent Sun press releases and AP stories about how they’re great for startups, probably millions of dollars of PR.

Seeing their constant press releases and hot-air filled announcements around “Web 2.0” just makes Sun seem that much more out of touch with reality and desparate to jump on hype which has already passed.

I’m not writing this, as Joyent did, with the hope of connecting with someone at Sun that will take my money. I’ve given up, I’ve lost hope, and our business has moved on. We’re growing fast and adding infrastructure faster, all on a tight budget, and I’m less inclined to depart from the LAMP-based architecture that Digg, Flickr, Wikipedia, and dozens of other sites I respect use.

This is also what I’ve started advising other startups, if we had trouble getting anything going I can’t imagine what trouble a brand new startup with a smaller profile would go through. (An exception to this is going through Joyent, which has packaged Sun’s stuff so well Sun should just buy them.)

Personally I’m far more excited about what Amazon is doing these days.

Update: In a move most would never expect of a public company CEO, Jonathan Schwartz has responded. More than that, he has apologized in a human and personal way that is utterly admirable. I’m going to work on a followup to this entry.

Update 2: I’ve posted the followup.

Economical Grids?

My friend Jason is blogging about Joyents grids and linked to one of their new whitepapers. The cost structures in the whitepaper were interesting to me, to give the executive summary for $2425 a month they got in aggregate 12 GB of RAM, 130GB of storage, and 750GB of public transfer. Some of the storage was block, and some was “distributed NFS mounted.” In systems terms, this came down to 800/mo for Big-IP load balancing, 1000/mo for 8 web servers, and 500/mo for 2 DBs.

Since a lot of startups ask me for hardware advice, I was curious this setup would compare to one of the better dedicated providers, LayeredTech. As disclosure, Automattic currently has 8 servers with Joyent that we got when WP.com first started (and they were called Textdrive) and we have 50+ with Layered Tech. Both are active, we serve blog traffic from Dallas (LT) and the main site/tags traffic from San Diego (Joyent).

However this raises an interesting question: If we had $2425/mo to burn and were just starting out, which would be a better choice?

We have the Joyent costs laid out there in a typographically-correct whitepaper, so I decided to dig into LayeredTech’s slightly-1999 website to see what would be comprable. First I started with the web nodes, their best deal currently seem to be the AMD 275 dual core with 2GB RAM, 250gb HD for 175/mo. Let’s get 10 of those. Let’s get two more but with dual 73gb SCSI mirrored drives for the DBs, at 190/mo. We need a file server, something CPU light, I went for this one and put two mirrored 500gb drives in it for 159/mo. Finally we need some load balancers. Two million hits a day is only 23 requests a second, which my laptop couuld probably do, but let’s get two 3800s at 109/mo each and we can put Pound + Wackamole on them for high availability. Those can easily balance probably about 30-40mbps of traffic each.

Continue reading Economical Grids?

New Servers

We’re looking to expand into a third datacenter for WordPress.com and related Automattic stuff. I’ve been asking around for quotes, but it occurred to me I should blog what we’re looking for because someone reading might be able to provide what we need, or know someone who could. One line summary of what we’re looking for: 22 boxes, AMD 3800 or similar, cheapest HD, 1-2gb of RAM, Debian AMD64 (this is important), private and public 100mbps network. I’d be happy to post more info in the comments if anyone wants it. A per-server monthly price including space and bandwidth is what we’re looking for.

Web Scaling Books

I’ve read two really good books on scaling large-scale web applications lately: Building Scalable Websites by Cal Henderson and Scalable Internet Architectures by Theo Schlossnagle. (Original titles, eh?) Both dispel common myths and misconceptions about scaling. While neither maps directly to the approaches we’re taking at Automattic, they’re both must-read for developers approaching or past the million/day pageview mark. I only wish I had more time to discuss my thoughts on the various concepts in the books—particularly Cal’s.

Paid Support

We just launched the Automattic Support Network which is a place for companies to purchase paid support for WordPress and MU. Originally I didn't think we'd need to do this, simply because the WordPress.org support forums are so amazing and there is such a good community around it. That hasn't changed, but some big companies and enterprise folks are uncomfortable with volunteer support, and want (and insist) on paying someone before deploying a product. Based on that feedback and a lot of input from Podz, we put together this new product, which is basically VIP support with a guaranteed response time. Toni has some more thoughts here. We also rolled out new pricing for commercial Akismet use a few days ago, and the response has been great so far.

Podsession Responses

There have been two interesting responses to the podcast I did with Om and Niall the other week. The first was Scott Johnson who responded in a podcast. As I expected, most people are taken aback by my statement to “let the engineers pick” what language and enviroment you use for your product. I think there is one important assumption that wasn’t articulated in that statement: you have brilliant engineers and you trust them. As a psuedo-engineer, I find it insulting when people suggest engineers are unable to factor anything other than their selfish language preferences, things like loaded costs, hardware costs, platforms, long-term viability, hiring, etc are simple variables that can be considered by any intelligent person. If anybody in Automattic came to me that was writing a tool in Python, C, Perl (it’s happened) or whatever, I might ask a question or two but at the end of the day I know they’re able to weigh the costs and benefits just like I would. If you’ve hired an engineer that isn’t able to make these decisions as well or better than you, then you’ve already lost the battle and over time more and more of your time will be spent plugging holes in a descent to mediocrity.

The second response was on the Pronet blog which in an amazing feat of blogging acrobatics managed to mention and link every single person tangentially associated with the podcast except me, even though I’m quoted in every heading. The Google Pages example is brought up again to illustrate that all the hardware in the world sometimes solve a scalability problem, but I still think that’s faulty because none of us had any idea why Pages was slow when it launched, it could have been a faulty router for all we know. Pronet responds to “Go with what your happiest working with” with a set of points to consider for a language, but again with the right people none of that matters. Happiness, in all things not just the language, should be the number goal and metric for everything in an early-stage startup. Happy engineers work smarter, longer, more efficient, attract better candidates, and have a better quality of life. (A corollary is that if you’re already set on a language path, don’t hire anyone who isn’t thrilled with working in that language.) For an example of how this can work in a really extreme case, I suggest everyone read the story of Viaweb and Lisp. (Another talk.)In my mind Lisp is a ridiculous language to build a web application in, but to them and their engineers it was heaven and they had better products earlier than their competition as a result of their unusual choice.

(As an aside, I wonder how many people said the same thing about Ruby for web apps before David Heinemeier Hansson, Rails, and 37signals, or even about PHP before Yahoo and Wikipedia? An example (and a little bit of promotion) is better than a thousand whitepapers.)

A Little Funding

The best thing that can ever happen to a web service is to have passionate users. Users that notice and email you the second there’s a database problem, users that really push the limits of what you can provide, and users that are phenomally successful and bring thousands of others to your doors.

As a service provider, you have a strong responsibility to these folks. They’re putting their life online with you, they deserve nothing less than 100% uptime. They tell all their friends to try you out, they deserve for the experience of the hundred thousandth user to be as great as the tenth. WordPress.com is serving 4.2 million hits a day on a handful of boxes. Akismet has gotten to the point where it’s blocking so many spams every second that any fraction of downtime is very noticable to users. (Like we had this morning.)

At Automattic we’ve always taken this very seriously, and from the bootstrap beginning I planned for it to be sustainable and frugal in the long term. Of course since I moved to San Francisco I’ve talked to dozens of really high-quality investors who were interested in what we were doing, but the bubble model of giant valuations and ultra-rapid growth never really appealed to me.

The growth of WordPress.com and Akismet has outpaced anyone’s expectations. Recently, I made the decision to sell a minority stake in the company to a few select partners who I think are going to bring a lot of value to the business far beyond mere dollars. This isn’t going to change how the business is run, or the people involved with it, but it will allow us to take better advantage of the opportunities before us and also for us to keep our promise to every one of you to maintain a fast, stable, and innovative platform in the long term.

Automattic isn’t going to get fancy SoMA offices, throw huge parties at SxSW, or “get big fast.” We took a small amount of capital to put things that were already growing fast in a stable position, so from month to month you’re not robbing Peter to pay Paul. We’re going to use the money to pre-emptively address scaling issues before they happen, and continue to share everything we can back to the community, like all of the code behind WP.com in WordPress MU, the spellchecking feature we sponsored, free Akismet for 99.9% of users, and a few other goodies we still have up our sleeve. In terms of hiring, we’re still going to grow very deliberately in line with our revenues and focus on the very best and brightest (and BBQ-loving), like Podz.

We’re going to publish more technical details about everything later, and this is already longer than I hoped — I’m sure you folks have some questions. I’m going to do something a little different and turn the comment section here into a FAQ. If you have a question, please post it below. If you want to say “congrats!” or “that sucks!” do it on this entry instead to keep the question and answer flow clean. If a question warrants a long enough answer I might turn it into a separate blog post.

Open Source Legal Docs

Not technically open source, because I don't know which license is best for regular text, but I just put a Creative Commons Sharealike license on the WordPress.com terms of service and Automattic privacy policy. People were stealing them anyway, might as well make it legit. πŸ™‚ Feel free to grab bits and pieces and search/replace your company/project in. If you want to throw us a link as a thank you, I'd be flattered.

Hours and Work

There’s no correlation between hours worked and work done. I think this is why traditional corporate structures are starting to crack at the seams, and the distributed model companies like Automattic, MySQL, SocialText, and many others use will start to gain real legs and acceptance. The best example of this was at a place I used to work: after lunch everything seemed to shut down. Several people obviously got very sleepy after lunch and would spend 2-4 hours of the afternoon on auto-pilot. (This was me sometimes too.) It would have been infinitely better for them to take a one hour nap and get back to productive work than spend 3 hours in an obviously hampered state. Happy, healthy, well-rested people work orders of magnitude better.