Categories
Automattic WordPress

Dance to Calypso

One of the hardest things to do in technology is disrupt yourself.

But we’re trying our darndest, and have some cool news to introduce today. When I took on the responsibility of CEO of Automattic January of last year, we faced two huge problems: our growth was constrained by lack of capital, and the technological foundations of the past decade weren’t strong enough for the demands of next one.

The first has a relatively straightforward answer. We found some fantastic partners, agreed on a fair price, issued new equity in the company to raise $160M, and started investing in areas we felt were high potential, like this year’s WooCommerce acquisition. This “war chest” gives us a huge array of options, especially given our fairly flat burn rate — we don’t need to raise money again to keep the company going, and any capital we raise in the future will be purely discretionary. (Since last May when the round happened we’ve only spent $3M of the investment on opex.)

The second is much harder to address. The WordPress codebase is actually incredible in many ways — the result of many thousands of people collaborating over 13 years — but some of WordPress’ greatest strengths were also holding it back.

The WordPress codebase contains a sea of institutional knowledge and countless bug fixes. It handles hundreds of edge cases. Integrates constant security improvements. Is coded to scale. Development moves at a fast clip, with six major releases over the past two years and more around the corner. Its power and flexibility is undeniable: WordPress just passed a huge milestone, and now powers 25% of the web. You can run it on a $5-a-month web host, or scale it up to serve billions of pageviews on one of the largest sites on the web, WordPress.com.

The interface, however, has been a struggle. Many of us attempted to give it a reboot with the MP6 project and the version 3.8 release, but what that release made clear to me is that an incremental approach wouldn’t give us the improvements we needed, and that two of the things that helped make WordPress the strong, stable, powerful tool it is — backward compatibility and working without JavaScript — were actually holding it back.

The basic paradigms of wp-admin are largely the same as they were five years ago. Working within them had become limiting. The time seemed ripe for something new, something big… but if you’re going to break back compat, it needs to be for a really good reason. A 20x improvement, not a 2x. Most open source projects fade away rather than make evolutionary jumps.

So we asked ourselves a big question. What would we build if we were starting from scratch today, knowing all we’ve learned over the past 13 years of building WordPress? At the beginning of last year, we decided to start experimenting and see.

Today we’re announcing something brand new, a new approach to WordPress, and open sourcing the code behind it. The project, codenamed Calypso, is the culmination of more than 20 months of work by dozens of the most talented engineers and designers I’ve had the pleasure of working with (127 contributors with over 26,000 commits!).

gm-2015-final

Calypso is…

  • Incredibly fast. It’ll charm you.
  • Written purely in JavaScript, leveraging libraries like Node and React.
  • 100% API-powered. Those APIs are open, and now available to every developer in the world.
  • A great place to read, allowing you to follow sites across the web (even if they’re not using WordPress).
  • Social, with stats, likes, and notifications baked in.
  • Fully responsive. Make it small and put it in your sidebar, or go full-screen.
  • Really fun to write in, especially the drag-and-drop image uploads.
  • Fully multi-site for advanced users, so you can manage hundreds of WordPresses from one place.
  • Able to manage plugins and themes on Jetpack sites, including auto-upgrading them!
  • 100% open source, with all future development happening in the open.
  • Available for anyone to adapt to make their own, including building custom interfaces, distributions, or working with web services besides WordPress.com.

A lot of people thought we should keep this proprietary, but throughout my life I’ve learned that the more you give away, the more you get back. We still have a ton to figure out around plugins, extensibility, contributions, Windows and Linux releases, API speed, localization, and harmonizing the WordPress.com API and WP-API so it can work with core WordPress. Thousands more PHP developers will need to become fluent with JavaScript to recreate their admin interfaces in this fashion. I’m also really excited to revisit and redesign many more screens now that we have this first version out the door.

This is a beginning, not an ending. (1.0 is the loneliest.) Better things are yet to come, as all of you dig in. Check out these links to read more about Calypso from different perpsectives:

This was a huge bet, incredibly risky, and difficult to execute, but it paid off. Like any disruption it is uncomfortable, and I’m sure will be controversial in some circles. What the team has accomplished in such a short time is amazing, and I’m incredibly proud of everyone who has contributed and will contribute in the future. This is the most exciting project I’ve been involved with in my career.

With core WordPress on the server and Calypso as a client I think we have a good chance to bring another 25% of the web onto open source, making the web a more open place, and people’s lives more free.

If you’re curious more about the before and after, what’s changed, here’s a chart:

Whats-New-WPcom@2x

 

124 replies on “Dance to Calypso”

Currently WordPress.com runs a completely custom REST API as it heavily predates the one going into WordPress core. We’ll be working to migrate over to using the WordPress core functionality in the future and when we do, it’ll be more straight-forward figuring out what’s custom.

However many features and endpoints are WordPress.com-specific such as paid upgrades for example.

Getting Calypso to talk directly to a WP.org install would likely not be easy, even after we switch REST APIs.

Wow! Exciting news. What a great move forward for WordPress. If Automattic continues down this path, with enough new plugins, perhaps WordPress might someday provide the critical mass to define common metadata standards to support a widely-adopted “social semantic desktop”?

See for example: http://semanticweb.org/wiki/Semantic_Desktop

From there (source text under the GFDL): “The Internet, electronic mail, and the Web have revolutionized the way we communicate and collaborate – their mass adoption is one of the major technological success stories of the 20th century. We all are now much more connected, and in turn face new resulting problems: information overload caused by insufficient support for information organization and collaboration. For example, sending a single file to a mailing list multiplies the cognitive processing effort of filtering and organizing this file times the number of recipients – leading to more and more of peoples’ time going into information filtering and information management activities. There is a need for smarter and more fine-grained computer support for personal and networked information that has to blend the boundaries between personal and group data, while simultaneously safeguarding privacy and establishing and deploying trust among collaborators. …
Furthermore we are in danger of resulting in many fragmented efforts, each of which will not reach critical mass and thus will not be able to penetrate the user space wide enough to result in mass adoption. …”

I’ve just shared this news with our dev team at The Next Web. We were about to start building our latest design iteration the week, but this changes things 🙂 If it’s up to me (lead designer with some proficiency in Node.js/Meteor and React) we’ll build our redesign on Calypso instead, and bid our adieus to PHP for good.

WOW! This is all really exciting, WordPress now has a official take on Node.js, React and REST API. Thank you Matt for open sourcing it. I’m pretty sure developers will dig deep and try to contribute and create better UI/UX, at least I will :).

This is exciting news! Congrats! However, when I read the 11 items under “Calypso is”, I didn’t notice “accessible” as one of those awesome items. This troubles me, but I’m hoping to be troubled for absolutely no reason. After all, it’s being created in 2015, so I’m hoping that accessibility such as is laid out by the WCAG 2.0 guidelines are being followed? WordPress has done a lot in terms of your commitment to accessibility, and I hope that this commitment can be truly realized with a brand new code base and a fantastic lack of legacy to deal with. There are so many projects in which if accessibility were only considered at the beginning, millions more users could benefit and interact fully with them. I’m hoping that Calypso has a systemic commitment to accessibility and that the community won’t need to build accessibility on as an afterthought. This is really critical.

It’s not my area of expertise, but worst case using the new APIs an accessibility-focused client could be created, or even different ones for different use cases be it mobility or vision. The WP-admin is still there as well for fallback.

Definitely trying to prevent worst-case senario. The last thing needed is fragmentation in regard to accessibility. And yes, I’ve gotten involved. 🙂

I’m just giddy about this change. I think this is precisely the shot in the arm that was needed to move WordPress forward, and I hope that .org will follow the lead. Kudos to you and your team Matt. Really impressive.

Amazing Matt. I love the part when you said ” A lot of people thought we should keep this proprietary, but throughout my life I’ve learned that the more you give away, the more you get back”
Thanks for the effort to make wordpress and web environtment better.

All javascript?! I block scripts from running on iOS with content blockers, not just because of trackers, but because so many devs abuse them and make stupidly heavy sites. Does this mean WP based sites will no longer work for me?

Do you anticipate the future of themes and plugins turning from PHP to JS as well? I mean I realize this is more wp-admin and core but curious if you think it will all move that direction with time.

Well this sounds absolutely incredible. My team and I were only discussing the need in the ever evolving world of user experience the necessity to drive towards a more consistent API approach.

This will be getting the full dissection tomorrow and comes at a time where our digital marketing agency are about to start work on a game changing platform for local services here in Ireland.

Excited doesn’t quite cut it.

Speed of JS has always fascinated me. But when it comes to SEO, I hold back and take the good old way.

Anyone has figured out how to solve SEO issue while creating a fast JS site using RestAPI? Solving this could truly open a lot of doors in WP theming/plugin scenario.

I think what I love best about the Calypso project is the incredible opportunity it’s going to provide for accessibility. Unfortunately, the accessibility efforts with WordPress.org got off to a late start, which has meant that we’ve been under a backlog of stuff along with new features. With Calypso, if the opportunity is seized upon, we don’t have to have that same problem. So congratulations on this Automattic.

[…] Web’in yüzde 25’ini tek başına omuzlayan WordPress‘in blog servisi WordPress.com baştan sona yenilendi. 13 yıllık geliştirmenin ardından geçtiğimiz yıldan itibaren Calypso kod adıyla başlanan WordPress.com’u sıfırdan tasarlama projesiyle gelen ilk ve büyük değişiklikleri, Automattic’in kurucusu ve CEO’su Matt Mullenweng, 23 Kasım’da kendi blogundan duyurdu. […]

[…] Today I downloaded this WordPress Desktop Application on my MacBook Pro and started using it. As soon as I saw the initial screen I understood that this is the same inteface that we all have seen on WordPress.com to manage all our sites from a single page website. Also the WordPress application for Android / iOS and Windows works on the same framework. I liked the MAC version and I am using it, in fact I am writing this post using this WordPress Desktop Application. If you are interested in knowing more about the framework which is open source and developed by Automattic, check this post by Matt. […]

Congratulations! I am so impressed. This is huge step for WordPress and the future of developing web, publishing content and sharing knowledge. Love that you’re first-moving on new technologies like React. Great choice, by the way. You are setting a great example.

While conceptually I applaud Calypso as a huge advance for WordPress, as far as underlying technological choices, one big reason I chose Mithril over React for a big project was Facebook’s overly-broad patent termination policy for React. Hopefully Automattic has reviewed that policy in detail and fully assessed the legal risks to Automattic and the WordPress community versus technical benefits of using React in Calypso versus other workable technical alternatives like Mithril? If picking React was not a fully-informed choice accepting a substantial theoretical long-term legal risk in exchange for short-term practical benefits or various community momentum effects, I’d suggest starting to convert Calypso from React to Mithril (or a similar system) *today*. Switching the codebase to TypeScript 1.6+ first might facilitate that refactoring.

==== Ironic consequence

I tried to post a longer version of this and it did not go through, probably because of too many supporting links. You can see that version there:
https://github.com/Automattic/wp-calypso/issues/650

Feel free to delete the long version I posted here previously which may well be awaiting moderation and is essentially identical to that issue.

Ironically, immediately after posting that issue to GitHub about patent risks of Facebook and React cutting off free speech about patents on the internet, GitHub decide I was not human and has now hidden my profile. 🙁 This also hides all the free software I’ve developed and published on GitHub. 🙁 That may well in practice ruin my career etc. unless GitHub decides to reinstate my profile. 🙁 I asked nicely just now, so I can be hopeful. 🙂 I guess it’s really true that no good deed goes unpunished — but I guess you still have to do them anyway. 🙂 And what kind of good career could it be if I foolishly let it become dependent on one centralized service like GitHub? 🙂

Anyway, that issue will not be available for viewing by others unless GitHub decides to reinstate my profile.

Thankfully, Andrew at GitHub has decided to reinstate my profile. Whew. 🙂 So that link mentioned above is now working for others who are interested in Facebook-scale licensing issue and React vs. Mithril. It was obviously disconcerting to see years of work disappear in terms of telling people to go to some place to get code or talk about it. That experience is something to reflect further on for everyone — how easy it is for a big centralized service you’ve become dependent on to suddenly cut you off or disappear (whether it folds for business reasons or for legal reasons like patents, or whether it cuts you off because of something you said that a spam-detecting robot did not like). Decentralization is one reason I like WordPress so much over, say, Facebook or Google+. Even when hosting at WordPress.com, you can move the site to other hosting and it will look pretty much the same (well, with some needed plugins perhaps). WordPress.com will even help you do that (for a fee). You can’t do that with Facebook, Google+, or even GitHub.

For example, months ago I asked GitHub why I could not get copies of issues or posts that I had made emailed to me as notifications as I created them. They did not have much of an answer for me. Their position seems to be that the current system only “notifies” you about changes you did not make (because any change you made you already know about, so their is no point in “notifying” you about it). Which conveniently contributes to vendor lock in as well, since it is not easily possible to have a full record of a GitHub project maintained as changes are made — unlike, say, WordPress where you can be asked to be emailed about all changes to a site. (Although it is true you can manually download web pages from GitHub with issues.) I did choose to use GitHub issues for one project, but I can wonder if that will ultimately turn out to be a mistake, as like above, when the whole thing just disappears and I don’t have a complete record of all issues? As Mr. Rogers sings: “What do you do with the mad that you feel?” That is true even when you feel mad with yourself about some dumb decision you made out of convenience without enough forethought… 🙂

To link this back to the Calypso project, imagine something like GitHub but running as decentralized social semantic desktop software under Calypso where you could easily move the entire project from one hosting provide to another, issues and all. 🙂 And maybe with just a little Domain Name System (DNS) magic similar to moving web hosting providers. To me, that is the ultimate potential of something like Calypso when considered broadly — to make all sorts of sophisticated collaborative software easier to write and maintain.

Paul, as an attorney and a WordPress developer, I mostly agree with your assessment.

Using react.js leaves WordPress seriously vulnerable to Facebook. It’s not unfathomable that Facebook can shut down WordPress should there be any legal dispute in the future.

However, I am quite certain that WordPress legal team has reviewed the legal ramification of using react.js and decided to proceed after determining that such a risk is too remote not to use react.js.

Thanks for asking, Chris. Andrew at GitHub had written back to me with: “Sorry about that! Sometimes our spam-detecting robots get a little overzealous and need to be reprogrammed. I’ve reset your profile, so your repositories and gists should be available again, and you shouldn’t see that message anymore.”

So, that’s all I know.

I wrote more just now about the issue, how it made me feel, and how it might relate to future Calypso innovations (as the desktop and the web blur together and as public keys perhaps get used as IDs for modding posters up and down). That essay can be found at the following link on my own website:
http://pdfernhout.net/on-having-my-github-account-temporarily-hidden.html

I wrote the new content here at first, but decided not to post it so as not to bog down Matt’s blog with another long comment by me and otherwise distract more from an amazing new product like Calypso. In general, real life discussions tend to meander, and it would be good to have better tools on the web for following discussion themes without having to otherwise limit side discussions. Too bad there is not an easy way to mark content as “tangential” on a WordPress blog — probably some WordPress plugin for that perhaps though? Or maybe Calypso might have one soon? 🙂

This is an awesome news! I have dreamed for a long time for the GIT support in WordPress, and you guys give a lot more by bringing node and react as well. This is a big big big milestone and I believe it can work!

Wow! This is a revolution… React is awesome. But am also so happy that PHP is still going to be used for the front-end and theme development… But yet I’m concerned is it going to fade away as time goes on or PHP is still going to stay?

@Matt, nice news bud but you got a lot of devs confused a bit. We are themes/plugins developers that work with multiple theme frameworks ( about 11k of us in 2 major GPL theme frameworks ) and we are all kinda guessing what this means to us.

Learning is not an issue but preparing for new way of doing things is, since we all are currently working on WP projects as we speak.

I did see Wendy asking you something along these lines http://prntscr.com/99iaug
but let us ask you direct questions so that we can continue the projects.

The questions are :

1) Are you planing to drop WP PHP completely and if when ? ( eta if possible )
2) Can we continue standard WP themes and plugins development or should we take a brake and learn new way of doing things ?
3) Is standalone WP going to be affected by Calypso ?

Hope you can find time to answered these. You can tell that we need more info so please try to direct us as much as possible so that we can focus on further development.

Thank you and keep up the great work!

1. PHP still makes a lot of sense for the server side of WordPress, to power the API.

2. Themes and front-end display of a blog are also still PHP, and there are no plans to change that currently. But plugins will need to evolve to be API-driven.

3. Today you can manage and publish to your standalone WP sites from Calypso, including auto-upgrading plugins. If Calypso is successful, I could see the core wp-admin interface taking a similar technical direction in the future.

SHARE YOUR THOUGHTS