Meteor demystified

June 12, 2013 · 6 min read

In which I open the hood of the Meteor framework and see what’s inside.

I recently started a new web project, and several friends suggested I check out Meteor, a new framework for building real-time webapps. Their intro screencast is impressive, and the project generated a lot of excitement when it was first announced. It lets you write real-time apps focusing just on your domain model and views, without all the plumbing—like Rails did for regular websites several years ago.

At the same time, the prospect of using Meteor for a production app would make any battle-scarred devops veteran nervous. The current version as of this writing is 0.6.3, which they call an “early preview”, saying: “Meteor is still under rapid development. Expect major API changes in each release.” (When will it hit 1.0? “More than a month, less than a year.”) The list of projects still has a lot of toy/demo apps; it’s unclear if anyone uses it in production on a major site.

Moreover, Meteor is “magic.” How does it work? What is it really doing? Will you even be able to understand it?

Despite all this, the prospect of rapid development of a real-time app was so seductive that I decided to look into it thoroughly—in effect, to take it apart and see how it works—to get underneath the magic.

What follows is a summary of what I learned from this exercise, which I haven’t seen explained quite this way anywhere else.

Meteor’s central goal

To understand Meteor you need to understand its central goal, which drives the design.

The core problem Meteor solves is a data synchronization problem. Your webapp is a distributed system, with data on both client and server. The main challenge is to synchronize the data instantly and seamlessly between the two—and more, to synchronize all clients with each other. Ideally it should appear to all users that they are interacting directly with a single, shared store of data.

To solve this, Meteor gives you:

1. A local copy of the data model, easily accessible via lookup and query interfaces. (This is common to all webapps that take the approach of client-side MVC.)

2. An open socket to the server to get real-time updates (over WebSockets, long polling, or a similar transport. This is common to all real-time apps.)

3. A latency-hiding mechanism that makes user updates appear to happen instantly. The method is clever: The core business logic exists on the client as well as on the server, and when the client sends a request, it also runs that logic locally, without waiting for the response—in effect simulating the server and predicting the outcome. (This is possible because JavaScript is the language on the server as well as the client, so the same code can run in both places.)

   This is an optimistic protocol: in the normal case, it hides all the latency of the network request; if there is an error or race condition, the client will get the authoritative response from the server soon enough. However, it doesn’t preclude security: the server is still the authoritative source of data; the client has no extra privileges. And you can share only a subset of the server code with the client, if you want to keep some of it private.

4. A dependency-tracking mechanism to automatically refresh views when their underlying models change. In most client apps, the view or controller observes its model in order to refresh the view when the model is updated, but usually these dependencies must be wired up by hand. Depending on the app, this can be tedious and error-prone. Meteor provides a subsystem to track these dependencies automatically.

5. A sophisticated template management subsystem that allows you to refresh or re-render views at any time without disturbing the user. For instance, the state of all inputs is preserved, so if the user has entered text it is not lost on a refresh. This makes it safe to do refreshes frequently and automatically by the mechanism just described.

The first three of these together mean that all interactions are local; the sync to the server, in both directions, is decoupled from user interaction. The last two mean that the view is kept in sync with the model by default, with minimal developer effort.

All together you get the central goal: a distributed app with the look and feel of a local one.

What you get: The parts list

With that context, let’s open the box and see what’s inside.

Here are the key pieces Meteor comes with. All of this is described in the documentation, which I’ll link to where appropriate:

• A web server on top of Node that lets you deploy some pieces of code to the server, deliver other pieces to the client, and have some that are in common to both (described here).
• An RPC framework, over WebSockets or long polling, that does latency hiding as described above.
• A MongoDB API that is unified across client and server. The client-side implementation uses the RPC framework to forward the calls to the server, and thus takes advantage of its latency-hiding mechanism.
• A pub/sub mechanism that implements the Distributed Data Protocol, an open standard defined by the Meteor group.
• A template management subsystem as described above.
• A dependency-tracking system as described above that is used to autowire views to their data dependencies. This is nicely decoupled into an independent component, so you can use it for anything, not just views and models.

Read the docs to learn more; they are clear and helpful.

Concerns

This technical deep dive had a paradoxical effect on me.

On the one hand, I became much more comfortable with Meteor’s “magic” once I understood how it worked. On the other hand, once I understood it, the gap between Meteor and existing web technologies appeared to shrink. After all, plenty of realtime apps are built on Node and MongoDB using REST APIs and a pub/sub package like Socket.IO. At that point, the only parts of Meteor you’re really missing are latency compensation, dependency tracking, and template management. Each is a solvable problem (especially if you don’t have to solve the general case).

In that light, some of my concerns about Meteor came to the fore:

• It’s not battle-hardened. Again, it’s an “early preview”. It hasn’t had time to bake in production—to test performance and correctness in the field, to resolve subtle incompatibilities with other popular packages and libraries, to determine what configuration options need to be exposed.

• Scaling. As of this writing, Meteor hasn’t really solved the horizontal scaling problem. As far as I understand, there’s no server-side pub/sub component yet (such as Redis), so once you get beyond one app server, you’re polling the database for updates. Fixing this is on the roadmap for 1.0 but not currently being worked on.

• Latency. When I deployed one of Meteor’s example apps to their own servers, there was noticeable latency on first load and even while navigating around a small, simple dataset (see for yourself). (This may be an artifact of Meteor’s hosting environment, which is only meant for demos and experiments, but I couldn’t deploy to Heroku to test that hypothesis: Heroku doesn’t yet support WebSockets, and Meteor doesn’t currently have a way to turn them off.) Improving app load time is also on the roadmap, but not until after 1.0.

And that brings us to some of what is odd about the design of the Meteor server. In its enthusiasm for a new way of writing apps, Meteor has left behind some of the basic architecture of the web. It fully embraces the single-page app concept, delivering the entire client bundle on each request (which contributes to the latency problem). There is not yet a way to break up an app into pages: the server actually delivers the same response to the client no matter what URL was requested; like a native iOS app, it’s up the client to interpret the URL and display the appropriate view. Meteor does not provide a REST API; it doesn’t even really use HTTP (except to deliver the initial client bundle), sending all requests and responses over a two-way connection in RPC fashion.

Indeed, a Meteor app is not truly a web app; it is an RPC-based client/server app that happens to be deployed over HTTP and run in a browser.

I think this belies a certain philosophical leaning on the part of Meteor’s designers—but it doesn’t need to become a holy war. The Meteor group is not opposed to a REST API or a page model; in fact, these things are on the roadmap for 1.0. They’re just not there yet.

Overall, Meteor is an impressive piece of engineering, but I’m not going to use it for my current project. Instead, I’m going with old, boring technologies that have been around for years and are already used in industrial-strength applications. Not that Meteor won’t be ready for prime time eventually—the team is smart and accomplished, and I wouldn’t bet against them. But in matters of production software, I’m just old enough and just crusty enough to stick with the devil I know.

Comment on Hacker News

Thanks to David Crawford and Andrew Miner for commenting on a draft of this post.