— Dave Rupert

— Bill Barnwell

To balance out all this ridiculous self-praise, I’ll talk about one of our screwups: how a well-intentioned decision to add a feature led us into a practical dilemma within our API. This wasn’t something I worked on myself, but this is my rough recollection of how it happened.

The idea

As you’ll recall from part one, our spot URLs were quite simple: http://gowalla.com/spots/16384. One day, one naïve day, someone said, “Hey, wouldn’t it be great if our spot URLs had the name of the spot in them? If the URL for the Wahoo’s down the street was /spots/wahoos-fish-tacos-austin instead of /spots/1019?”

This is easily justifiable as an SEO decision — when someone searches for a venue, you’d want the Gowalla page for that venue to be on the first page of results, and that’s more likely to happen when the venue name is in the URL. It’s also justifiable on usability grounds; if I’ve been to the Wahoo’s page on Gowalla, and I’m trying to get back to it, I know I can type gowalla.com/spots into my browser and let the URL auto-complete tell me where to go from there. (Though today’s Firefox-style address bars make this a less compelling argument.)

We decided to generate a unique slug for each spot by downcasing, removing punctuation, replacing spaces with hyphens, adding the city name to the end, and (if necessary) adding further components to the end until the slug was unique: state, address (if we knew it), postal code, and finally the spot ID, should all else fail to disambiguate it.

We had some logic in our routes so that when the app saw /spots/1019, it would know to look the spot up by its ID, but when it saw /spots/wahoos-fish-tacos-austin, it would know to look the spot up by its slug. So that we had a hard-and-fast rule to separate these cases, we made sure that a slug wouldn’t start with a number. As I recall, this was all Mattt Thompson’s handiwork, and he brought his usual brilliance to the task.

We’d generate the slug immediately after spot creation. I’m guessing we wrote a Resque job to add slugs to spots that already existed. And when all spots had slugs, we triumphantly turned it on. And then, a few hours later, we turned it off, somewhat less triumphantly.

Our blunder

If you read part one, you might already suspect what happened. In our API, URLs are the primary identifiers of resources. If you asked the API for a particular checkin at Wahoo’s, the response would have a spot object whose URL would be /spots/1019. When we changed a spot URL, we changed that unique identifier.

Now, because we were big on HATEOAS, this wasn’t a huge deal. And spots, in particular, were natually linked to from other resources. You wouldn’t ever start an API-related task on an individual spot page; you’d probably start at /users/me (the magical URL for getting info about the logged-in user) or at /spots (the URL for doing a spot search). And even if you had an old spot URL lying around, it would work fine, because we still responded to the ID-based URLs. So as long as we made the switch all at once, this was not a big deal.

Unfortunately, we didn’t make the switch all at once. Though we had updated the logic on the spot model so that asking for the URL would return the slug-based URL, we forgot about the one place where we took a shortcut.

Visited spots

When I gave you the excerpt from a sample call to /users/savetheclocktower in part one, I left in a weird property: something called visited_spots_urls_url. This was a URL that, when requested, would return a list of URLs for all the spots that a user had checked into.

// GET /users/savetheclocktower/visited_spots_urls
{
  "urls": [
    "/spots/1019",
    "/spots/15555",
    "/spots/91142",
    // ...
  ]
}

Why on earth did we have this? Because there are plenty of use cases for a unique list of all the unique places a user has been without any time/frequency context. For one, we used it in the mobile clients so that, in a list of spots, we could mark the spots you’d checked into before, hopefully helping you pick the correct spot out of a list so that you could check in just a bit faster.

Maybe we deserved it for having a magical API resource that returned a list of indeterminate size, but this was a solution that had worked for us so far.

The first time someone asked for this resource, we made a database query (on the stamps table, not the gigantic checkins table, thank god) to obtain the result. Then we stuffed it into memcached so that we wouldn’t have to make that awful database query again until the user checked into a new spot.

Except, well, not quite. We didn’t put spot URLs into memcached; we put spot IDs into memcached, because we wanted that collection around for server-side purposes, too, and on the server side you’d rather be working with IDs than URLs. To go from spot IDs to spot URLs, we did this:

def uniq_visited_spot_urls
  uniq_visited_spot_ids.map { |id| "/spots/#{id}" }
end

I took the scenic route, but here’s our problem: to make a switch to slug-based URLs, we needed the whole API to use the same logic for generating spot URLs. Except that logic lived on the spot model itself, and if you had just a spot ID (as in this example) there was no quick way to turn it into a spot slug.

So even after we flipped the switch, /users/me/visited_spot_urls returned old-style spot URLs because it was taking a shortcut for performance reasons. A couple people complained on the API mailing list; someone in the office said, “hey, the app isn’t showing check marks next to spots I’ve checked into”; and then we figured out the problem and switched back to the old spot URLs until we could think this whole thing through.

Stuck

The problem I described above wasn’t a deal-breaker. We could’ve figured out a better way to cache spots, or we could’ve just maintained separate cache keys for visited spot IDs versus visited spot URLs.

We never did go back to the slug URLs, though. As far as I know, we just got busy with other stuff. But I know that the more I thought about this problem, the more I felt that there wasn’t a good way around it. Three concerns fenced us in:

1. In the API, a resource’s URL was both its unique identifier and a hyperlink. It was impossible to change one without changing the other.

2. For the purposes of browsing gowalla.com in a web browser, it was in our interest for spots to have long, informative URLs.

3. For the purposes of using Gowalla as an app, it was in our interest for spots to have short, bare-bones identifiers.

This was the main tension. There are some clever ways around this, but none that are quite clever enough:

1. Try to satisfy all three concerns at once. Stick with the short URLs for the API. But if someone requests /spots/1019 in a browser, just do a 302 redirect to /spots/wahoos-fish-tacos-austin. Assuming this would have the same impact on SEO, it certainly seems like a prudent solution. But to do a proper redirect, we’d first have to look up the spot to find its slug. So we’re talking about a round trip to either a database or a caching layer before we know which URL to redirect to. Even in the best-case scenario, it would add a perceptible lag to page load, and that’s no fun.

2. Forget about concern #1; separate the hyperlink from the unique identifier. Stick with the short URLs for the API. But put a full_url property in the JSON response for spots. OK, but now there’s just one resource for which we have to maintain this dichotomy, and I doubt API consumers would bother with the full URL if the short URL still worked.

3. Forget about concern #3; just move to long URLs everywhere. Except a slug-based URL would be twice as long as an ID-based URL, maybe more. For most API calls, that’s a small (but still significant) increase. But our /users/me/visited_spots_urls response would be twice as large. This is probably the least distasteful of these three options, but mobile apps always have to be sensitive about the amount of data they’re sending over the wire, and it would make no one happy to bite that bullet.

Let’s also remember that any client-side caching involving spot URLs (and, yes, the Gowalla app did a lot of that) is invalid now that we’ve made this switch.

And one more groan-inducing thing: what happens when a spot’s name changes? Gowalla spots could be created by anyone, and it wasn’t uncommon for someone to make a typo when creating a spot name. We had a network of volunteer super-users who spent their time renaming Starbuck’s to Starbucks, Barnes & Nobles to Barnes & Noble, Chervon to Chevron. And the name is part of a spot slug. So we either change the slug when the name changes, even though cool URIs don’t change; or we say that the first slug is the only slug, and that gas station will forever bear the URL /spots/chervon-giddings-tx through no fault of its own.

Or we forget about putting the spot name in the URL — and then our problem goes away.

The answer

So, yeah, maybe the answer was not to do the thing in the first place. Or maybe the answer was to do it, but not until we released the next version of the API in order to break stuff as little as possible. Or maybe the answer was to plow ahead, consequences be damned.

None of this exposes a critical flaw with hypermedia APIs. These were mundane practical problems caused by previous decisions that we ourselves made. This is merely a story about how you can do all the right stuff and still get bitten by those Rumsfeldian unknown unknowns.

Nonetheless, when I see DHH arguing against Hypermedia APIs, I worry that we’re in serious “die, heretic scum” territory. I’m no expert, but the difference between REST and Hypermedia really doesn’t seem that large, especially in a universe where SOAP is a thing. Moreover, Rails deserves a lot of credit for demonstrating that web APIs could work within HTTP rather than try to reinvent it. Out of the box, Rails checks three out of four of Steve Klabnik’s boxes, and all we’re arguing over is that last one.

Anyway, what prompted this was a post by Adam Keys, my former Gowalla colleague. I agree with most of what he’s saying here. My gut reaction to Hypermedia APIs is this:

Roughly 90% of it is sensible stuff that I’ve already seen in the wild and which is demonstrably a Good Idea. The remaining 10% is the stuff that (at this early stage) seems non-intuitive, or overkill, or YAGNI, or whatever the word is for a thing that you think is awesome but which your users won’t give a damn about.

In fact, that last thing is my chiefest concern. The final 10% seems to require nontrival re-education on the part of consumers. I don’t mean they’d have to be brainwashed; I just mean that some of the stated benefits only come to pass if the consumers buy in, and in my experience an API consumer wants to do the simplest thing that could possibly work. I believe this is what Adam is getting at in his follow-up post.

The Gowalla API

Adam’s opinions on hypermedia are informed, in part, by his time at Gowalla, and so are mine. Before I convince myself it’s a bad idea, let’s take a retrospective look at the Gowalla API (which, by the way, was started in 2008–2009) and see how it measures up against a hypermedia rubric.

Things we did right

Addressability

URLs identified resources. A spot had the same URL whether you were requesting an HTML representation in a web browser or a JSON representation from curl. If a form could create a resource by POSTing some multipart form data to a URL, odds are a client could create the same resource by POSTing some JSON to that same URL.

This was less like a knowing philosophical decision and more like a thing that Rails just does by default. Until rather late in the game, if you were using the Gowalla API, your requests were hitting the same controllers and actions as web users’ requests. (Eventually we decided to move API stuff into dedicated controllers for maintainability’s sake, but that tilting-at-a-windmill saga will have to be told on another day.)

Content negotiation

As implied above, the API was driven by content negotiation. If you asked for HTML, you got a browser representation; if you asked for JSON, you got a pure data representation. (If you asked for XML, we pretended we didn’t hear you.)

HATEOAS

We endeavored to practice what Steve calls HATEOAS: Hypertext As The Engine Of Application State. To over-simplify: a response should publicize the URLs of any resources that are reasonably related to it.

(By the way: I do not come down on one side or the other here. If there’s a natural workflow to your API, as there was for Gowalla’s, it obviously makes sense to publicize related resources rather than force a user to memorize your URL-making conventions. On the other hand, odds are high that your API consumers will make assumptions about your URL schemes anyway. So I’m not sure what HATEOAS gets you in the real world, except for the ability to say “I told you so.” Which, admittedly, is underrated.)

But back to Gowalla. If you were authenticated with Gowalla and requested the resource for your own user profile, this is a snapshot of what you saw:

// GET /users/savetheclocktower
{
  "stamps_count": 14,
  "stamps_url": "/users/savetheclocktower/stamps",

  "pins_count": 11,
  "pins_url": "/users/savetheclocktower/pins",

  "top_spots_url": "/users/savetheclocktower/top_spots",

  "friends_count": 44,
  "friends_url": "/users/savetheclocktower/friends",

  // ...

  "visited_spots_urls_url": "/users/savetheclocktower/visited_spots_urls"
}

Nearly every meaningful kind of resource is discoverable by starting at this response and navigating through the various URLs. (Of course, not every API use case would start with loading a specific user’s profile. For instance, those that were interested mainly in the place database would probably start with the result set of spots from a geographical search.) Though the URL conventions were simple enough that a client could build URLs on their own, we tried to make it so that building URLs was harder than just using the URLs that we’d given you in the response. This gave us a theoretical freedom to change URLs in the future (not that we’d ever want to do so, we thought).

This style — in which everything ending in url points to another resource — is just one version of what HAL or Collection+JSON are trying to formalize. It’s a pattern that worked very well for us. It made our API very “surfable,” and though I doubt we had machine discovery in mind when we were doing it, it did mean that the API explorer I built was a lot of fun to use — anything that looked like a URL was hyperlinked, and clicking on it would load that new resource in the explorer. We updated the URL hash, too, so the back button would return you to the previous resource.

Crucial to all of this is that the API used a resource’s URL as its unique identifier, rather than a raw ID. This is the part that Rails didn’t give you out of the box, so credit to Scott for designing it this way.

What we could’ve done better

API Versioning

Rather than version our API with MIME types, we used a separate X-Gowalla-API-Version header, defaulting to the most recent version if a JSON-requesting client omitted this header.

I don’t necessarily think that our approach was wrong — only that if we’d made people opt into a particular MIME-type, rather than just the generic application/json, and if the MIME-type was tied to a particular API version, we likely would’ve had fewer incidents where changes we made inadvertently broke third-party tools.

Discoverability

When I said that every resource was discoverable, I was lying. Nearly all GET requests were discoverable. Anything that required a POST (and any GET that involved query parameters) wasn’t documented within the API itself, so you’d have to dig into the API documentation to figure out exactly how they worked. If we were doing it over again now, it’s possible that we’d toss in query templates or something like it, but I suspect we wouldn’t have bothered.

Sub-resources

We never really figured out the best way to do sub-resources. Consider a checkin, which referenced one user and one spot:

// GET /checkins/131072

{
  "created_at": "2010-12-21T01:03:15-06:00",
  "message": "I am eating here under protest.",
  "url": "/checkins/131072",

  "user": {
    "first_name": "Andrew",
    "last_name": "Dupont",
    "url": "/users/savetheclocktower",
    "image_url": "http://some.crazy.cdn.url/jklyjksljkrewus.jpg",
    "hometown": "Austin, TX",
    "photos_url": "/users/savetheclocktower/photos"
  },

  "spot": {
    "name": "Red Lobster",
    "url": "/spots/15555",
    "image_url": "http://some.crazy.cdn.url/jjkpwopresas.jpg",
    "lat": -90.105324,
    "lng": 30.448674,
    "address": {
      "street_address": "123 Fake St.",
      "locality": "New Orleans",
      "region": "LA",
      "iso3166": "US"
    }
  }

  // ...
}

Now, we don’t want to dump the whole user resource into our response, but neither do we want to force someone to follow a URL to learn anything about the person who checked in. So we chose the middle ground: include a “concise” representation of the resource. In this case, the properties we show from the sub-resources are the things we’d need to know if we were rendering the checkin in a list; with this response, I can render the sentence “Andrew checked in at Red Lobster,” along with a user avatar and a spot icon, without having to make any other requests.

This eventually got crazy, though, because a sub-resource could plausibly have a half-dozen representations of varying lengths, each of which could be justified from context. For instance, if you requested a user’s checkins, you’d get a list of these:

// GET /users/savetheclocktower/checkins
{
  "checkins": [
    {
      "created_at": "2010-12-21T01:03:15-06:00",
      "message": "I am eating here under protest.",
      "url": "/checkins/131072",

      "user": {
        "first_name": "Andrew",
        "last_name": "Dupont",
        "url": "/users/savetheclocktower"
      },

      "spot": {
        "name": "Red Lobster",
        "url": "/spots/15555",
        "image_url": "http://some.crazy.cdn.url/jjkpwopresas.jpg",
        "lat": -90.105324,
        "lng": 30.448674,
        "address": {
          "street_address": "123 Fake St.",
          "locality": "New Orleans",
          "region": "LA",
          "iso3166": "US"
        }
      }
    },
    {
      "created_at": "2010-12-21T01:02:44-06:00",
      "message": "I am in need of fuel for my car.",
      "url": "/checkins/130808",

      "user": {
        "first_name": "Andrew",
        "last_name": "Dupont",
        "url": "/users/savetheclocktower"
      },

      "spot": {
        "name": "Chevron",
        "url": "/spots/91142",
        "image_url": "http://some.crazy.cdn.url/oahkhjs.jpg",
        "lat": -90.105416,
        "lng": 30.444994,
        "address": {
          "street_address": "919 Fake St.",
          "locality": "New Orleans",
          "region": "LA",
          "iso3166": "US"
        }
      }
    },

    // ...
  ]
}

Here, the spot resource is using the same representation that it did for an individual checkin, but the user resource is much more sparse. Why? Because (a) in this response, all the checkins are guaranteed to be from the same user, and the redundancy bothered the hell out of me; (b) chances are you followed this URL from the response for /users/savetheclocktower and thus already have the full representation of this user.

If you were to ask for a single spot’s checkins, the situation would be reversed — the user representation would be the same as for a single checkin, but the spot representation would be as minimal as possible.

We managed this complexity as best we could. First we added a to_public_json method on models — so named because it wasn’t trying to be exhaustive like to_json; it merely wanted to expose properties that would be relevant for a public API. It optionally took a symbol argument that would specify a named represenation, much like DateTime#to_formatted_s lets you choose between date formats. When even that got too complicated, Brad Fults wrote an awesome thing called Boxer that centralized all this logic in a place that was neither a controller nor a model.

I’d always wished for YAML-style anchors and references in JSON, but I didn’t want to do anything crazy with our JSON responses that put an extra burden on API consumers. Still, if I were to do it over again, I’d probably do something like this:

// (hypothetically)
// GET /users/savetheclocktower/checkins

{
  "includes": {
    "users": {
      "savetheclocktower": {
        "first_name": "Andrew",
        "last_name": "Dupont",
        "url": "/users/savetheclocktower",
        "image_url": "http://some.crazy.cdn.url/jklyjksljkrewus.jpg",
        "hometown": "Austin, TX",
        "photos_url": "/users/savetheclocktower/photos"
      }
    },
    "spots": {
      "15555": {
        "name": "Red Lobster",
        "url": "/spots/15555",
        "image_url": "http://some.crazy.cdn.url/jjkpwopresas.jpg",
        "lat": -90.105324,
        "lng": 30.448674,
        "address": {
          "street_address": "123 Fake St.",
          "locality": "New Orleans",
          "region": "LA",
          "iso3166": "US"
        }
      },
      "91142": {
        "name": "Chevron",
        "url": "/spots/91142",
        "image_url": "http://some.crazy.cdn.url/oahkhjs.jpg",
        "lat": -90.105416,
        "lng": 30.444994,
        "address": {
          "street_address": "919 Fake St.",
          "locality": "New Orleans",
          "region": "LA",
          "iso3166": "US"
        }
      }
    }
  },

  "checkins": [
    {
      "created_at": "2010-12-21T01:03:15-06:00",
      "message": "I am eating here under protest.",
      "url": "/checkins/131072",

      "user": { "include": "/users/savetheclocktower" },
      "spot": { "include": "/spots/15555" }
    },
    {
      "created_at": "2010-12-21T01:02:44-06:00",
      "message": "I am in need of fuel for my car.",
      "url": "/checkins/130808",

      "user": { "include": "/users/savetheclocktower" },
      "spot": { "include": "/spots/91142" }
    },
    // ...
  ]
}

All sub-resources would get put into a hierarchical repository at the root of the response, and the structure of that repository would mirror the URL structure, so that when you saw an object with an “include” property, you could try to look it up locally and then fall back to another HTTP request if necessary. This is probably overkill, but dammit, if I’m going to introduce an extra-language convention into JSON, I’m going to give it some style.

The Verdict

On reflection, I think we did pretty well, especially considering that these decisions were made incrementally over the course of two years. I can think of only one instance when the API design painted us into a corner, and that’s the story I’ll save for next time.

My brain lies to me sometimes. Because I’d loved Mass Effect, and because Mass Effect 2 changed several of the things I’d loved about the first game, I convinced myself that the sequel wasn’t quite as good. It had been a year and a half since I’d touched either game, so after I finished Mass Effect 3 I decided to go back to the very beginning and do a marathon playthrough with a fresh character. Not only would it be a plot refresher, but it’d let me undo all of the dumb mistakes I made in games one and two that I ended up having to pay for in game three.

So ME3 deserves credit for that, at the very least. When you play ME1 and ME2 back-to-back, it’s clear which is the better game. Yes, I like ME1’s skills tree better; yes, I like being able to customize weapons and earn tiny amounts of XP for every little thing I do. But there’s no way I can go back to ME1’s unwieldy combat or awkward pacing. I’d like to apologize to ME2 for being so mean to it.

I was expecting not to like Mass Effect 3. As it turns out, ME3 contains several of the most emotionally poignant moments I’ve had playing video games. It has the hard task of incorporating decisions you’ve made in previous games — and making the player feel that those decisions were significant — while maintaining an economy of story and a clarity of plot. It’s done that amazingly well. Until ME3 I don’t think I’d ever played a game that added replay value to its prequel. Have you?

My favorite parts of ME3 happen hours before the game’s notorious and infuriating ending. Yes, the ending is bad. Even when you’re expecting it to be bad it manages to underwhelm. It’s bad in almost all the ways a sci-fi ending can be bad. It’s thematically disjoint, arbitrary, and derivative. It offers a false choice. It violates continuity. It violates canon. And it’s so maddeningly ambiguous that it doesn’t feel much like an ending to anything.

The upcoming free DLC promises to provide more context to each ending. But at best it’ll address only some of this. I seriously doubt this whole thing can be un-fucked.

Here’s the thing: I never cared much about this whole galaxy-wide threat. For me, the Reapers were a glorified MacGuffin, an excuse for me to ride around on a ship and meet cool alien races and shoot robots with sniper rifles. A Mass Effect game is at its best during side missions, when you can pretend you’re dealing with a Star Trek–style episodic threat and forget about that thing that’s trying to destroy the universe.

That’s why the ending, awful as it is, doesn’t ruin this game for me. I’ve played it for at least forty hours now and only three of those hours have been unenjoyable. Does the underwhelming finale of Seinfeld ruin the entire series, or even just the final season? What about The Sopranos? On Metacritic, Mass Effect 3 has a score of 93/100 according to critics, but only 50/100 according to Metacritic users. I understand their frustrations, but I’m on the critics’ side here.

But the ME3 ending disappoints me in a deeper way. I would play at least six more games just like ME3. I had been looking forward to playing those games someday. I had thought that Bioware was looking for their own rich universe for sci-fi storytelling. The ending of ME3 — no matter which one you pick — seems to salt the earth, as if they wanted to rule out any future stories that would take place after the events of the game. They could do prequels, but there’s only a thirty-year window between humans’ first contact with other races and the events of ME3.

Aside from a spin-off game or two, I doubt Bioware is interested in telling more Mass Effect stories. And that’s profoundly disappointing. I’m most of the way through my second ME3 playthrough, but I’m playing it more and more slowly. I’m in no hurry to get to an ending that reminds me, clearly and bluntly, that it’s all over.

Why do I want a game to have long-lasting impact? Why does it have to be profound to me after I’m done with it? I played the original Infamous for at least fifteen hours — doesn’t that say more about its quality than whatever I feel about it eighteen months later?

Because I feel the same way about Infamous 2. It’s just as good as Infamous in all the ways that Infamous was good, and bad in all the ways its predecessor was bad.

At the beginning of Infamous 2, plot contrivances lead Cole and his annoying sidekick Zeke to make their way from Fake New York (Empire City, the setting of Infamous) to Fake New Orleans (New Marais). A giant evil thing called The Beast is heading in their direction, slowly and surely, but in the meantime there’s a new city to play in. The open world is full of open-worldy stuff; there are scattered things to collect, and territory missions, and even user-generated side missions.

And there’s the main storyline, a set of missions where you meet other people with crazy superpowers and fight generic monsters and find blast cores (which function as rather blatant plot coupons; The Beast gets closer each time you collect one). All of this is fine, I suppose, and I’ve loved games with plots just as ridiculous as this.

Maybe I’m just getting tired of games that don’t seem to care about their own plots. There’s no sense of pacing or cresecendo; Cole seems to react to everything that happens with the same sort of grizzled nonchalance. There’s no attempt to make exposition seamless or elegant; if there’s something you need to know, a supporting character will give you a clumsy info-dump the moment you need to know, and no sooner. (Midway through the game, Lucy tells Cole about a plague that’s killing half the city. If it’s that large, shouldn’t Cole know about it already?)

Or maybe it’s that I can’t think of a single thing Infamous 2 does that hasn’t been done better by another game. Maybe it’s good in a way that’s too balanced, and if it had tried hard to excel in a certain aspect I’d at least be left with something to grab onto.

— Judge Sam Sparks

— Sam Harris

— Stanley B. Greenberg

— Charles P. Pierce