Engineer for Serendipity

The title of this post is a quote from Roy T. Fielding (he had a hand in creating the HTTP 1.1 spec in 1999, basically defined / put into words the architecture and behaviors of the internet as a hypermedia system – REST, and has done about a million other things).

The quote alludes to making your “thing” so that users accidentally find something fortunate. While I do not believe in chance, I do believe in the serendipitous nature of innovation. Because of Fielding and others like him and their "out of the box" thinking we have the internet and all of its wonder.

This video (posted via @digital_nomads) illustrates what happens when people think outside of the web browser or app window and use technologies established over 10 years ago to make people stand and clap. It seems the device in the video uses web based APIs to get data and use it in an unbelievably awesome way.

As you build applications, I challenge you to kick out the ladder and set it on fire, challenge the walls of your device – push against them, bring them down.

If we do our jobs as software developers and API Architects, consumers will find themselves in the middle of serendipity.

RESTful Thoughts - OAUTH implementation for the F1 RESTful API Part 1

We took some time with this implementation and really did some soul searching on Authentication / Authorization mechanisms for the new Fellowship One API. The past few APIs that we've released used credential based authentication ( + signed keys for Payment Gateway) which was appropriate considering the intended usage.

With the RESTful API we know that churches, 3rd parties, vendors, volunteers, and the like will be consuming the API's secure resources - so we chose OAUTH.

The OAUTH features enable all of those consumers to create web sites or applications (consumers) that can access resources available via Fellowship One (Service Provider) securely, and without having to store or gather user credentials.

The benefits of our implementation of OAUTH are:

• Both the church and user will be able to protect / authorize access to their resources
• Resources will only be available to applications that both the church and users of that church has access to (configurable)
• The Consumer will never have access to login information or credentials - so the login is totally owned by the service provider

All requests for resources (excluding tokens and login) must have a Consumer Key, Access Token, and signature.

Roles of our implementation of OAUTH are:

• Service Provider: Fellowship One RESTful API - A web application (API) that allows access via OAuth.
• Consumer: 3rd party / church / Individual - A website or application that uses OAuth to access the Service Provider on behalf of the User.
• User: Portal User, User, Weblink User - An individual who has an account with the Service Provider.

OAUTH Parameter information (from the protocol documents):

• Consumer Key: A value used by the Consumer to identify itself to the Service Provider.
• Consumer Secret: A secret used by the Consumer to establish ownership of the Consumer Key.
• Request Token: A value used by the Consumer to obtain authorization from the User, and exchanged for an Access Token.
• Access Token: A value used by the Consumer to gain access to the Protected Resources on behalf of the User, instead of using the User’s Service Provider credentials.
• Token Secret: A secret used by the Consumer to establish ownership of a given Token.
• OAuth Protocol Parameters: Parameters with names beginning with oauth_.

Brief overview of the F1 OAUTH process (assuming all valid credentials and authority):

• The Consumer application will use a consumer key and secret to sign all URIs so that the F1 API will be able to verify the consumers identity and credentials.
• The Consumer application first requests a "Request Token" to use when the Users access the login pages -> The Service provider grants the request token.
• The Consumer application will direct the User to the Service Provider to obtain User Authorization
• The Service Provider will obtain authorization from the User and direct the User back to the Consumer
• The Consumer application will then request an Access Token -> The Service provider grants the request token and marks the "Content-Location" header with a URI to the User information.
• The Consumer application will use the Access Token to access Protected Resources

The process defined above requires accessing the following URIs:

Request token URI:
http://{churchcode}.{domain}/{version}/Tokens/RequestToken [POST]

Login URIs:
Portal Users http://{churchcode}.{domain}/{version}/PortalUser/Login [GET]
Weblink Users http://{churchcode}.{domain}/{version}/WeblinkUser/Login [GET]
Groups Users http://{churchcode}.{domain}/{version}/User/Login [GET]

Access Token URI:
http://{churchcode}.{domain}/{version}/Tokens/AccessToken [POST]

RESTful Thoughts - Planned HTTP Headers for the upcoming RESTful API

Here is the HTTP Header map for the upcoming Fellowship One RESTful API. Please comment / ask questions / make suggestions.

Just like the response codes, many decisions made for this map came from RFC2616, Fielding's dissertation, and analysis of several dozen RESTful or REST "like" APIs currently on the programmable web.


Request Headers

Accept : type/subtype

Example - Accept : application/xml (Available types and sub types: application/xml, application/json)

Notes - Defaults to application/xml if one is not passed in, and when a user sends application/* the resulting content sub type will be XML. Multiple types/sub types can be sent in but as per the specification the first match in degree of specificity will be the first used.
i.e. if user sends in: application/*, application/xml, */* the order would be as follows:
1) application/xml
2) application/*
3) */*

If an invalid value is passed in via Accept header the response will result in a 415 Unsupported Media Type

Accept-Charset : charset

Example - Accept-Charset: utf-8

Notes - Accept-Charset will not be used and will always default to utf-8

Accept-Encoding : encoding

Example - Accept-Encoding : deflate, compress, gzip

Notes - If none is sent in then the API will default to no compression.
If an invalid value is passed in via Accept-Encoding header the response will result in a 406 Not acceptable

Authorization : credentials

Example - Authorization : Oauth realm=http://api.f1.com

Notes - This is where the OAUTH credentials will go

Date : HTTP-date

Example - Date: Thu, 29 Jan 2009 15:28:25 GMT

Notes - Optional, most commonly passed in with PUTs and POSTs

User-Agent : product/comment

Example - User-Agent : MyChurchSite/v123.x

Notes - Used to identify the HTTP library or client library that was used to consume the API

Response Headers

Allow : method

Example - Allow: GET, HEAD, PUT

Notes - An Allow header field MUST be present in a 405 (Method Not Allowed) response.

Content-Encoding : content-coding

Example - Content-Encoding : gzip

Notes - This will tell the client what type of compression was used on the resource

Content-Length : DIGIT

Example - Content-Length : 1254

Notes - Sent back with each request. Will possibly be available via HEAD requests

Content-Location : absoluteURI | relativeURI

Example - Content-Location : http://api.f1.com/API/People/22114944

Notes - Sent back with each GET request

Content-Type : media-type

Example - Content-Type : application/xml, utf-8

Notes - Details the type of content being returned to the client

Date : HTTP-date

Example - Date: Thu, 29 Jan 2009 15:28:25 GMT

Notes - Will be returned with every response, possibly excluding responses returning status codes of 500

Location : absoluteURI

Example - Location: http://api.f1.com/API/People/22114944

Notes - Applies to 201 and 301 only

WWW-Authenticate : challenge

Example - WWW-Authenticate : Oauth realm: http://api.f1.com

Notes - The URI will change per environment, per version

Notes on caching
All headers having to do with caching are still being implemented.

This HTTP header map is subject to change, however, in the current build of the API these rules exist as defined.

RESTful Thoughts - Planned response status codes for the upcoming RESTful API

In the interest of honoring standards here is the response status code map for the upcoming Fellowship One RESTful API. Please comment / ask questions / make suggestions.

Many decisions made for this map came from RFC2616, Fielding's dissertation, and analysis of several dozen RESTful or REST "like" APIs currently on the programmable web.

200 - ok (GETs, PUTs, and DELETEs)

Entity body - The resource

Example - GET People/1

201 - Resource created (POST)

Response Header - Location will contain the canonical URI

Entity body - the new resource

Example - POST People

301 - Moved Permanently

Response Header - Location will contain the new permanent canonical URI

Entity body - will be empty

Example - GET People/1 has been merged and no longer exists the new URI is People/1

400 - Bad Request: There is an error on the client side.

Response Header - No modification

Entity body - Send back a message on why the request was bad

Example - Malformed XML fails to serialize eon the server

401 - Unauthorized

Response Header - WWW-Authenticate : OAUTH and the challenge or required parameter

Entity body - Optionally, a doc describing the failure

Example - missing OAUTH credentials

403 - Forbidden - Failed authorization

Response Header - No modification

Entity body - explanation of why authorization failed

Example - OAUTH credentials were good, however, the token used for the request was not valid or user associated with the token does not have rights to the resource

404 - Not Found: When a client requests a URI that does not map to a resource on the server

Response Header - No modification

Entity body - will be empty

Example - GET People/1 does not exist, optionally a 410 may be used (see below)

405 - Method not allowed

Response Header - Allow: GET, POST - list the HTTP methods the resource supports

Entity body - will be empty

Example - POST People/1/Status <- an attempt to create a status when only GET is available

409 - Conflict: Client tried to put the servers resource in an impossible or inconsistent state

Response Header - Allow: GET, POST - list the HTTP methods the resource supports

Entity body - will be empty

Example - POST People/1/Status <- an attempt to create a status when only GET is available

410 - Gone: server knew there used to be a resource but it's gone now

Response Header - No modification

Entity body - explanation of conflicts

Example - PUT People/1 modifying "Weblink" credentials for People/1 that are already used by People/2

415 - Unsupported Media Type

Response Header - Code detailed supported media types for the given resource

Entity body - will be empty

Example - Resource only supports XML and JSON, but the client sends application/atom+xm

500 - Internal Server Error

Response Header - No modification

Entity body - will be empty

Example - The server encountered an unexpected condition which prevented it from fulfilling the request.

This response status code map is subject to change, however, in the current build of the API these rules exist as defined above.

Making ASP .NET MVC URIs RESTful

Out of the box the ASP .NET MVC implementation is nice, easy, and elegant. From what I understand they used a few patterns from, the RoR world. One thing that is not part of the default implementation is RESTful routing.

Thanks to the guys who are infinitely smarter than me that did MVCContrib, adding RESTful behavior (using SimplyRestfulRouteHandler) to a .NET MVC app is pretty simple.

1. Download the MVCContrib libraries (I downloaded the source because I like to break stuff)
2. Open up your ASP .NET MVC application and add a reference to the MVCContrib.dll
3. Open up the Global.asax.cs and add the following line inside the ResgisterRoutes method: MvcContrib.SimplyRestful.SimplyRestfulRouteHandler.BuildRoutes(routes);

This is what you get:

Action	URL	Http	Form
Show	[controller]/[id]	GET
Create	[controller]	POST
Update	[controller]/[id]	PUT
Update	[controller]/[id]	POST	PUT
Destroy	[controller]/[id]	DELETE
Destroy	[controller]/[id]	POST	DELETE
Index	[controller]	GET
New	[controller]/new	GET
Edit	[controller]/[id]/edit	GET
Delete	[controller]/[id]/delete	GET

So if we we're dealing with people, for instance, the URI and verb maps might look like:

HTTP verb: [GET] (for show)
URI: https://api.mydomain.com/v1/People/1234

HTTP verb: [PUT] (for update)
URI: https://api.mydomain.com/v1/People/1234

HTTP verb: [POST] (for create)
URI: https://api.mydomain.com/v1/People/

so on...

So what if you wanted to address hierarchy with the object using the same URI scheme? Let's say you wanted to get all of the addresses for a particular person and addresses were a child of person:

HTTP verb: [GET]
URI: https://api.mydomain.com/v1/People/1234/addresses

Thanks to the slick implementation of the RouteHandler all you have to do is add another call to BuildRoutes using an overload that accepts an "areaPrefix"

SimplyRestfulRouteHandler.BuildRoutes(routes, "People/{individualID}");

So now you can use:
https://api.mydomain.com/v1/People/1/addresses <-- collection
https://api.mydomain.com/v1/People/1/addresses/3 <-- single
https://api.mydomain.com/v1/addresses/3 <--single no context

RESTful thoughts - Why ASP .NET MVC?

Let me begin by building the frame of how we approached the idea for a new API before I go into the details of the technologies we chose. We needed an API that would address the following:

1. Secure
2. Follows common community standards / practices
   
3. Public interface
4. Extremely easy to consume
   
5. Supported formats (at least) XML, JSON --> optionally to be extended to XHTML, ATOM, etc...
6. Needs to scale and understand complex relationships
7. Needs to scale fast and with agility
   

The architecture / protocols we chose were - REST(delivery / interaction) + OAUTH(Authentication). I personally feel that this approach is instrumental to the success of your software when you're developing an externally facing API.

Technologies face sweeping changes , patterns / standards change either incrementally or not at all. Fielding did not create REST, he just documented the architectural principles (extremely well). Consider how much simpler it is to develop for a browser when it follows css standards (Firefox, Safari, Opera).

Why REST? The pattern addresses 6(#2-7) of the 7 needs above. Plus, adoption since Fielding's dissertation in 2000, has sky rocketed. Most Web Devs have at least heard of REST and I am pretty sure the majority of the community has knowingly / unknowingly used the pattern.

That guy is old! But he has one job and he does it well... (courtesy: Disney)

Patterns are tried and true - that's why they are patterns. Communities of people come together and agree that pattern x or specification y should be a standard. This blog is my opinion, though I would like to think it is "fact."

The real fact is that there are people out there in the tech pot that are far more intelligent than I am, that are writing specifications that my opinions try to enforce - accurately or inaccurately. Remember the I/O tower in Tron? It had 1 purpose -> communication and apparently it did it well, just look at how old that guy was.

That's how I want this API to turn out, not old and crusty, but constantly predictable. Let me and the patterns worry about the consistency of the API while you enjoy the serendipity of consuming it.

MCP: Ironic that the same acronym is used for a Microsoft Certification?(courtesy: Disney)

Why OAUTH? Look at the majority of web facing, public APIs available to us. Talk about a seriously cool time to be a developer - the days of making an RPC to some arcane server buried at some university are gone. Remember the MCP in Tron? Yes he gobbled up all of the resources and shut down systems, but that's not the point I am trying to make, so we'll ignore that. He acted as the gatekeeper - he was built to be a way to protect and secure stuff, to check and authorize credentials. OAUTH is a lot like that and it is a proven pattern with implementation libraries to boot!

Consumers of my API will not have to wait for me to produce documentation and sample code libraries for some custom authentication implementation I tossed together - they can just head over to oauth.net. OAUTH is an open protocol that has been tested, is proven, and is simple to implement - #1 answered, as well as supporting #2 and #4.

For an API, you don't want to have to introduce anything synonymous to the holly hack or your typical web app "browser test" --> that road will take you to an API that collects dust and not consumers. So, first thing first - standards over technology

Once we selected the standards we started targeting some web frameworks for considerations. Here are some of the technologies that we considered:

Ruby + Framework (Rails, Merb, etc...)
While Rails or Merb we're solid contenders. Rails specifically, I enjoy the opinionated approach, the scaffolding that comes right out of the box, and the DRY that it embraces. I built the 2nd version of the API using Rails but later bailed because implementation would be more difficult considering our existing domain and established environments. If we ever did want to come back around and use Rails or something like it we could do so without impacting the consumers - remember pattern / standards over technology.

The good:

• Opinionated
• Testability
• Elegant, simplistic
• RESTful patterns built in

The bad:

• No established local or prod environments to support the code,
• Existing domain is .net, existing domain / data model is not opinionated - so we would either modify Rails or our model which would exceed our time line to implementation.

ADO.NET Data Services / WCF + RESTful Toolkit
I'll be honest, when I saw Astoria come up on the Microsoft pipe I immediately thought SOAP web services. Was that the right thing to do? No. Did it skew my perception of ADO.NET data services? Yes. So far, regarding some patterns, Microsoft has taken x and added .NET to it. This is OK, in fact it's inevitable - you embrace patterns and are passionate about your technology some of your principles and opinions will become integrated into the pattern.

Sark: Guy got too big for his shoes and got his wig split(courtesy: Disney)

Unfortunately sometimes those opinions cross lines which make it difficult for developers to implement the technology. XML web services as Microsoft SOAP web services, REST as WCF / ADO .NET services.

Last Tron analogy, I promise. Let's talk about Sark? The guy was an overbearing heavy hitter. He wasn't like Tron. Tron played games, that's all he did and he did it extremely well.

End result: he beat the guy who tried to do everything.

The fact is that no piece of software or framework will be everything to everyone - so don't try to make it that. Why does Microsoft have 2 technologies that attempt to do the same thing 2 different ways?

So why not use WCF or ADO .NET WS for a RESTful API? I answered that question with a question: What is it that I am trying to develop?

I simply need a RESTful web application, bottom line.

WCF + RESTful Toolkit
The good:

• It does everything I need and lots more
• It can be web based
• Has a super-cool feature for documentation

The bad:

• It does lots more - too much in fact
• It is highly complex
• It was originally based on SOAP and WS-*
• URIs are difficult to control / manipulate
• HTTP modules are necessary to have extensionless URLs (I am definitely not a fan of .svc)
• According to Flanders' new book RESTful .NET WCF "...actually did have support for building services using REST, but the WCF programming model lacked explicit support for doing so."

WCF, to me, feels like communication between systems was a priority and communication via Hypermedia based systems "RESTfully" is kinda like an add on. I need more focus and less technology getting in my way. I wrote the 1st prototype of the API using WCF, but trashed it because of complexity and overhead.

ADO.NET Data Services

The good:

• Web based
• Microsoft apparently started from scratch on this one - not a Web Services carry over - according to Pablo Castro's Mix08 presentation
• Supports ATOM, json, XML content by default
• Some cool mechanisms for querying
• Wicked easy to extend and develop for / against

The bad:

• Architecture (Data provider and DataSets) - DataSets have historically been bloated. I cannot help but think that there was some sort of pattern carry over from the old Web Services
• I personally do not like the URIs - http://api/service.svc/Users(1)/Bookmarks
• The service as a whole feels "REST-like" and not RESTful
• Things like "$parm", "Expand", etc... tell me that they are trying to be everything to everyone - They cannot use normal request parms because they are tied to LINQ
• URIs are difficult to control / manipulate
• HTTP modules are necessary to have extensionless URLs (I am definitely not a fan of .svc)

I used ADO .NET web services back in the Astoria days, I liked it, I thought it had some really cool features. The technology, however, does not meet my expectations or my RESTful needs

ASP .NET MVC (Beta)

The good:

• Simply, a web application
• I love the MVC pattern
• Easy to develop for
• Easy to consume
• Routes can easily be configured to impose "pure" REST architecture principles - no "REST-like", it is all HTTP - no need for some services framework to lay on top of it and "interpret" requests, low overhead
• No complexities of a service framework
• You can extent request and response formats easily
  
• Most of all - implementing RESTful architecture is simple because ASP .NET MVC at its root is a web based, hypermedia application framework

The bad:

• You have to enforce RESTful principles (unlike Rails) - MVCContrib helps,
• No default documentation or documenting feature (WCF RESTful toolkit has a very nice documentation feature built in)

Tron: Yeah, all I do is play games, but I do it well!(courtesy: Disney)

As I stated above, simple and focused can get you really far down the road. We talked to a lot of people, and did a lot of research, however, most of the above is my opinion. After using APIs that claimed to be RESTful across the web, I realized that the most important thing for this effort would be to stick to simplicity and go pure REST -> ASP .NET MVC fit our needs and wants to get that done.

WS-Deathstar how I hate thee

Long ago, in a web far, far away Microsoft brought us .NET SOAP (JAVA the Hut was also part of the SOAP movement) propagating WS-* ("Deathstar") principles and practices.

The WSDL entered into SOA giving some developers strongly typed web service implementations rapidly. The WSDL was not a requirement to have a SOAP endpoint, but it made it really easy for .NET apps to consume .NET web services.

Anyone who has ever implemented an API and opened it up for public consumption knows that the interface contract typically rules the roost, so-to-speak. It impacts everything from when you version, what is open to the consumers, content returned by your API, and so on...

When the interface changes on your API, with out a solid versioning architecture, your consumers will, and do hurl curses at you and your API (and would probably seek alternatives).

So, back to the "Deathstar" - Years ago I dropped a .NET web service out into the smelting pot to be consumed by all. I made many mistakes but refined / fixed it as much as I could without killing my consumers - WS-* made it pretty easy to make "big mistakes." One such mistake that I made was that I never set the namespace on the service.

[WebService()] <--bad
[WebService(Namespace="http://sub.domain.suffix")] <--good

I basically forgot to do it, and once I realized it - my consumers we're eating tempuri soup and I was feeling a bit like a clown. There is no silver bullet to fix what is currently out there without forcing consumers to change code - I will live with that mistake there to remind me look before I leap.

lesson learned: Set my namespaces on my SOAP web services before they make it to production, or don't use the "Deathstar."

RESTful thoughts - versioning

I've done a lot of thinking on versioning for an API I've been working on. I feel that the mystical nature of API versions should not be the last consideration of an API implementation - why do that to yourself; why do that to your customers?

So how does one approach versioning for the "Programmable Web?" What do your users do when they come to a fork in road?

Approach it using basic considerations of any API architecture (not holding true to Zachman's approach - we'll handle the where, when, and why some other day)

• Who - always consider the consumer. I would consider myself a clown if I didn't listen to the users of the API. Beyond patterns, best practices, and textbook principles - the user has perspective that exists outside of any "should be" reality. They understand the serendipitous side of your API so-to-speak. Side note: This consideration goes way beyond versioning and down to the deep roots of the API's foundation.
  
• How - What are the options. If I have half a dozen choices, what are some proven patterns, what is the industry moving on? For instance, authentication for an API - let's say you have an API that will be used by open-source consumers for social mashups that share resources. The seemingly obvious option would be OAUTH. You don't eat soup with a fork (unless you're eating a soup sandwich) - why should you skip thinking about the appropriate use of technology for an API
  
• What - Typically if it is a web-based API, you'll be dealing with resources - especially when you're dealing with an API based on RESTful principles. What type of payloads are you dealing with? Are you going to implement resource extensions (.json, .xml, etc...)?
  

What are the options:

• URIs
• Custom Headers
• By way of HATEOAS (Hypermedia as the engine of application state - one of the basic proposed constraints of REST, or some other resource driven-hypermedia makeup)
  
• Request definition
• etc...

URIs
http://api.mysite.com/v2/people/1 <- like
http://api2.mysite.com/people/1<- not a fan

Custom Headers
Request.Headers.Add("X_HTTP_MYAPI_VERSION","2.0") <- Though explicit, less visible, would be buried in code

HATEOAS
While, by default the HATEOAS principles are built-in to a RESTful API - think of it like a page on a website and the context of that page tells what can be editied, created, viewed, who you are, etc...

So, in REST one of the parts of this principle is how your application will interpret n * document representations / formats that you API supports.

For instance, I could create a document representation with versioning for People information coming out of my API in an XML format: application/MyAPI-person-v3+xml

I understand the RESTful approach to explicitly seek a resource, it still feels like it is breaking some low level HTTP rule - my take, HTTP has worked this long for a reason, I don't feel very comfortable making up content types for a versioning implementation.

But... media-types seem to be a good option because we are dealing specific resources with specific implementations.

Request Definition
Probably my least favorite. Basically, you put all information about the resource in the parameters (NOT the header) of the request. This approach pretty much does the same thing as the other options, except, makes it more difficult for the programmer to implement. This pattern is commonly used on pin-point APIs where there is only one interface to consume - basically 1 URI for multiple resources - RESTifarians, please don't get upset at me for even mentioning this as an option for RESTful versioning.

So to the RESTful "Who" out there - what are your thoughts on a versioning pattern?