Engineer for Serendipity

The title of this post is a quote from Roy T. Fielding (he had a hand in creating the HTTP 1.1 spec in 1999, basically defined / put into words the architecture and behaviors of the internet as a hypermedia system – REST, and has done about a million other things).

The quote alludes to making your “thing” so that users accidentally find something fortunate. While I do not believe in chance, I do believe in the serendipitous nature of innovation. Because of Fielding and others like him and their "out of the box" thinking we have the internet and all of its wonder.

This video (posted via @digital_nomads) illustrates what happens when people think outside of the web browser or app window and use technologies established over 10 years ago to make people stand and clap. It seems the device in the video uses web based APIs to get data and use it in an unbelievably awesome way.

As you build applications, I challenge you to kick out the ladder and set it on fire, challenge the walls of your device – push against them, bring them down.

If we do our jobs as software developers and API Architects, consumers will find themselves in the middle of serendipity.

RESTful Thoughts - OAUTH implementation for the F1 RESTful API Part 1

We took some time with this implementation and really did some soul searching on Authentication / Authorization mechanisms for the new Fellowship One API. The past few APIs that we've released used credential based authentication ( + signed keys for Payment Gateway) which was appropriate considering the intended usage.

With the RESTful API we know that churches, 3rd parties, vendors, volunteers, and the like will be consuming the API's secure resources - so we chose OAUTH.

The OAUTH features enable all of those consumers to create web sites or applications (consumers) that can access resources available via Fellowship One (Service Provider) securely, and without having to store or gather user credentials.

The benefits of our implementation of OAUTH are:

• Both the church and user will be able to protect / authorize access to their resources
• Resources will only be available to applications that both the church and users of that church has access to (configurable)
• The Consumer will never have access to login information or credentials - so the login is totally owned by the service provider

All requests for resources (excluding tokens and login) must have a Consumer Key, Access Token, and signature.

Roles of our implementation of OAUTH are:

• Service Provider: Fellowship One RESTful API - A web application (API) that allows access via OAuth.
• Consumer: 3rd party / church / Individual - A website or application that uses OAuth to access the Service Provider on behalf of the User.
• User: Portal User, User, Weblink User - An individual who has an account with the Service Provider.

OAUTH Parameter information (from the protocol documents):

• Consumer Key: A value used by the Consumer to identify itself to the Service Provider.
• Consumer Secret: A secret used by the Consumer to establish ownership of the Consumer Key.
• Request Token: A value used by the Consumer to obtain authorization from the User, and exchanged for an Access Token.
• Access Token: A value used by the Consumer to gain access to the Protected Resources on behalf of the User, instead of using the User’s Service Provider credentials.
• Token Secret: A secret used by the Consumer to establish ownership of a given Token.
• OAuth Protocol Parameters: Parameters with names beginning with oauth_.

Brief overview of the F1 OAUTH process (assuming all valid credentials and authority):

• The Consumer application will use a consumer key and secret to sign all URIs so that the F1 API will be able to verify the consumers identity and credentials.
• The Consumer application first requests a "Request Token" to use when the Users access the login pages -> The Service provider grants the request token.
• The Consumer application will direct the User to the Service Provider to obtain User Authorization
• The Service Provider will obtain authorization from the User and direct the User back to the Consumer
• The Consumer application will then request an Access Token -> The Service provider grants the request token and marks the "Content-Location" header with a URI to the User information.
• The Consumer application will use the Access Token to access Protected Resources

The process defined above requires accessing the following URIs:

Request token URI:
http://{churchcode}.{domain}/{version}/Tokens/RequestToken [POST]

Login URIs:
Portal Users http://{churchcode}.{domain}/{version}/PortalUser/Login [GET]
Weblink Users http://{churchcode}.{domain}/{version}/WeblinkUser/Login [GET]
Groups Users http://{churchcode}.{domain}/{version}/User/Login [GET]

Access Token URI:
http://{churchcode}.{domain}/{version}/Tokens/AccessToken [POST]

RESTful Thoughts - Planned HTTP Headers for the upcoming RESTful API

Here is the HTTP Header map for the upcoming Fellowship One RESTful API. Please comment / ask questions / make suggestions.

Just like the response codes, many decisions made for this map came from RFC2616, Fielding's dissertation, and analysis of several dozen RESTful or REST "like" APIs currently on the programmable web.


Request Headers

Accept : type/subtype

Example - Accept : application/xml (Available types and sub types: application/xml, application/json)

Notes - Defaults to application/xml if one is not passed in, and when a user sends application/* the resulting content sub type will be XML. Multiple types/sub types can be sent in but as per the specification the first match in degree of specificity will be the first used.
i.e. if user sends in: application/*, application/xml, */* the order would be as follows:
1) application/xml
2) application/*
3) */*

If an invalid value is passed in via Accept header the response will result in a 415 Unsupported Media Type

Accept-Charset : charset

Example - Accept-Charset: utf-8

Notes - Accept-Charset will not be used and will always default to utf-8

Accept-Encoding : encoding

Example - Accept-Encoding : deflate, compress, gzip

Notes - If none is sent in then the API will default to no compression.
If an invalid value is passed in via Accept-Encoding header the response will result in a 406 Not acceptable

Authorization : credentials

Example - Authorization : Oauth realm=http://api.f1.com

Notes - This is where the OAUTH credentials will go

Date : HTTP-date

Example - Date: Thu, 29 Jan 2009 15:28:25 GMT

Notes - Optional, most commonly passed in with PUTs and POSTs

User-Agent : product/comment

Example - User-Agent : MyChurchSite/v123.x

Notes - Used to identify the HTTP library or client library that was used to consume the API

Response Headers

Allow : method

Example - Allow: GET, HEAD, PUT

Notes - An Allow header field MUST be present in a 405 (Method Not Allowed) response.

Content-Encoding : content-coding

Example - Content-Encoding : gzip

Notes - This will tell the client what type of compression was used on the resource

Content-Length : DIGIT

Example - Content-Length : 1254

Notes - Sent back with each request. Will possibly be available via HEAD requests

Content-Location : absoluteURI | relativeURI

Example - Content-Location : http://api.f1.com/API/People/22114944

Notes - Sent back with each GET request

Content-Type : media-type

Example - Content-Type : application/xml, utf-8

Notes - Details the type of content being returned to the client

Date : HTTP-date

Example - Date: Thu, 29 Jan 2009 15:28:25 GMT

Notes - Will be returned with every response, possibly excluding responses returning status codes of 500

Location : absoluteURI

Example - Location: http://api.f1.com/API/People/22114944

Notes - Applies to 201 and 301 only

WWW-Authenticate : challenge

Example - WWW-Authenticate : Oauth realm: http://api.f1.com

Notes - The URI will change per environment, per version

Notes on caching
All headers having to do with caching are still being implemented.

This HTTP header map is subject to change, however, in the current build of the API these rules exist as defined.

RESTful Thoughts - Planned response status codes for the upcoming RESTful API

In the interest of honoring standards here is the response status code map for the upcoming Fellowship One RESTful API. Please comment / ask questions / make suggestions.

Many decisions made for this map came from RFC2616, Fielding's dissertation, and analysis of several dozen RESTful or REST "like" APIs currently on the programmable web.

200 - ok (GETs, PUTs, and DELETEs)

Entity body - The resource

Example - GET People/1

201 - Resource created (POST)

Response Header - Location will contain the canonical URI

Entity body - the new resource

Example - POST People

301 - Moved Permanently

Response Header - Location will contain the new permanent canonical URI

Entity body - will be empty

Example - GET People/1 has been merged and no longer exists the new URI is People/1

400 - Bad Request: There is an error on the client side.

Response Header - No modification

Entity body - Send back a message on why the request was bad

Example - Malformed XML fails to serialize eon the server

401 - Unauthorized

Response Header - WWW-Authenticate : OAUTH and the challenge or required parameter

Entity body - Optionally, a doc describing the failure

Example - missing OAUTH credentials

403 - Forbidden - Failed authorization

Response Header - No modification

Entity body - explanation of why authorization failed

Example - OAUTH credentials were good, however, the token used for the request was not valid or user associated with the token does not have rights to the resource

404 - Not Found: When a client requests a URI that does not map to a resource on the server

Response Header - No modification

Entity body - will be empty

Example - GET People/1 does not exist, optionally a 410 may be used (see below)

405 - Method not allowed

Response Header - Allow: GET, POST - list the HTTP methods the resource supports

Entity body - will be empty

Example - POST People/1/Status <- an attempt to create a status when only GET is available

409 - Conflict: Client tried to put the servers resource in an impossible or inconsistent state

Response Header - Allow: GET, POST - list the HTTP methods the resource supports

Entity body - will be empty

Example - POST People/1/Status <- an attempt to create a status when only GET is available

410 - Gone: server knew there used to be a resource but it's gone now

Response Header - No modification

Entity body - explanation of conflicts

Example - PUT People/1 modifying "Weblink" credentials for People/1 that are already used by People/2

415 - Unsupported Media Type

Response Header - Code detailed supported media types for the given resource

Entity body - will be empty

Example - Resource only supports XML and JSON, but the client sends application/atom+xm

500 - Internal Server Error

Response Header - No modification

Entity body - will be empty

Example - The server encountered an unexpected condition which prevented it from fulfilling the request.

This response status code map is subject to change, however, in the current build of the API these rules exist as defined above.

Web dev - Back to the basics

Some of the most important aspects of creating web based API, or any web based application is messaging and behavior - user experience. Back in 1999 when the HTTP 1.1 protocol was defined and the HTML 4.0.1 spec was released the web began to get more lively.

People stopped using World Wide Web and AOL interchangeably and we all jumped in our web dev boots, and rushed to inflate the bubble that would burst only a few years later.

courtesy of: Dana Summers, The Orlando Sentinel

Ironically, with web usage up, daily life being immersed in web jargon, popups, ads, and UI travesties, standards seemed to become less important.

All of those RFCs written by Fielding and gang seemed to melt away and get replaced by blinking text and Internet Explorer 5.0. I know I am generalizing; there we're a few outliers that challenged the norm and chose standards like the Netscape, but it seemed for the most part we used the web to showcase our madd skillz with using bloated back ground images and auto playing midi files for each "lucky" visitor <- I was, indeed, one of the worst violators of midi web attacks. The rule breaking was not limited to UI though, many services misused protocols and redefined established patterns in order to fit the web into antiquated business models.

Maybe that history muddied the standards / protocol waters; I mean shouldn't every web developer / architect know how HTTP works? What if we as developers simply relied on a tool or framework like visual studio and .NET to handle all of that crazy HTTP stuff for us? Would you hire a "Web developer" that did not know what the difference between a GET and a POST was?

I was blessed to have had an opportunity to go to MSDN Dev Con yesterday. I was sitting in a session ASP .NET 4.0 road map given by Ben Scheirman. He said something during his presentation that struck a web cord with me, I'll paraphrase:

With ASP .NET MVC, Microsoft looked at other MVC based frameworks and put together a solid tool to let developers do what they want to do and let the .net framework help and not get in the way.

For so long now, many .net developers have been using tools to write their HTML and CSS for them. We have ignored the basics. I chose ASP .NET MVC because of the flexibility of the pattern (not because of the Microsoft technology) and I truly enjoy the fact that standards and practices whet into the making of it (begin soapbox - I do not like how you can decorate your actions with HTTP verbs - would I ever do a post on a show? If so, wouldn't I be breaking the convention of the pattern? - end soapbox).

I find myself having to go back to things like RFC2616, Fielding's dissertation (when working on the RESTful API), and even the HTML spec daily. I find those tools far more valuable than an IDE or a managed framework.

Fearlessly tweak IIS

I find that I wreck my development machine often, either because I over-tweak or because I a messing with something that I have never messed with before.

IIS on Vista / Server 2008 is one of those things where messing with it outside of the configuration interface can really jack things up - almost to the point of having to reinstall IIS or even your OS.

I came across this a while back (I can't remember the source, so credit can go to the tech fairy) - before messing with IIS (even through the configuration panel) you should backup the configuration files IIS uses - here's how...

Let's go really low-tech and create a simple bat file to get the job done:

1. Create a new bat file and name it something like: backupiis.bat
2. Put the following code in it (changing the value of DestPath) -
   SET DestPath=e:\backups\

@For /F "tokens=2,3,4 delims=/ " %%A in ('Date /t') do @(
Set Month=%%A
Set Day=%%B
Set Year=%%C
)

copy C:\Windows\Microsoft.NET\Framework\v2.0.50727\CONFIG\machine.config %DestPath%machine.config_%Year%%Month%%Day%
copy C:\Windows\System32\inetsrv\config\applicationHost.config %DestPath%applicationHost.config_%Year%%Month%%Day%
3. Execute by double clicking on the bat file or setup a scheduled task

Now you can fearlessly go tweak IIS!

Final thoughts and upgrades on the Dell Mini 9

The Mini seems to be an overall win for my wife. She's really been enjoying it.

Her ups are that she can take it any where she goes, she can actually use it while holding the baby, she gets "oohs" and "ahhs" everywhere she goes - people find it hard to believe that she just pulled a full blown computer out of her purse, it's comfortable to her - she is not bothered by heat or weight when setting it in her lap, after the upgrades I made (more below) she says that the speed and multitask-ability is out of the park!

Her downs are sometimes when she gets into a typing grove she mistypes due to the slightly smaller keys and the screen resolution has taken her a little time to get used to (she is accustomed to 1024 x 768 and not the current max 1024 x 600).

Here's the story:
After it arrived on Dec. 23 I knew it would be pretty cool to transfer all of her docs, and settings from her old laptop (D600) to the Mini, wrap it up and stick it under the tree. So that night I whipped out her old box, the Mini, and a usb DVD drive and got to work.

Initially when the Mini booted I was somewhat impressed. After going through setting up the user, computer name, network, etc... the thing would boot up in around ~10seconds, and would finish loading apps, anti virus, etc.. in upwards of 30seconds - not too bad.

I installed Office 07, configured the local network shares (transferred docs and pst files from network shares), setup her email, and then started the windows updates. Prior to the updates I noticed that the transfer of many files (her entire my docs directory) over wireless to our NAS was significantly faster than even the m90 I use for work. I cracked it open and looked at the wireless card - nothing special just an 802.11g card. I never did any benchmarks, by my impression is that the file writes and reads over the network are fast (possibly the new Atom processor, or SSD and it was simply writing faster to the media - I don't know, and I really don't care).

I was disappointed with the number of windows updates I had to do. The installed configuration was Windows XP sp3 and Office 07 retail. I ended up having to download somewhere in the ball park of 70+ updates coming in at 600~700mb total! This wasn't a major problem for me since we have FIOS and the updates we're downloaded almost instantly but the reboots for the updates and shear number was disconcerting - I can't imagine the frustration of a Cable or DSL user. Dell and Microsoft need to work on this one, whether its an additional CD packaged with the product when the updates reach a certain number, or the image Dell uses is updated more frequently - not sure, I am just guessing there must be a better solution.

After all of the housekeeping was done I cleaned up the loose / temp files installed firefox, configured the box to hibernate on lid close, and wrapped it up.

She was excited when she opened it, her first words about it, "this thing looks sweet." After some usage she was feeling the pinch of her usage patterns pushing the limits of the netbook. It began to crumble over her 1gb archive of email and Windows was constantly having to adjust the pagefile size because of virtual memory usage.

I began questioning if I had made the right decision, maybe I should have gotten her a Dell Studio or a something else. I did a brief bit of research and discovered one of the major bottle necks of many of the netbooks is the write speed of the SSD and multitaskers should have as much memory as possible (side note, the tech specs on Dell's site says Up to 1gb - the current configuration I have is running and using 2gb).

Thanks to mydigitaldiscount and the jkkmobile blog all of my problems when away for $140. I bought the new RunCore PCI-e 32gb SSD and a 2gb stick of Transcend memory. With the help of the awesome configuration of the RunCore drive and the Acronis trial I had her drive cloned in 10 minutes.

I first bench marked the current drive that came with the mini - 16gb SSD by STEC. I was disappointed in both the reads and the writes with this drive. It's seems that Dell might have skimped on this, I am not sure why though - I personally feel that this is one of the more important areas of the unit. At least give me an option on the site to upgrade to a high performance SSD when I am buying the Mini (this btw is my only gripe about the Mini 9 - everything else has far exceeded my expectations).

Dell really out did themselves with the hardware accessibility of this unit. You can flip the unit over and pop off the back panel (roughly 3in x 4in opening). There you have access to WLAN, Wireless, Memory, and SSD - perfect!

I dropped in the new SSD and memory closed it up - all in about 5 minutes. I booted up the Mini and voila - an exact copy of the drive minutes before, running perfectly - all done in under 20 minutes.

The benchmarks on the new RunCore SSD we're impressive and crushed the SSD from STEC that came with the Mini.

(Standard 16gb SSD that ships with the Mini)

(RunCore 32gb SSD purchased for $119.00)

Final thoughts:
Initially I was concerned about getting a netbook but, during my research when considering what to get her based on her needs the Mini 9 kept coming up. The configuration seemed solid and the size is a perfect fit for her. She delights in simplistic things - "don't give me something I don't need, I want to get stuff done, plain and simple. If it gets in my way of getting things done, I get rid of it..."

If you are an "on-the-go" user, a digital nomad of sorts and you live in email, the web, and in spreadsheets than this is the box for you. She has zero problems with photo and movie editing as well (especially after the upgrades). It is rare that a tool shows up in the tech smelting pot that actually makes the user who moves through insane situations and crazy environments more effective - I believe this is one of those tools. It's an elegant, fast, no frills solution to the person with the high demand life where every second counts.

Well done Dell, though your flagship laptop line is the Precision series - I really do think this one will change / impact more lives. My wife so far has been relishing in the benefits of using this computer in all areas of our lives. Get the SDD right, or at least provide an option and you'll own the netbook market.

Go to my Flickr photostream to see all of the photos from the upgrade.