What Does URI Stand For?

I get a lot of complaints about an API that I maintain (http://api.joind.in) which is "missing" the ID field. This ID field is the database's primary key; if the user doesn't have access to the database (they don't), then it seems to me that the primary key probably isn't all that useful.

Instead, the API publishes each record with a unique uri field. If this record is referred to by another record, then this full identifier will be used in every case. If this record should be included in a collection, this exact same identifier will be used there, too. You can reach the resource directly by requesting its URI. In the same way that we might refer to a website by its URL, we refer to records in RESTful systems by their URI*. If you need to store these somewhere for your own use, you can use whatever key you like with the local storage, you may even choose to use the uri field as it is unique.

* URI stands for Unique Resource Identifier

Are Subqueries RESTful?

Twitter is great for one-liners, but it's very difficult to carry on any kind of advanced conversation there. Therefore when I saw this tweet yesterday, I knew I'd be picking a different medium to reply:

The blog seems like a good place, as I can put examples and all kinds other things here, and waffle at length (which is really why I like it!). Because when condensed to tweet form, the answer is really "it depends".

The Problem(s)

REST is all about representations of resources. They might come in different formats, and they might appear at their own URI as well as in one or more collections, but essentially you just get a representation of a thing. This is great, apart from when it isn't.

  • What if you want a smaller result set with only a limited number of fields?
  • What if you want related data? For every resource in a collection?

Continue reading

Five Clues That Your API isn't RESTful

I get a lot of emails asking me to get involved with API projects, and that means I see a lot of both implemented and planned "RESTful" APIs. Now, I absolutely love REST and for a data-driven application, it would be my first choice. A service of some other description may work better for other scenarios or skill sets, and non-RESTful services can be very, very useful. If you tell me that your service is RESTful, then I expect it to be. If you're not sure, look out for these clues:

It has a single endpoint

I don't really care what else is going on in your API, any "RESTful" API which has a statement such as "all requests are made to http://example.com/rest" is ... not RESTful. REST is all about handling representations of resources, each is represented by its own URI and we operate directly on that. If it looks like "pretty URLs", then it's probably along the right lines. Continue reading

Building A RESTful PHP Server: Understanding the Request

Once upon a time, what seems like a lifetime ago, I was away for a couple of weeks, and I wrote a series of posts about serving RESTful APIs from PHP to keep my blog going while I was away. Fast forward a few years and those posts are outdated and still wildly popular - so I thought it was about time I revisited this and showed how I'm writing RESTful PHP servers today!

In the first part of this (probably) 3-part series, we'll begin with the basics. It might seem boring, but the most important thing to get right with REST is parsing all the various elements of the HTTP request and responding accordingly. I've put in code samples from from a small-scale toy project I created to make me think about the steps involved (should I put the code somewhere so you can see it? Let me know). Without further ado, let's dive in and begin by sending all requests through one bootstrap script: Continue reading

GETting RESTful collections - may I filter?

At work at Ibuildings recently, I've been teaching some classes on web services, and its a topic that I've spoken about once or twice at conferences. But something has always bothered me, so I find myself in the unusual position of blogging a question.

RESTful collections

So, when you are retrieving information from a RESTful service. You have two options: retrieve a specific resource, whose URL you know; or retrieve a collection, which may contain a list of resources. I've also seen some nice ways of filtering collections, by creating kind of "sub collections" or "views", similar to what twitter does with the URL of lists, for example http://twitter.com/dpcon/speakers10 which is like a filtered list of twitter users.

What if I want to search and sort?

Is it RESTful to add GET parameters to a collection in order to add functionality such as filtering, sorting, or pagination? What I have in mind is a URL that looks something like this:

  • http://example.com/users?orderby=firstname&start=0
  • http://example.com/users?start=0&limit=25
  • http://example.com/users?active=1&orderby=join_date&limit=12

This is what I would do with a search results page in a web application, and I use the same approach to web services which works really well and I recommend it to everyone! But is it RESTful?

I am also wondering where OpenSearch would fit into the answer for all this, I only noticed it recently but the more I look at it the more I think it could be an interesting addition!

Thoughts, links for me to RTFM, and all other additions are welcome in the comments box :)

Adding PUT variables to Request Object in Zend Framework

When I wrote recently about testing web services within Zend Framework, I missed out a really key piece of information! I didn't explain how I was reading the PUT vars in my controller code in the first place - so I'll rectify that omission now.

Its very simple: I have extended Zend_Controller_Action with my own, and all controllers inherit from here. This has a routeAction() which grabs the incoming variables from a PUT request and sets them as parameters within the usual $this->getRequest() scope, then forwards on the request. Here is my class:

class My_Controller_Action extends Zend_Controller_Action
{
    public function routeAction()    
    {
        // handle grabbing PUT vars
        if($this->getRequest()->isPut()) {
            parse_str($this->getRequest()->getRawBody(), $params);
            foreach($params as $key => $value) {
                $this->getRequest()->setParam($key, $value);
            }
        }
        $this->_forward(strtolower($this->getRequest()->getMethod()));
    }
}

So in my controller code, I simply call out to $this->getRequest()->getParam('name') or whatever, and in my tests I can set those parameters as I showed in my other article.

I hope this makes sense, its one of those things I set up once and use a lot (and now I'll be able to refer to how I did it!), if you have any queries, comments, improvements or if this helps you then please leave a comment - I haven't come across anyone else doing anything similar but I know there must be, so let me know!

Using Zend_Test for Web Services

Recently I had cause to develop a web service and so I wrote some tests to go along with it - or I was about to. When I looked at the asserts available in Zend_Test, they were all geared towards HTML/CSS output - but we can test just as effectively on another output.

Using Zend_Test, I set up the request object I wanted to send, and dispatched it. Then I retrieved the body and, since this service returns JSON, I json_decoded it. This gives me an object - and I can go ahead and use all the functionality of PHPUnit, with or without Zend_Test's additions, to test my service. Its perhaps easiest to show this in a few steps.

Setting up the request object

The idea here is that you set up any parameters you need to including the HTTP verb to use and cookies if needed, then despatch the request. Here's a few examples, first a simple GET method, with a cookie.

        $request = $this->getRequest();
        $request->setMethod('GET');
        $request->setCookie('token','xxxx');
        $this->dispatch('/user/24');

Including data with a POST request:

        $request = $this->getRequest();
        $request->setMethod('POST');
        $request->setPost(array(
            'name' => 'new user',
            'organisation' => 49
            ));
        $this->dispatch('/user');

This is a REST service, so I also tested PUT and DELETE methods. DELETE just needs the setMethod() call since it doesn't have any data with it, but PUT was a bit trickier - here's an example of what I used:

        $request = $this->getRequest();
        $request->setMethod('PUT');
        $params = array('name' => 'Harry Potter');
        $request->setRawBody(http_build_query($params));
        $this->dispatch('/user/48');

Decoding the Response

This is the easy part, all I do is check the status code is what was expected, and then decode the response. My web service returns JSON so this part of each test looks something like:

        $this->assertResponseCode('200');
 
        $response = json_decode($this->getResponse()->getBody());

Testing the content of the response

Here we get into classic PHPUnit territory and simply use the assertTrue and assertEquals calls we'd use when testing anything else, an example is included for completeness:

        $this->assertEquals($response->contentType, 'user');
        $this->assertTrue(is_numeric($response->id));
        $this->assertEquals($response->name, 'new user');
        $this->assertEquals($response->organisation, 49);

In Conclusion

By combining the request/response awareness of Zend_Test with standard PHPUnit testing strategies, its easy to test web services as well as web pages. I hope the examples are helpful - if they help you or if you have anything to add, then leave a comment!

Status Codes for Web Services

This the last entry in a mini series on points to consider when designing and building web services. So far there have been posts on error feedback for web services, auth mechanisms for web services, saving state in web services and using version parameters with web services.

Unlike the other posts in this series, this one is quite specific to one type of service - REST - since it deals with status codes, specifically HTTP ones. The ideas are transferrable however and other types of service can return statuses in a similar way.

There's a few key things to think about when returning status codes. In earlier posts in this series these was discussion of using existing application framework to serve pages and changing the output mechanism accordingly. Usually a web page will return a status 200 for OK or also 302 for found, so this is fine when things are working normally. But when things aren't going quite so well, its useful to give alternative feedback that can be easily picked up by a client application.

When things go wrong there are a couple of different schools of thought of how the service should respond. One is that if, for example, the user supplies data which fails validation, the service could provide the OK response and a message to the user to let them know what needs validating - exactly as we'd return information messages to a user filling in a form. To be considered restful however, the service should more correctly return one of the "400" status codes, which means that the client made an error. Interesting and useful codes* in this series are:

  • 401 Unauthorized
  • 403 Forbidden
  • 404 Not Found
  • 405 Method Not Allowed
  • 406 Not Acceptable
  • 408 Request Timeout
  • 417 Expectation Failed
  • 418 I'm a teapot

* I didn't say they were both useful and interesting

Using descriptive status codes allows the client to get the headline of the problem without having to parse a whole request to find out whether it is good or not. HTTP already has this feature built-in, and so we make use of it (HTTP is pretty cool really, makes a great protocol for services!).

Where an error occurs on the server side - it is usual to return a 500 error or another in the 500 series. This lets the client know there is a problem outside of their control; it is useful to include information about whether the client should retry and when. Having a defined protocol for retries helps avoid the situation where a system comes back up only to fall over again with all the traffic from people retrying every minute (or other interval) - this is a real concern for systems that are under heavy load.

Status codes are like a headline to the calling entity about what happened, and are a valuable tool in the web service toolkit. For bonus points, leave me a comment and tell me which is your favourite status code :)

Error Feedback for Web Services

I have been thinking, writing and speaking a little bit about web services recently, starting with a few thoughts on authorisation mechanisms for services. Today we'll look at another really important aspect of authoring web services, and one feature that will definitely get used - error handling and feedback! Having clear and robust error handling in your service will help those trying to consume it immeasurably. Nothing is more annoying that impenetrable errors, unclear responses, or a service which accepts your input but then turns out not to have done what you expected. And I say that from experience.

Stacks of Errors

What's more annoying than getting an error from a web service? Getting another error from the service every time you fix the first one!

Most services have a few steps of checking incoming variables. Checking that the user has supplied all required fields, and that all incoming fields are of the required format, and that the data they refer to does actually exist - there's a lot going on here. Too many systems take fright at the first sight of an error, and return straight to the user like a child reporting a sibling's misdeeds to a parent. I mean something along these lines:

if(!isset($_POST['username'])) {
  return 'username is missing!';
}
if(!isset($_POST['password'])) {
  return 'password is missing!';
}
foreach($_POST as $key => $value) {
  $expected_fields = array(
    "username",
    "password"
  );
  if(!in_array($key,$expected_fields)) {
    return "unexpected field ".$key;
  }
}

What's more useful is if the user can have a better overall view of everything that is going wrong, since often they might be caused by the same misunderstanding or perhaps be related to one another. So I'm considering code that looks more like this:

$messages = array();
$can_proceed = true;
 
if(!isset($_POST['username'])) {
  $messages[] = 'username is missing!';
  $can_proceed = false;
}
if(!isset($_POST['password'])) {
  $messages[] = 'password is missing!';
  $can_proceed = false;
}
foreach($_POST as $key => $value) {
  $expected_fields = array(
    "username",
    "password"
  );
  if(!in_array($key,$expected_fields)) {
    $messages[] = "unexpected field ".$key;
    $can_proceed = false;
  }
}
 
if(!$can_proceed) {
  return $messages;
}

The nice thing about something like this is you'll see a series of messages where there are problems - so when you mis-spell a field name, you'll see the "missing field" message for a field you know you are sending, but you'll also see the "unexpected field" message and hopefully that will make it easier to spot what's gone amiss.

Error format

Its tempting to return error information in a completely different format, after all it is quite different from the request that probably would have been returned from a correct request. Some web service protocols dictate how errors should be sent - SOAP has the soap-error response, for example. But for something where we have more control, such as an RPC style or REST service, we can choose. Usually I think its appropriate to return an appropriate status code (for REST) or wrapper (for RPC) and then include the error information in the same format as the response would have been. This is mostly for ease of consuming the response, saving clients from having to parse an additional format!

Approaching Errors

Having malformed input to services is inevitable - through misunderstandings, typos, and of course rubbish input by users. Making sure that all these eventualities are gracefully caught and information returned to the user means that the user stands a much better chance of being able to interact successfully. If only the success case works, but the service either doesn't respond, returns nonsense (or worse, misleading information!), or appears to work but actually hasn't, your users won't hang out for long to work out why.

I've covered some really basic ideas here but I'm sure there are plenty of other nice ways to help guide users - feel free to add comments for the features you implement in your systems to help users when things aren't going quite right!

By giving more information to users, it becomes much easier for them to develop against your service quickly and easily - and its not much more to add on the service side.