GETting RESTful collections - may I filter?
Tuesday, March 23. 2010
At work at Ibuildings recently, I've been teaching some classes on web services, and its a topic that I've spoken about once or twice at conferences. But something has always bothered me, so I find myself in the unusual position of blogging a question.
So, when you are retrieving information from a RESTful service. You have two options: retrieve a specific resource, whose URL you know; or retrieve a collection, which may contain a list of resources. I've also seen some nice ways of filtering collections, by creating kind of "sub collections" or "views", similar to what twitter does with the URL of lists, for example http://twitter.com/dpcon/speakers10 which is like a filtered list of twitter users.
Is it RESTful to add GET parameters to a collection in order to add functionality such as filtering, sorting, or pagination? What I have in mind is a URL that looks something like this:
This is what I would do with a search results page in a web application, and I use the same approach to web services which works really well and I recommend it to everyone! But is it RESTful?
I am also wondering where OpenSearch would fit into the answer for all this, I only noticed it recently but the more I look at it the more I think it could be an interesting addition!
Thoughts, links for me to RTFM, and all other additions are welcome in the comments box :)
RESTful collections
So, when you are retrieving information from a RESTful service. You have two options: retrieve a specific resource, whose URL you know; or retrieve a collection, which may contain a list of resources. I've also seen some nice ways of filtering collections, by creating kind of "sub collections" or "views", similar to what twitter does with the URL of lists, for example http://twitter.com/dpcon/speakers10 which is like a filtered list of twitter users.
What if I want to search and sort?
Is it RESTful to add GET parameters to a collection in order to add functionality such as filtering, sorting, or pagination? What I have in mind is a URL that looks something like this:
- http://example.com/users?orderby=firstname&start=0
- http://example.com/users?start=0&limit=25
- http://example.com/users?active=1&orderby=join_date&limit=12
This is what I would do with a search results page in a web application, and I use the same approach to web services which works really well and I recommend it to everyone! But is it RESTful?
I am also wondering where OpenSearch would fit into the answer for all this, I only noticed it recently but the more I look at it the more I think it could be an interesting addition!
Thoughts, links for me to RTFM, and all other additions are welcome in the comments box :)
Adding PUT variables to Request Object in Zend Framework
Monday, August 17. 2009
When I wrote recently about testing web services within Zend Framework, I missed out a really key piece of information! I didn't explain how I was reading the PUT vars in my controller code in the first place - so I'll rectify that omission now.
Its very simple: I have extended Zend_Controller_Action with my own, and all controllers inherit from here. This has a routeAction() which grabs the incoming variables from a PUT request and sets them as parameters within the usual $this->getRequest() scope, then forwards on the request. Here is my class:
class My_Controller_Action extends Zend_Controller_Action
{
public function routeAction()
{
// handle grabbing PUT vars
if($this->getRequest()->isPut()) {
parse_str($this->getRequest()->getRawBody(), $params);
foreach($params as $key => $value) {
$this->getRequest()->setParam($key, $value);
}
}
$this->_forward(strtolower($this->getRequest()->getMethod()));
}
}
So in my controller code, I simply call out to $this->getRequest()->getParam('name') or whatever, and in my tests I can set those parameters as I showed in my other article.
I hope this makes sense, its one of those things I set up once and use a lot (and now I'll be able to refer to how I did it!), if you have any queries, comments, improvements or if this helps you then please leave a comment - I haven't come across anyone else doing anything similar but I know there must be, so let me know!
Its very simple: I have extended Zend_Controller_Action with my own, and all controllers inherit from here. This has a routeAction() which grabs the incoming variables from a PUT request and sets them as parameters within the usual $this->getRequest() scope, then forwards on the request. Here is my class:
class My_Controller_Action extends Zend_Controller_Action
{
public function routeAction()
{
// handle grabbing PUT vars
if($this->getRequest()->isPut()) {
parse_str($this->getRequest()->getRawBody(), $params);
foreach($params as $key => $value) {
$this->getRequest()->setParam($key, $value);
}
}
$this->_forward(strtolower($this->getRequest()->getMethod()));
}
}
So in my controller code, I simply call out to $this->getRequest()->getParam('name') or whatever, and in my tests I can set those parameters as I showed in my other article.
I hope this makes sense, its one of those things I set up once and use a lot (and now I'll be able to refer to how I did it!), if you have any queries, comments, improvements or if this helps you then please leave a comment - I haven't come across anyone else doing anything similar but I know there must be, so let me know!
Using Zend_Test for Web Services
Thursday, August 13. 2009
Recently I had cause to develop a web service and so I wrote some tests to go along with it - or I was about to. When I looked at the asserts available in Zend_Test, they were all geared towards HTML/CSS output - but we can test just as effectively on another output.
Using Zend_Test, I set up the request object I wanted to send, and dispatched it. Then I retrieved the body and, since this service returns JSON, I json_decoded it. This gives me an object - and I can go ahead and use all the functionality of PHPUnit, with or without Zend_Test's additions, to test my service. Its perhaps easiest to show this in a few steps.
The idea here is that you set up any parameters you need to including the HTTP verb to use and cookies if needed, then despatch the request. Here's a few examples, first a simple GET method, with a cookie.
$request = $this->getRequest();
$request->setMethod('GET');
$request->setCookie('token','xxxx');
$this->dispatch('/user/24');
Including data with a POST request:
$request = $this->getRequest();
$request->setMethod('POST');
$request->setPost(array(
'name' => 'new user',
'organisation' => 49
));
$this->dispatch('/user');
This is a REST service, so I also tested PUT and DELETE methods. DELETE just needs the setMethod() call since it doesn't have any data with it, but PUT was a bit trickier - here's an example of what I used:
$request = $this->getRequest();
$request->setMethod('PUT');
$params = array('name' => 'Harry Potter');
$request->setRawBody(http_build_query($params));
$this->dispatch('/user/48');
This is the easy part, all I do is check the status code is what was expected, and then decode the response. My web service returns JSON so this part of each test looks something like:
$this->assertResponseCode('200');
$response = json_decode($this->getResponse()->getBody());
Here we get into classic PHPUnit territory and simply use the assertTrue and assertEquals calls we'd use when testing anything else, an example is included for completeness:
$this->assertEquals($response->contentType, 'user');
$this->assertTrue(is_numeric($response->id));
$this->assertEquals($response->name, 'new user');
$this->assertEquals($response->organisation, 49);
By combining the request/response awareness of Zend_Test with standard PHPUnit testing strategies, its easy to test web services as well as web pages. I hope the examples are helpful - if they help you or if you have anything to add, then leave a comment!
Using Zend_Test, I set up the request object I wanted to send, and dispatched it. Then I retrieved the body and, since this service returns JSON, I json_decoded it. This gives me an object - and I can go ahead and use all the functionality of PHPUnit, with or without Zend_Test's additions, to test my service. Its perhaps easiest to show this in a few steps.
Setting up the request object
The idea here is that you set up any parameters you need to including the HTTP verb to use and cookies if needed, then despatch the request. Here's a few examples, first a simple GET method, with a cookie.
$request = $this->getRequest();
$request->setMethod('GET');
$request->setCookie('token','xxxx');
$this->dispatch('/user/24');
Including data with a POST request:
$request = $this->getRequest();
$request->setMethod('POST');
$request->setPost(array(
'name' => 'new user',
'organisation' => 49
));
$this->dispatch('/user');
This is a REST service, so I also tested PUT and DELETE methods. DELETE just needs the setMethod() call since it doesn't have any data with it, but PUT was a bit trickier - here's an example of what I used:
$request = $this->getRequest();
$request->setMethod('PUT');
$params = array('name' => 'Harry Potter');
$request->setRawBody(http_build_query($params));
$this->dispatch('/user/48');
Decoding the Response
This is the easy part, all I do is check the status code is what was expected, and then decode the response. My web service returns JSON so this part of each test looks something like:
$this->assertResponseCode('200');
$response = json_decode($this->getResponse()->getBody());
Testing the content of the response
Here we get into classic PHPUnit territory and simply use the assertTrue and assertEquals calls we'd use when testing anything else, an example is included for completeness:
$this->assertEquals($response->contentType, 'user');
$this->assertTrue(is_numeric($response->id));
$this->assertEquals($response->name, 'new user');
$this->assertEquals($response->organisation, 49);
In Conclusion
By combining the request/response awareness of Zend_Test with standard PHPUnit testing strategies, its easy to test web services as well as web pages. I hope the examples are helpful - if they help you or if you have anything to add, then leave a comment!
Status Codes for Web Services
Thursday, July 2. 2009
This the last entry in a mini series on points to consider when designing and building web services. So far there have been posts on error feedback for web services, auth mechanisms for web services, saving state in web services and using version parameters with web services.
Unlike the other posts in this series, this one is quite specific to one type of service - REST - since it deals with status codes, specifically HTTP ones. The ideas are transferrable however and other types of service can return statuses in a similar way.
There's a few key things to think about when returning status codes. In earlier posts in this series these was discussion of using existing application framework to serve pages and changing the output mechanism accordingly. Usually a web page will return a status 200 for OK or also 302 for found, so this is fine when things are working normally. But when things aren't going quite so well, its useful to give alternative feedback that can be easily picked up by a client application.
When things go wrong there are a couple of different schools of thought of how the service should respond. One is that if, for example, the user supplies data which fails validation, the service could provide the OK response and a message to the user to let them know what needs validating - exactly as we'd return information messages to a user filling in a form. To be considered restful however, the service should more correctly return one of the "400" status codes, which means that the client made an error. Interesting and useful codes* in this series are:
* I didn't say they were both useful and interesting
Using descriptive status codes allows the client to get the headline of the problem without having to parse a whole request to find out whether it is good or not. HTTP already has this feature built-in, and so we make use of it (HTTP is pretty cool really, makes a great protocol for services!).
Where an error occurs on the server side - it is usual to return a 500 error or another in the 500 series. This lets the client know there is a problem outside of their control; it is useful to include information about whether the client should retry and when. Having a defined protocol for retries helps avoid the situation where a system comes back up only to fall over again with all the traffic from people retrying every minute (or other interval) - this is a real concern for systems that are under heavy load.
Status codes are like a headline to the calling entity about what happened, and are a valuable tool in the web service toolkit. For bonus points, leave me a comment and tell me which is your favourite status code :)
Unlike the other posts in this series, this one is quite specific to one type of service - REST - since it deals with status codes, specifically HTTP ones. The ideas are transferrable however and other types of service can return statuses in a similar way.
There's a few key things to think about when returning status codes. In earlier posts in this series these was discussion of using existing application framework to serve pages and changing the output mechanism accordingly. Usually a web page will return a status 200 for OK or also 302 for found, so this is fine when things are working normally. But when things aren't going quite so well, its useful to give alternative feedback that can be easily picked up by a client application.
When things go wrong there are a couple of different schools of thought of how the service should respond. One is that if, for example, the user supplies data which fails validation, the service could provide the OK response and a message to the user to let them know what needs validating - exactly as we'd return information messages to a user filling in a form. To be considered restful however, the service should more correctly return one of the "400" status codes, which means that the client made an error. Interesting and useful codes* in this series are:
- 401 Unauthorized
- 403 Forbidden
- 404 Not Found
- 405 Method Not Allowed
- 406 Not Acceptable
- 408 Request Timeout
- 417 Expectation Failed
- 418 I'm a teapot
* I didn't say they were both useful and interesting
Using descriptive status codes allows the client to get the headline of the problem without having to parse a whole request to find out whether it is good or not. HTTP already has this feature built-in, and so we make use of it (HTTP is pretty cool really, makes a great protocol for services!).
Where an error occurs on the server side - it is usual to return a 500 error or another in the 500 series. This lets the client know there is a problem outside of their control; it is useful to include information about whether the client should retry and when. Having a defined protocol for retries helps avoid the situation where a system comes back up only to fall over again with all the traffic from people retrying every minute (or other interval) - this is a real concern for systems that are under heavy load.
Status codes are like a headline to the calling entity about what happened, and are a valuable tool in the web service toolkit. For bonus points, leave me a comment and tell me which is your favourite status code :)
Error Feedback for Web Services
Monday, May 18. 2009
I have been thinking, writing and speaking a little bit about web services recently, starting with a few thoughts on authorisation mechanisms for services. Today we'll look at another really important aspect of authoring web services, and one feature that will definitely get used - error handling and feedback! Having clear and robust error handling in your service will help those trying to consume it immeasurably. Nothing is more annoying that impenetrable errors, unclear responses, or a service which accepts your input but then turns out not to have done what you expected. And I say that from experience.
What's more annoying than getting an error from a web service? Getting another error from the service every time you fix the first one!
Most services have a few steps of checking incoming variables. Checking that the user has supplied all required fields, and that all incoming fields are of the required format, and that the data they refer to does actually exist - there's a lot going on here. Too many systems take fright at the first sight of an error, and return straight to the user like a child reporting a sibling's misdeeds to a parent. I mean something along these lines:
if(!isset($_POST['username'])) {
return 'username is missing!';
}
if(!isset($_POST['password'])) {
return 'password is missing!';
}
foreach($_POST as $key => $value) {
$expected_fields = array(
"username",
"password"
);
if(!in_array($key,$expected_fields)) {
return "unexpected field ".$key;
}
}
What's more useful is if the user can have a better overall view of everything that is going wrong, since often they might be caused by the same misunderstanding or perhaps be related to one another. So I'm considering code that looks more like this:
$messages = array();
$can_proceed = true;
if(!isset($_POST['username'])) {
$messages[] = 'username is missing!';
$can_proceed = false;
}
if(!isset($_POST['password'])) {
$messages[] = 'password is missing!';
$can_proceed = false;
}
foreach($_POST as $key => $value) {
$expected_fields = array(
"username",
"password"
);
if(!in_array($key,$expected_fields)) {
$messages[] = "unexpected field ".$key;
$can_proceed = false;
}
}
if(!$can_proceed) {
return $messages;
}
The nice thing about something like this is you'll see a series of messages where there are problems - so when you mis-spell a field name, you'll see the "missing field" message for a field you know you are sending, but you'll also see the "unexpected field" message and hopefully that will make it easier to spot what's gone amiss.
Its tempting to return error information in a completely different format, after all it is quite different from the request that probably would have been returned from a correct request. Some web service protocols dictate how errors should be sent - SOAP has the soap-error response, for example. But for something where we have more control, such as an RPC style or REST service, we can choose. Usually I think its appropriate to return an appropriate status code (for REST) or wrapper (for RPC) and then include the error information in the same format as the response would have been. This is mostly for ease of consuming the response, saving clients from having to parse an additional format!
Having malformed input to services is inevitable - through misunderstandings, typos, and of course rubbish input by users. Making sure that all these eventualities are gracefully caught and information returned to the user means that the user stands a much better chance of being able to interact successfully. If only the success case works, but the service either doesn't respond, returns nonsense (or worse, misleading information!), or appears to work but actually hasn't, your users won't hang out for long to work out why.
I've covered some really basic ideas here but I'm sure there are plenty of other nice ways to help guide users - feel free to add comments for the features you implement in your systems to help users when things aren't going quite right!
By giving more information to users, it becomes much easier for them to develop against your service quickly and easily - and its not much more to add on the service side.
Stacks of Errors
What's more annoying than getting an error from a web service? Getting another error from the service every time you fix the first one!
Most services have a few steps of checking incoming variables. Checking that the user has supplied all required fields, and that all incoming fields are of the required format, and that the data they refer to does actually exist - there's a lot going on here. Too many systems take fright at the first sight of an error, and return straight to the user like a child reporting a sibling's misdeeds to a parent. I mean something along these lines:
if(!isset($_POST['username'])) {
return 'username is missing!';
}
if(!isset($_POST['password'])) {
return 'password is missing!';
}
foreach($_POST as $key => $value) {
$expected_fields = array(
"username",
"password"
);
if(!in_array($key,$expected_fields)) {
return "unexpected field ".$key;
}
}
What's more useful is if the user can have a better overall view of everything that is going wrong, since often they might be caused by the same misunderstanding or perhaps be related to one another. So I'm considering code that looks more like this:
$messages = array();
$can_proceed = true;
if(!isset($_POST['username'])) {
$messages[] = 'username is missing!';
$can_proceed = false;
}
if(!isset($_POST['password'])) {
$messages[] = 'password is missing!';
$can_proceed = false;
}
foreach($_POST as $key => $value) {
$expected_fields = array(
"username",
"password"
);
if(!in_array($key,$expected_fields)) {
$messages[] = "unexpected field ".$key;
$can_proceed = false;
}
}
if(!$can_proceed) {
return $messages;
}
The nice thing about something like this is you'll see a series of messages where there are problems - so when you mis-spell a field name, you'll see the "missing field" message for a field you know you are sending, but you'll also see the "unexpected field" message and hopefully that will make it easier to spot what's gone amiss.
Error format
Its tempting to return error information in a completely different format, after all it is quite different from the request that probably would have been returned from a correct request. Some web service protocols dictate how errors should be sent - SOAP has the soap-error response, for example. But for something where we have more control, such as an RPC style or REST service, we can choose. Usually I think its appropriate to return an appropriate status code (for REST) or wrapper (for RPC) and then include the error information in the same format as the response would have been. This is mostly for ease of consuming the response, saving clients from having to parse an additional format!
Approaching Errors
Having malformed input to services is inevitable - through misunderstandings, typos, and of course rubbish input by users. Making sure that all these eventualities are gracefully caught and information returned to the user means that the user stands a much better chance of being able to interact successfully. If only the success case works, but the service either doesn't respond, returns nonsense (or worse, misleading information!), or appears to work but actually hasn't, your users won't hang out for long to work out why.
I've covered some really basic ideas here but I'm sure there are plenty of other nice ways to help guide users - feel free to add comments for the features you implement in your systems to help users when things aren't going quite right!
By giving more information to users, it becomes much easier for them to develop against your service quickly and easily - and its not much more to add on the service side.
Using curl and PHP to talk to a REST service
Monday, September 15. 2008
Having recently written articles about curl and about writing a PHP REST server, I thought I'd complete the circle and put here a few notes on using PHP's curl wrapper as a Rest client. Also I had to look some of this up when I needed to actually do it so next time I need only look in one place!
If you don't know about PHP, Rest, or curl, then I recommend you do a little reading around each of those subjects before reading this as its unlikely to make much sense - I'm not including background on these topics as there are better resources elsewhere.
I've written about using curl before from the command line, but this example uses PHP's curl to access the service. This is just a simple example, but hopefully if you are doing something along these lines you can adapt for your needs.
In the example, we set the URL we'd like to call and initialise the curl object to point to that. Then we create an array of post data, and configure curl to use POST to make the request and to use our data array.
$service_url = 'http://example.com/rest/user/';
$curl = curl_init($service_url);
$curl_post_data = array(
"user_id" => 42,
"emailaddress" => 'lorna@example.com',
);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_POST, true);
curl_setopt($curl, CURLOPT_POSTFIELDS, $curl_post_data);
$curl_response = curl_exec($curl);
curl_close($curl);
$xml = new SimpleXMLElement($curl_response);
We execute the request and capture the response. Health warning: by default curl will echo the response (for reasons that aren't clear to me), if you want to parse it then you will need to use the CURLOPT_RETURNTRANSFER flag to have curl return the response rather than a boolean indication of success - this fooled me completely for a while, I have no idea why it works this way. As you can see I've parsed the resulting XML and my script can then continue with the data it acquired. Depending what you need to do next, you can manipulate the SimpleXMLElement object as you need to.
If you're working with PHP and services, then hopefully this will get you started, if you have any questions or comments, then please add them below!
If you don't know about PHP, Rest, or curl, then I recommend you do a little reading around each of those subjects before reading this as its unlikely to make much sense - I'm not including background on these topics as there are better resources elsewhere.
I've written about using curl before from the command line, but this example uses PHP's curl to access the service. This is just a simple example, but hopefully if you are doing something along these lines you can adapt for your needs.
In the example, we set the URL we'd like to call and initialise the curl object to point to that. Then we create an array of post data, and configure curl to use POST to make the request and to use our data array.
$service_url = 'http://example.com/rest/user/';
$curl = curl_init($service_url);
$curl_post_data = array(
"user_id" => 42,
"emailaddress" => 'lorna@example.com',
);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_POST, true);
curl_setopt($curl, CURLOPT_POSTFIELDS, $curl_post_data);
$curl_response = curl_exec($curl);
curl_close($curl);
$xml = new SimpleXMLElement($curl_response);
We execute the request and capture the response. Health warning: by default curl will echo the response (for reasons that aren't clear to me), if you want to parse it then you will need to use the CURLOPT_RETURNTRANSFER flag to have curl return the response rather than a boolean indication of success - this fooled me completely for a while, I have no idea why it works this way. As you can see I've parsed the resulting XML and my script can then continue with the data it acquired. Depending what you need to do next, you can manipulate the SimpleXMLElement object as you need to.
If you're working with PHP and services, then hopefully this will get you started, if you have any questions or comments, then please add them below!
(Page 1 of 2, totaling 10 entries)
» next page
Comments