PHPUnit with Zend_Controller_Action_Helper

I’m currently working on a REST service built in Zend Framework and I ran into problems very quickly – when I tried to write the first unit test for the first action in fact! PHPUnit was just dying when I asked it to dispatch() any URL which didn’t return HTML, it wasn’t even giving its usual output.

What was actually happening was I was making use of Zend_Controller_Action_Helper_Json, my service returns JSON and this action helper takes input, transforms it into JSON, sets the content-type correctly and tells ZF not to look for a view since we don’t need one. I thought this was pretty neat.

But look at the start of the declaration for the action helper:

class Zend_Controller_Action_Helper_Json extends Zend_Controller_Action_Helper_Abstract
{
    /**
     * Suppress exit when sendJson() called
     * @var boolean
     */
    public $suppressExit = false;
...

By default the JSON action helper will exit() – which of course when run from phpunit causes that to exit as well! There is the class variable there so it was simple to turn off – I just extended my class and changed the value of that $suppressExit variable.

class Zend_Controller_Action_Helper_ServiceJson extends Zend_Controller_Action_Helper_Json {
    public $suppressExit = true;
}

My tests now run successfully and I can build my application, hopefully next time I’ll realise what I’m doing wrong faster!

PUTting data fields with PHP cURL

This is a little post about how to PUT multiple data fields using the PHP cURL extension. Why I wanted to do this in the first place is beyond the scope of this post, since its quite a long story. The curl command line allows data fields to be sent with a PUT request, and I wanted to do the same from PHP. Here is a snippet of code to show how I did it.

        $data = array("a" => $a);
        $ch = curl_init($this->_serviceUrl . $id);

        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PUT");
        curl_setopt($ch, CURLOPT_POSTFIELDS,http_build_query($data));

        $response = curl_exec($ch);
        if(!$response) {
            return false;
        }

I’m putting this here so I remember for next time – if it helps you as well then even better :)

Status Codes for Web Services

This the last entry in a mini series on points to consider when designing and building web services. So far there have been posts on error feedback for web services, auth mechanisms for web services, saving state in web services and using version parameters with web services.

Unlike the other posts in this series, this one is quite specific to one type of service – REST – since it deals with status codes, specifically HTTP ones. The ideas are transferrable however and other types of service can return statuses in a similar way.

There’s a few key things to think about when returning status codes. In earlier posts in this series these was discussion of using existing application framework to serve pages and changing the output mechanism accordingly. Usually a web page will return a status 200 for OK or also 302 for found, so this is fine when things are working normally. But when things aren’t going quite so well, its useful to give alternative feedback that can be easily picked up by a client application.

When things go wrong there are a couple of different schools of thought of how the service should respond. One is that if, for example, the user supplies data which fails validation, the service could provide the OK response and a message to the user to let them know what needs validating – exactly as we’d return information messages to a user filling in a form. To be considered restful however, the service should more correctly return one of the “400” status codes, which means that the client made an error. Interesting and useful codes* in this series are:

  • 401 Unauthorized
  • 403 Forbidden
  • 404 Not Found
  • 405 Method Not Allowed
  • 406 Not Acceptable
  • 408 Request Timeout
  • 417 Expectation Failed
  • 418 I’m a teapot

* I didn’t say they were both useful and interesting

Using descriptive status codes allows the client to get the headline of the problem without having to parse a whole request to find out whether it is good or not. HTTP already has this feature built-in, and so we make use of it (HTTP is pretty cool really, makes a great protocol for services!).

Where an error occurs on the server side – it is usual to return a 500 error or another in the 500 series. This lets the client know there is a problem outside of their control; it is useful to include information about whether the client should retry and when. Having a defined protocol for retries helps avoid the situation where a system comes back up only to fall over again with all the traffic from people retrying every minute (or other interval) – this is a real concern for systems that are under heavy load.

Status codes are like a headline to the calling entity about what happened, and are a valuable tool in the web service toolkit. For bonus points, leave me a comment and tell me which is your favourite status code :)

Version Parameters for Web Services

This is a mini series on points to consider when designing and building web services. So far there have been posts on error feedback for web services, auth mechanisms for web services and saving state in web services.

When designing a service API, there are lots of things you can do right, and plenty of pitfalls. Most of both of these are completely specific to the situation you are designing for but I have one tip that has helped me out in a number of scenarios: Include a version parameter with every method call.

This is invaluable, not just in development where you can increment when you change the API (which could be quite often!) but also in production. For example if you want to extend or alter an existing service, you can identify which version of the API the client thinks it is accessing and behave accordingly – either letting them know there’s a new version, or preserving previous behaviours. Its never ideal to change the API of an existing service but sometimes it makes more sense to do so, especially if its just to include an additional parameter or something else quite minor, but which does cause an API change.

When you publish your shiny new service which does everything you need (including the hoovering), it might seem a bit redundant to require a field which is always set to the string “1.0” … but in 18 months time, you’ll be patting yourself on the back.

Saving State in a Web Service

This is a mini series on points to consider when designing and building web services. So far there have been posts on error feedback for web services and auth mechanisms for web services.

Web services, by their very natures, are stateless, and this is no different to any other web application with a frontend. Its often helpful to keep track of the sequence of events something experiences. Similarly we might want to store the details a particular piece of data encounters on its lifetime in the system. When it was created, when it was changed, where in a given process it is up to, and so on.

A good example of this is a service I run for a few friends which accepts a URL, requests it and takes a screenshot, then turns that into a thumbnail and makes it available. There’s a few definite states in that sequence and its better to store which it is than try to guess from which fields have dates in or something. So the states are something like “new”, “error”, “in progress”, “ready” and “expired”. I can look at any of the jobs that come in, at any time, and know exactly where that item is up to.

When Things Go Wrong

It’s also pretty to have a strategy for failure. I already talked about giving users sensible feedback from a service, but the service also needs to keep track of what’s happening. There are decisions to make which depend entirely on the application under consideration but for example whether a failure is terminal, or whether the system should keep trying is quite an important thing to consider. Perhaps you just want to flag records that failed, so you can check on them later. Or you might want to retry – a set number of times? Or at decreasing intervals? An undesirable outcome would be to have the same record submitted every minute (or whatever) forever, so its worth planning to avoid this eventuality. Think also about the likely causes for failure – a split-second glitch or a likely human error which might take a day or two to be noticed and rectified?

Logging

Logging is something which is ideally of zero value. If nothing ever went wrong, you’d never care to look at an individual record and dig into its journey. However in the real world, we do need to be able to debug systems, sometimes under pressure. With logging there are a few different choices – whether to log to file or database, and whether to log continuously, or only at given times. Times when you might want to enable or increase logging are in development and when tracing a bug. However having some background amount of logging (and a way to stop the archives becoming too large) is recommended to keep track of who did what and when. Bear it in mind for a service just as you would for a web application.

Architecting Web Services – FOWA Tour

I spoke at the FOWA Tour event in Leeds yesterday, as their local speaker (they set aside a slot for each place they go). It was a great event, good atmosphere and I really enjoyed the both crowd and content. I can’t imagine what the logistics for something like this are like but it seemed like everything was going pretty smoothly.

My talk was “Architecting Web Services” – just a half-hour slot to give an overview of what services are and take a look at some things to bear in mind when designing them. The slides are on slideshare so feel free to take a look and let me know if you have any comments or questions.

I had a great time meeting up with so many friends – old and new – yesterday. Hope that I’ll see some of those again at future events or online. Anyone for Bar Camp Leeds?

Error Feedback for Web Services

I have been thinking, writing and speaking a little bit about web services recently, starting with a few thoughts on authorisation mechanisms for services. Today we’ll look at another really important aspect of authoring web services, and one feature that will definitely get used – error handling and feedback! Having clear and robust error handling in your service will help those trying to consume it immeasurably. Nothing is more annoying that impenetrable errors, unclear responses, or a service which accepts your input but then turns out not to have done what you expected. And I say that from experience.

Stacks of Errors

What’s more annoying than getting an error from a web service? Getting another error from the service every time you fix the first one!

Most services have a few steps of checking incoming variables. Checking that the user has supplied all required fields, and that all incoming fields are of the required format, and that the data they refer to does actually exist – there’s a lot going on here. Too many systems take fright at the first sight of an error, and return straight to the user like a child reporting a sibling’s misdeeds to a parent. I mean something along these lines:

if(!isset($_POST['username'])) {
  return 'username is missing!';
}
if(!isset($_POST['password'])) {
  return 'password is missing!';
}
foreach($_POST as $key => $value) {
  $expected_fields = array(
    "username",
    "password"
  );
  if(!in_array($key,$expected_fields)) {
    return "unexpected field ".$key;
  }
}

What’s more useful is if the user can have a better overall view of everything that is going wrong, since often they might be caused by the same misunderstanding or perhaps be related to one another. So I’m considering code that looks more like this:

$messages = array();
$can_proceed = true;

if(!isset($_POST['username'])) {
  $messages[] = 'username is missing!';
  $can_proceed = false;
}
if(!isset($_POST['password'])) {
  $messages[] = 'password is missing!';
  $can_proceed = false;
}
foreach($_POST as $key => $value) {
  $expected_fields = array(
    "username",
    "password"
  );
  if(!in_array($key,$expected_fields)) {
    $messages[] = "unexpected field ".$key;
    $can_proceed = false;
  }
}

if(!$can_proceed) {
  return $messages;
}

The nice thing about something like this is you’ll see a series of messages where there are problems – so when you mis-spell a field name, you’ll see the “missing field” message for a field you know you are sending, but you’ll also see the “unexpected field” message and hopefully that will make it easier to spot what’s gone amiss.

Error format

Its tempting to return error information in a completely different format, after all it is quite different from the request that probably would have been returned from a correct request. Some web service protocols dictate how errors should be sent – SOAP has the soap-error response, for example. But for something where we have more control, such as an RPC style or REST service, we can choose. Usually I think its appropriate to return an appropriate status code (for REST) or wrapper (for RPC) and then include the error information in the same format as the response would have been. This is mostly for ease of consuming the response, saving clients from having to parse an additional format!

Approaching Errors

Having malformed input to services is inevitable – through misunderstandings, typos, and of course rubbish input by users. Making sure that all these eventualities are gracefully caught and information returned to the user means that the user stands a much better chance of being able to interact successfully. If only the success case works, but the service either doesn’t respond, returns nonsense (or worse, misleading information!), or appears to work but actually hasn’t, your users won’t hang out for long to work out why.

I’ve covered some really basic ideas here but I’m sure there are plenty of other nice ways to help guide users – feel free to add comments for the features you implement in your systems to help users when things aren’t going quite right!

By giving more information to users, it becomes much easier for them to develop against your service quickly and easily – and its not much more to add on the service side.

Speaking at FOWA Tour, Leeds, May 28th

I’m surprised and delighted to announce that I will be one of the speakers at the rather excellent FOWA Tour even in Leeds on 28th May (two weeks today!). The day is like a roadshow version of the famous FOWA conferences, with a morning of (free!) cloud workshops and an afternoon of excellent speakers. I had been hoping to attend this event anyway so to get the invite to speak is very exciting indeed!

My talk is entitled “Architecting Web Services” and will take a look at the types of service available, and identifying functions which would be useful as a service. I’ll run through how decisions about service and format can be taken and we’ll look at some examples of these while we’re at it. I’ve been building a few services lately for one thing and another and I’m pretty excited about the whole topic! Luckily I only have a half-hour on stage but rest assured I’ll probably still be talking about it over drinks later in the day!

Anyone going? Thinking of going? Let me know and make sure to come and say hi on the day :)

Auth Mechanisms for Web Services

Having been involved in quite a few service-related activities in the last year or so, I’ve been having a few thoughts about what I’ve learned from this and what decisions I make when designing a service. Hopefully there will be a few of these posts – but to start with, I’m considering the options for authorising users.

Quite a lot of services don’t require any authentication at all, similar to quite a lot of the web. In either setting, the information is there for users to consume when they want. However the difference comes when services start doing more than making data available. If changes can be effected by the service, then we need to identify who is requesting the change.

Traditional websites use a username and password, and we can do exactly the same thing here. Services work on a series of discrete requests and its common to require that the username and password be supplied with every request. However for high-load services or where a particularly fast response time is needed, we can use something similar to sessions, where the user first has to authenticate and is given a token. On subsequent requests they supply the token and we wave them through without requiring their credentials again.

There are a number of considerations involved in deciding whether this setup can work for a particular application:

  • Does it take time to authenticate? For example is there an external system to wait for or lots of user information to retrieve?
  • How guessable is the token? Any kind of reasonable length hashing will help you here. I tend to use salted md5 tokens*.
  • How long will the token be valid for? If interaction with the service is likely to be a burst of related requests, you might allow validity for 30 minutes for example.
  • Will you require other identifying information as well as the token? For example you might require that the user also supply their username, which would have to match the token. I’ve also seen systems which only accept tokens from the same user ip address as the user’s original authentication call came from.

Also think about storing these tokens. They can go in their own table along with the information you want to use frequently – this is the same idea as storing information about a user in a session, for example. So user id, maybe display name plus the token itself, some information about when it was created or when it expires, and anything else that will be needed to check the token’s validity. With this information being independent and just used to verify the user, there is also the option of storing this in an alternative, faster, mechanism such as memcache.

This isn’t by any means everything there is to think of, but just some ideas of things to consider when designing a service.

* I blogged about salting md5s in PHP recently, if you are interested

PHP Advent Article Published

I was wildly excited a few weeks ago to receive an email inviting me to be one of the contributors to this year’s PHP Advent. Actually the biggest kick was seeing my name in a list of PHP luminaries! The article was published today, you can find it at http://phpadvent.org/2008/which-web-service-by-lorna-mitchell – its a short overview of various types of web services around with some pros and cons about when each is useful. To be included in the PHP Advent project has been huge fun, in fact I’m so delighted that I’ve broken one of my usual rules and blogged twice in one day :)