Using Gearman from PHP

I've introduced Gearman into a project I'm working on, and since a few people have asked I thought I'd share my experiences. Basically, this application generates some PDFs from a variety of data sources, makes images, and emails it. Since the whole data processing, image handling, PDF generation process is fairly heavy, I'm putting the requests to generate these onto a gearman queue and having some workers process the jobs. The eventual aim is to bring up EC2 instances (or php-specific cloud hosting perhaps? Recommendations gratefully received!) to do this work but right now I have one worker and it's all on one server.

I installed Gearman and the pecl module for it, and then looked at how to integrate it into my application. I was replacing an existing synchronous script that generated the output, so I already had the code I wanted to be performed by the worker, and since this was a ZF application I set up a CLI bootstrap for it and ran my existing functionality from a controller called that way.

Adding Jobs to Gearman

Adding jobs to the gearman queue was easy, but I struggled to understand which function I needed from the documentation. It turns out that if you just want a job to wait for some resource to be available to do it, then GearmanClient::doBackground() is the way forward. My code to add jobs looks like this:

$client = new GearmanClient();
$client->addServers('127.0.0.1:4730');
$client->doBackground('basicReport', json_encode($job_data));

The $job_data variable holds an array of data that is needed by the worker to know what to generate.

Writing a Gearman Worker in PHP

As I say, I bootstrapped my ZF application from the commandline and created a controller action to be called. It contains code like this:

        $worker = new GearmanWorker();
        $worker->addServer('127.0.0.1');
        $worker->addFunction('basicReport', [callback]);
        while ($worker->work()) {
            echo "working ... \n";ob_flush();
            if (GEARMAN_SUCCESS != $worker->returnCode()) {
                echo "Worker failed: " . $worker->error() . "\n";
            }                                                                                                                                                                       
        }

I named the model class and function in my callback (wrote about callbacks the other day) and the job data that I json_encoded and passed in when I put the job in the queue is passed as the parameter to the callback, which enabled me to basically hook into my original synchronous code and have it pick up where it left off!

I've been very impressed with gearman so far and I'm looking forward to putting the workers onto the cloud and scaling the application as it needs it. As I say, I'm literally using doBackground() to put jobs into a queue for the workers to come along and perform asynchronously, and gearman seems like a great tool for that.

Edit (Feb 2013): This post was edited to reflect changes in the addServers() method in GearmanClient

7 thoughts on “Using Gearman from PHP

  1. Hi Lorna,

    just a thing to say: I really love your kind of posts, because they seem like the little documentation/snippets everyone should daily write down.

  2. odino: Thank you so much for the kind words :) The truth is: I write these things down because otherwise I would not remember for the next time. If they help you too, then that is even better!

  3. Lorna,

    If you're using ZF, I have a fairly simple wrapper that utilizes a custom dispatch and ties into your bootstrap so that you can bootstrap specific items.

    See https://github.com/mwillbanks/Zend_Gearman

    It does need some slight refactoring since a little too much is happening in the constructor but it is quite nice. Been using it in production for over 6 months on a heavy traffic site.

  4. I wrote recently about having gearman in my application, however I have been seeing problems with the long-running PHP worker scripts. My logs had entries like this: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away The work

  5. long running php script may cause some strange thing happened , like memory leak, if you use some
    resource handlers (like mysql connection or file handler), it should be careful considered for these resources are precious , which should be opened only when needed and closed after your works be done! especially in loop statement , there will allocate some memory to current method ,these increasing memory allocation is terrible

  6. Pingback: Any Gearman tutorial? - PHP Solutions - Developers Q & A

Leave a Reply

Please use [code] and [/code] around any source code you wish to share.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>