PHP and Gearman: Unable to connect after upgrade

I upgraded PHP and related pecl modules on my development machine today, and ran into a problem with Gearman. Actually I ran into more than one! Firstly the challenge of getting the newest pecl version working with a gearman version. Then an error where my existing PHP application couldn't connect to gearman after upgrade.
Continue reading

Gearman Priorities And Persistent Storage

I have been writing a bit about Gearman lately, including installing it for PHP and Ubuntu, actually using it from PHP and also how I use persistent storage with Gearman. I'm moving on to look at adding jobs of different priorities.

I use Gearman entirely as a point to introduce asynchronous-ness in my application. There is a complicated and image-heavy PDF to generate and this happens on an automated schedule. To do this, I use the GearmanClient::doBackground method. This inserts a priority 1 job into my queue.

Using the doHighBackground() and the doLowBackground() methods insert jobs into the queue and checking out my persistent storage I see that the priorities work like this:

priority method
0 doHighBackground()
1 doBackground()
2 doLowBackground()

Gearman works out which task is the next highest priority and will hand it to the next available worker - which means that I can set my automated reporting lower priority than the reports requested by real live people wanting them now, and everyone is happy!

Using Persistent Storage with Gearman

I'm using gearman for the first time in a new project, and two things in particular were bothering me. Firstly, there doesn't seem to be a built-in way to see what's in the queue. Secondly, if the gearman server dies (which seemed quite likely when I was first getting to grips with this stuff and writing really buggy code!) you lose your queue. Therefore I decided that I would switch gearman over to running with persistent storage. Continue reading

Dealing with MySQL Gone Away in Zend Framework

I wrote recently about having gearman in my application, however I have been seeing problems with the long-running PHP worker scripts. My logs had entries like this:

SQLSTATE[HY000]: General error: 2006 MySQL server has gone away

The worker is a Zend Framework application, run from the CLI, and it seemed like the Zend_Db_Adapter had no way of knowing when MySQL had let go of its end of the connection. I tried a few different things, including Zend_Db_Adapter::getConnection(), but without success - until I dug through the source code (with some help from a friend) and realised that ZF was not reconnecting at all if it thought it already had a connection. So instead, I expressly disconnected and reconnected the database handler. At bootstrap time, I place my database handle into the registry, so I simply added this at the start of the actual function that the gearman worker calls:

$db = Zend_Registry::get('db');
$db->getConnection();

At the end of my script, before it returns to the loop waiting for another gearman job, I just disconnect my database:

$db->closeConnection();

Now Zend_Db_Adapter knows that when I ask it to connect, it needs to go off and make a new connection, and everything works really well! I was seeing the errors because I'm still only testing the system so it can go days between getting any new jobs, and the timeout on MySQL is shorter than that.

Using Gearman from PHP

I've introduced Gearman into a project I'm working on, and since a few people have asked I thought I'd share my experiences. Basically, this application generates some PDFs from a variety of data sources, makes images, and emails it. Since the whole data processing, image handling, PDF generation process is fairly heavy, I'm putting the requests to generate these onto a gearman queue and having some workers process the jobs. The eventual aim is to bring up EC2 instances (or php-specific cloud hosting perhaps? Recommendations gratefully received!) to do this work but right now I have one worker and it's all on one server.

Continue reading

Installing Gearman for PHP and Ubuntu

I've been using Gearman lately in a project that I'm working on, and of course a month later when I came to deploy the code, I had to look up all over again what was required for a gearman server in order to put it on the new platform. Here is the short version for my future reference (and yours, if you like)

Continue reading