Mar 19 2008
Yes Simon, We Need a Queue
Simon Wistow asks whether or not we need a better open source queuing system. My answer is yes. I’ve built two one-off hackish queuing systems for $work in the past three years, both of which used database tables as queues, both of which were only accessible by a language-locked api, neither of which was accessible by any sort of neutral service. Both had very specific tasks, hold tasks in a queue for offline (read: outside of a web transaction) processing.
Both jobs could have been fulfilled by a nice message queue service, but it didn’t seem like the effort was worth the payoff. That said, now that I’ve built two of exactly the same thing, I’m 100% of the belief that I’d love to have a easy to install web service-y queue (that isn’t owned by Amazon).
So, why the need? It’s sort of like the evolution of CouchDB & friends I wrote about back in December, all web programmers eventually are expected to be programmers, then they start examining the tools the traditionalists have built, and they question, “why can’t this be better?” Relational databases might not be best for web apps (I’m still not convinced, but for the sake of argument, go for it.) Full featured queue services might not be best for web apps. If for no other reason than they introduce a complicated system into your ecosphere for what is actually a very simple problem. I like Simon’s pointing at memcache and saying, “make a queue that works like that.” That would be pretty cool. I’m voting +1.
That said, like a good American I’m making this vote without fully examining all the candidates. Maybe the existing queues are simple enough…educate me lazy-web.
We wrote a queue for work: http://www.openplans.org/projects/cabochon/ — I know the author looked at ActiveMQ, but I don’t recognize some of the other systems linked to there. We just wanted a simple HTTP-based event queue. All the queue systems I’ve seen seem… well, focused on performance, transactions, etc. I just want to pass around some messages with a simple publish/subscribe model. As casual users the current batch of systems I see all seem quite complex. But maybe there is a less complex subset that would be workable.
Have you seen memcacheq: http://memcachedb.org/memcacheq/ ?