Lonely Lion

Simon Wistow asks whether or not we need a better open source queuing system. My answer is yes. I’ve built two one-off hackish queuing systems for $work in the past three years, both of which used database tables as queues, both of which were only accessible by a language-locked api, neither of which was accessible by any sort of neutral service. Both had very specific tasks, hold tasks in a queue for offline (read: outside of a web transaction) processing.

Both jobs could have been fulfilled by a nice message queue service, but it didn’t seem like the effort was worth the payoff. That said, now that I’ve built two of exactly the same thing, I’m 100% of the belief that I’d love to have a easy to install web service-y queue (that isn’t owned by Amazon).

So, why the need? It’s sort of like the evolution of CouchDB & friends I wrote about back in December, all web programmers eventually are expected to be programmers, then they start examining the tools the traditionalists have built, and they question, “why can’t this be better?” Relational databases might not be best for web apps (I’m still not convinced, but for the sake of argument, go for it.) Full featured queue services might not be best for web apps. If for no other reason than they introduce a complicated system into your ecosphere for what is actually a very simple problem. I like Simon’s pointing at memcache and saying, “make a queue that works like that.” That would be pretty cool. I’m voting +1.

That said, like a good American I’m making this vote without fully examining all the candidates. Maybe the existing queues are simple enough…educate me lazy-web.

Amazon is starting a limited beta of an online database they’re calling SimpleDB. It’s functionally very similar to CouchDB, in that it uses REST for queries and inserts, stores data in flat “schema-less” pages, and is generally geared towards web-app-like data. The NYTimes DBSlayer takes a half-fast approach by layering a JSON REST API on top of an existing relational database.

Although these three projects each take a different stab at the same problem, they all agree on one thing, “you should really put DB in your name…somewhere.”

Amazon SimpleDB solves the problem that a lot of EC2 explorers end up running into the minute they want to have persistent data across sessions. Because EC2 is a fresh slate on each reboot, typical databases aren’t really usable, unless you spend an awful lot of time figuring out how to off-load your data to Amazon S3 on a regular basis.

I wish I had some sort of pithy analysis of the recent leanings by web-visionaries away from SQL databases and towards these sort of REST db’s. The timeline in my head goes something like this:

Raw SQL begat ORMs begat full stack web frameworks begat wrapping a REST interface around an ORM begat “hey, let’s have our controllers talk to the REST interface, rather than the ORM” begat “hey, why do we need an ORM, or a traditional database? Let’s just make a REST accessible database!”

I guess that’s relatively pithy.

Really though, I’m not entirely sure what to make of this “trend.” I’ve yet to build even a trivial project using any of the three (Amazon is a closed beta, so I’m not sure if it’ll even be a possibility anytime soon). Maybe it’s time I take a crack at one of them.

Just a quick response to this write up on the lack of growth in Python 3000. Python 3000 really isn’t the giant sea change that I thought it was going to be. Most of the significant work is under the covers, optimizations that will benefit me without me necessarily knowing it. The author of the above post (I’m sorry, I couldn’t find the author’s name), like me, is a fan of Ruby and Python. I think he wants Python 3000 to take on some of the traits we both love about Ruby (extreme flexibility, cleverness, many ways to do it). I know where he’s coming from, but that wouldn’t be very ‘Pythonic,’ it would be ‘Rubyonic.’

One of my favorite bits about being a polyglot is the differences between languages. What’s the point of making Python like Ruby? What’s the point of making Perl like Python? If everyone is sort of doing the same thing, where’s the fun in learning new stuff? I love Python, I love Ruby, I even love Perl. Let’s preserve their differences. Python doesn’t want you to muck around with the syntax, Ruby does. Use Ruby when appropriate, use Python when appropriate. Right?

Speaking of polyglotism, I bought that Prag-Prog Errrrrlang book over the weekend. All variables are constant, and must start with a capital letter. How’s that for balls nuts? Woo hoo! Viva la Different!