Lonely Lion

I’ve been getting the word out about Threadless’ move to Python.  Here’s my keynote at Pycon in Hot-lanta,

And my keynote at Flourish here in Chi-town,

Both talks went well, I’m a little embarrassed that I got a little swear-y in the name of “edginess” at Pycon.  I’ve given plenty of talks in the past, but it’s relatively new to me to be recorded while I’m talking.  Going back to watch the talks again is…weird.  Anywho, note to myself: regardless of what the crowd in front of you will tolerate, if there’s a camera pointing at you, don’t tell swear jokes. That said, the convore reviews were mostly positive.

Last week I gave an Amazon EC2 presentation at the Day of Cloud conference here in Chicago. My slides from the presentation cover some basics about EC2. It was a good day, lots of good speakers, and lots of attendees that were interested in getting their apps into the mysterious clooooouuudd.

The kicker is, I spent a fair amount of time explaining how to set up a relational database on EC2, just in time for Amazon to announce that they’re releasing very easy to use MySQL instances. Luckily, I have a half hour tomorrow at the Chicago Google Tools User Group to revisit the Amazon talk. I’ll revise the talk to focus on Amazon’s new offerings, specifically how to get a reasonable web app up on Amazon using RDS as a database back end.

I understand the irony of giving the talk at a Google tools group, but it can’t hurt to know how to move your applications off of AppEngine and on to other services. It’s all part of Data Liberation, right?

Now, with video:

Well…it’s either a game changer, or it will end up being really popular in Brazil.  I’m leaning towards the game changer.  Really, it’s about time, it’s good to see a big company being ambitious instead of just making a new thing that’s just like the old thing, and they’re open sourcing it.

Imagine a big cast iron pot, throw in Appengine, XMPP, Blogs, Wikis, stir, simmer, and serve.  Wave!  Some particularly interesting bits from the API preview doc:

You define the behavior of your robot by defining the events which you wish your robot to be notified. Wave contacts the robot whenever one of these events occur, such as a change made to a wave in which the robot is a participant.

I spent an absurd amount of time hand crafting an email-bot a few years ago that could respond to my old improv team’s email list with information about our upcoming shows.  It ended up being pretty useful, but took a bunch of time to write.  Anytime an announcement includes robots, I’m hooked.

The Java and Python client libraries allow you to design your robot

Python!  (Note to Google, please rewrite the above to read: The Python and Java client libraries.)

Lastly, there’s a Chess gadget!

Obviously, the real test will be adoption.  I signed up for sandbox access, because it feels like this is the future, and I want to support it.  A slew of my non-technical friends now treat Facebook (nice, but a non-open standard) like it’s email.  As a consultant I’d love to be able to suggest modern collaboration tools that don’t tie you into one particular vendor.  And most importantly, Wave looks like a heck of a lot of fun to play with.  Although I haven’t released anything of any importance on Appengine, I love the concept.  The Appengine deployment process, applied to collaboration, seems like a real win++ to me.  Nice work Googlers.

Remember that whole Appengine got the JVM link I posted yesterday?  I forgot to add one thing.  Rhino is on that list.  Meaning, you can write really awesome Javascript applications on the Google-server.  In Ted Leung’s Pycon talk he specifically points out the performance gains that several big companies have baked into Javascript, how Javascript is one of the most “known” languages, and how Python & Ruby folks should know that as soon as Javascript becomes an accepted server side language, they’re probably going to have to join or die.

Obviously, I’m paraphrasing.

That said, picture Ted as a crazy prophet saying that when the sky rains blood, it means the end of Python.  Then picture Google Appengine raining blood.  See where this is going?  Google totally released server side javascript on Tuesday.  The prophecy is coming true!

(If you watch the video, feel free to skip over the Q&A period, in which I ask a batshit crazy question and blame democracy for the inability of Python to feed more families.)

I started playing around with Amazon’s SimpleDB this week. I knew I wanted to move portions of an existing MySQL database to it, but slowly realized that it was probably just easier to upload the whole thing. I added my finished script to the Python Cookbook as Recipe 576548: Converting MySQL database to an Amazon Simple DB database.

It worked for me, it will probably work for others. With a little TLC, it could probably be a bit easier to run. It’s really more of a rumor than a recipe. Regardless, it let me put a bunch of data into the cloud quickly. Sort of quickly. It turns out, SimpleDB is pretty slow for uploading. My first version of the script was single threaded, it took forever. I threw together a simple thread doo-dad, and it seems to have sped up. Although I didn’t time it.

This whole cloud thing has me sort of giddy. When I finish what I’m working on, I’ll write it up.

Phew…good summer folks. So good, that I haven’t worked on a single side project since June. ChiPy ignored, Tastebud ignored…just about everything, unless it was work or named Wil & Camri, totally, epically, ignored.

Now that the temperature is dropping faster than the leaves, I’ve turned my attention to messing around with cool toys again. My Dad and I have been playing a bunch of Eve Online, growing our fancy Mining corporation to around 300 million isk in assets. It’s the nerdiest thing I’ve ever done, but it’s ok, because I’m doing it with my Dad. We don’t fish, we mine pretend ore and sell it for pretend money.

At the same time, I’ve been paying attention to some stuff that Harper Reed has been doing. If I were to boil his recent work down to a theme, he’s connecting lots of services together in new and interesting ways. It used to be called “mashing up” but now its really just more of “connecting the wires.” You don’t have to do a whole bunch of screen scraping and reverse engineering anymore, it’s all pretty much out there for the connecting. The idea is appealing to me, as I’m largely lazy, and don’t like to pay attention to a project for more than a few hours after the boy goes to sleep.

Now that my Dad and I are sitting on a huge asset (pun intentional), we’ve been managing our business with a handful of shared Google Spreadsheets. There’s a site called Eve Central that publishes an XML API of market data. We pulled that data into our spreadsheet to figure out profit margins on the 10 or so industrial goods we manufacture to supplement our mining.

The mineral data we get from Eve Central is good, but it’s based on universe wide prices, rather then the prices in the region we call home. Also, we’re totally risk adverse, so we never buy minerals in low security space, only in high security. The prices on Eve Central ended up being low, artificially inflating our margins. When we stepped back and did the math by hand, we realized production was actually losing us money.

After retooling our business to focus on ore mining and mineral refining (production will catch up one of these days) I took a stab at cleaning up the spreadsheet. I wanted to filter the data on Eve Central, to only show prices in our region, and only in high security space. However, like I said, I’m lazy. The thought of putting together a Django app, just to do some filtering, seemed like wicked overkill. So I sort of left the project sit.

Then I read this great entry on Data Scraping Wikipedia with Google Spreadsheets. He uses Yahoo Pipes! I totally forgot they existed. So I sat down for a bit with the Pipes docs, and ended up pulling together a simple filter for the Eve Central feed, which gives you a nice RSS feed. After dinking around with it for a while, I found out that you can also grab a CSV file by changed the _render option to ‘csv’. I pulled the filtered feeds into spreadsheet and saw that we’re losing isk across the board.

I have a couple of other ideas in the queue, based on this new style of easy-peasy mashup. As long as I can make significant headway between 7:30 and 9:30 pm, I’ll probably get them accomplished.

Django 1.0 rc is coming this August, according to the RFC Jacob posted last night. Let’s keep our fingers crossed on that one, I’d love to see it happen.

I started my first AppEngine project this evening. I was lucky enough to get into the Beta. I don’t want to be too gushy or hyperbolic, but this is the future folks.

I’m not going to cover the bits about what appengine is, I’m assuming you’ve heard. If not, head over to the appengine site and do some reading. If you’re a little bit confused about WSGI, I encourage you to read anything Ian Bicking has written over the past few years, or my humble article on building a wsgi app.

So…my plan is to build a tomato tracking application on top of THE CLOUD (cue the music). The wife and I are planting a bunch of tomatoes this year, so I’m sort of hopped up on them. The app will let folks to report on the life, well being, and eventual yields of their backyard tomatoes. It’s going to be the world’s greatest web application. I’m going to develop it before a live studio audience via the TomatoBase Google Code Project. The app will live at http://tomatobase.appspot.com.

So far, there’s not much there. Just a landing page that says something pithy, that’s protected by a Google user login. I initially thought I’d jump feet first into the Google themed stack, using their very fancy webapp api, but after reading how easy it is to just go for Django, I decided to go with what I know. There’s a few gotchas, it’s really more like Django-Lite, but it’s still pretty darn good. You lose the ORM and Admin, but still get the url mapper, controllers, middleware, templates…oh, and BigTable. Pretty good trade as far as I’m concerned.

Developing with the toolkit is pretty slick, everything runs locally until you’re ready to push, and then it’s a quick “appcfg.py update django-mater” and it goes live. So simple. I love it.

I haven’t really ventured beyond exploration into actual TomatoBase development just yet. I started exploring the user system a bit, it’s pretty straight-forward. So straight-forward that I contributed a decorator to Django Snippets that lets you use the user system in a @loginrequired sort of way. It’s here: Google AppEngine Login Decorator.

I’m moving much quicker on this than I anticipated. I thought I wouldn’t have time to explore until this weekend, but it’s a really compelling product. I’m excited for them to open it up to a wider audience. At some point I’ll write a bit about what I think this all means (this means something), but for now I’m too busy playing.

It was only a matter of time. Great great great!

At the state of Django talk on Saturday at Pycon, I asked what the status of Django on Jython was, as I think for guys like me (trying to explain Python and Django to traditional JVM-happy shops) that stack is going to get some traction. If I can tell someone that my crack team of consultants can build a full featured web application in record time, and deploy it in their existing JVM heavy infrastructure, I’ll turn some heads.

If I can turnaround and say, the exact same app will also run in your .NET infrastructure, now the heads are spinning off into the cosmos.

Python for the win!

Simon Wistow asks whether or not we need a better open source queuing system. My answer is yes. I’ve built two one-off hackish queuing systems for $work in the past three years, both of which used database tables as queues, both of which were only accessible by a language-locked api, neither of which was accessible by any sort of neutral service. Both had very specific tasks, hold tasks in a queue for offline (read: outside of a web transaction) processing.

Both jobs could have been fulfilled by a nice message queue service, but it didn’t seem like the effort was worth the payoff. That said, now that I’ve built two of exactly the same thing, I’m 100% of the belief that I’d love to have a easy to install web service-y queue (that isn’t owned by Amazon).

So, why the need? It’s sort of like the evolution of CouchDB & friends I wrote about back in December, all web programmers eventually are expected to be programmers, then they start examining the tools the traditionalists have built, and they question, “why can’t this be better?” Relational databases might not be best for web apps (I’m still not convinced, but for the sake of argument, go for it.) Full featured queue services might not be best for web apps. If for no other reason than they introduce a complicated system into your ecosphere for what is actually a very simple problem. I like Simon’s pointing at memcache and saying, “make a queue that works like that.” That would be pretty cool. I’m voting +1.

That said, like a good American I’m making this vote without fully examining all the candidates. Maybe the existing queues are simple enough…educate me lazy-web.