Lonely Lion

Last week I gave an Amazon EC2 presentation at the Day of Cloud conference here in Chicago. My slides from the presentation cover some basics about EC2. It was a good day, lots of good speakers, and lots of attendees that were interested in getting their apps into the mysterious clooooouuudd.

The kicker is, I spent a fair amount of time explaining how to set up a relational database on EC2, just in time for Amazon to announce that they’re releasing very easy to use MySQL instances. Luckily, I have a half hour tomorrow at the Chicago Google Tools User Group to revisit the Amazon talk. I’ll revise the talk to focus on Amazon’s new offerings, specifically how to get a reasonable web app up on Amazon using RDS as a database back end.

I understand the irony of giving the talk at a Google tools group, but it can’t hurt to know how to move your applications off of AppEngine and on to other services. It’s all part of Data Liberation, right?

Now, with video:

I’ve been involved with a handful of EC2 deployments at PSC, and they’ve all ended up with a custom AMI. Basically, starting up an existing base Ubuntu server, installing and configuring packages, and saving the new image as a custom AMI. It’s not a bad way of doing things, but I had a nagging feeling that there was an easier way. Especially given the ease of deploying an AppEngine app. Granted, they’re two completely different animals, but still…there had to be something better.

Eric Hammond, of Alestic fame, answered a bunch of my nagging subconscious questions with the release of runurl. Now, you build and host your configuration scripts, and pass a list of the scripts to run on server start up through passed user-data scripts. In theory, it’s possible to start up a new base server, and automate the build out through hosted scripts. It means slower start up time, but significantly less configuration and AMI maintenance. For a webserver cluster, where you have some time to get new nodes up, this method makes more sense than the standard AMI-saving method. Nice work Eric!

Well…it’s either a game changer, or it will end up being really popular in Brazil.  I’m leaning towards the game changer.  Really, it’s about time, it’s good to see a big company being ambitious instead of just making a new thing that’s just like the old thing, and they’re open sourcing it.

Imagine a big cast iron pot, throw in Appengine, XMPP, Blogs, Wikis, stir, simmer, and serve.  Wave!  Some particularly interesting bits from the API preview doc:

You define the behavior of your robot by defining the events which you wish your robot to be notified. Wave contacts the robot whenever one of these events occur, such as a change made to a wave in which the robot is a participant.

I spent an absurd amount of time hand crafting an email-bot a few years ago that could respond to my old improv team’s email list with information about our upcoming shows.  It ended up being pretty useful, but took a bunch of time to write.  Anytime an announcement includes robots, I’m hooked.

The Java and Python client libraries allow you to design your robot

Python!  (Note to Google, please rewrite the above to read: The Python and Java client libraries.)

Lastly, there’s a Chess gadget!

Obviously, the real test will be adoption.  I signed up for sandbox access, because it feels like this is the future, and I want to support it.  A slew of my non-technical friends now treat Facebook (nice, but a non-open standard) like it’s email.  As a consultant I’d love to be able to suggest modern collaboration tools that don’t tie you into one particular vendor.  And most importantly, Wave looks like a heck of a lot of fun to play with.  Although I haven’t released anything of any importance on Appengine, I love the concept.  The Appengine deployment process, applied to collaboration, seems like a real win++ to me.  Nice work Googlers.

Remember that whole Appengine got the JVM link I posted yesterday?  I forgot to add one thing.  Rhino is on that list.  Meaning, you can write really awesome Javascript applications on the Google-server.  In Ted Leung’s Pycon talk he specifically points out the performance gains that several big companies have baked into Javascript, how Javascript is one of the most “known” languages, and how Python & Ruby folks should know that as soon as Javascript becomes an accepted server side language, they’re probably going to have to join or die.

Obviously, I’m paraphrasing.

That said, picture Ted as a crazy prophet saying that when the sky rains blood, it means the end of Python.  Then picture Google Appengine raining blood.  See where this is going?  Google totally released server side javascript on Tuesday.  The prophecy is coming true!

(If you watch the video, feel free to skip over the Q&A period, in which I ask a batshit crazy question and blame democracy for the inability of Python to feed more families.)

I’m often heard to mutter, “The JVM is the future man…if you’re a Python or Ruby developer, you should really get up on it. It’s the future.” Google agrees with me (it isn’t the first time, they’re constantly hitting me up for advice). They released the JVM for Appengine complete with a fancy Eclipse plugin.

Although the majority of the documentation talks about the release of Java for Appengine, there’s an important distinction to make…they reallly released the JVM for Appengine, as evidenced by this nice post about which JVM languages will work on Appengine.

So, really, in one swoop, they released Jython, JRuby, Scala, Javascript, and Clojure. That’s pretty slick.

Sure Java is arguably way less nice than Python and Ruby, but the JVM is pretty great. Languages that target it as a platform open up a lot of otherwise closed Enterprise doors. If you want to write Python or Ruby as your day-job, you should really start paying attention to the JVM.

Pycon 2009, as of 2pm on Friday, is the best Pycon I’ve been to yet.  The talks this year are really well put together, it’s clear that the speakers are high caliber and spent time on their talks.  Despite the lower attendance numbers (thanks Economy!), it feels like Python is poised for some significant growth.

The Python VM panel, with representatives from Jython, IronPython, PyPy, cPython, and the relatively new Google-cPython-speedup project-unladen-swallow, is laying out a future for Python the language points pretty clearly at “Python is an objectively good language, let’s get rid of this silly implementation idea.”  Meaning, developers want to write Python, and they want it to run everywhere, and they want to leverage other tools, regardless of origin language, in their language of choice, which…thankfully, is Python.

Like a lot of developers nowadays, I’m spending some time learning Objective-C & the Cocoa Framework so that I can jump into iPhone development. I’m clearly a little late to the game, but I usually am, so…no worries.

So far, I’m keeping an open mind (which can be tough for me, as I’m 32 but act like a cranky 72 year old). It’s surprising, Objective-C isn’t as wildly complicated as it first looks, the really disorienting thing is working in Xcode. I’m so in love with the straight simplicity of building apps in Emacs that all the dragging and dropping throws me off. Granted, in my recent (undocumented) foolings with Java, I started to covet all the work that Eclipse was doing for me, however, something about all the magical dragging and clicking makes me nervous. What’s it doing in there?

I’ve also got that old sharecropper feeling, which is kind of rough. Once you develop your fancy iPhone app in Objective-C and Cocoa, you can reuse libraries to build it out as a OSX app, but that’s about it. You can develop a OSX app in Python or Ruby through bridges to Cocoa, but you can’t use that code on the iPhone, iPhone is Objective-C only. You could move some of your heavy lifting to a web service, but then you’re tethering your fancy iPhone app to require a network connection.

Anywho, it’s an interesting thing to learn, I’m sure I’ll have fun with it, it just doesn’t feel like it has the wide open opportunities that learning Python did, or Java does.

[youtube=http://www.youtube.com/watch?v=InI5n3NTvR4&color1=0xb1b1b1&color2=0xcfcfcf&hl=en&feature=player_embedded&fs=1]

I heard mashup and cloud computing. Despite the buzz-ish nature of both, it’s an encouraging sign. As much as I love Everyblock, I think they’d agree that it would be nice to see the federal, state and local governments publishing data, rather than concerned parties scraping stuff.

I started playing around with Amazon’s SimpleDB this week. I knew I wanted to move portions of an existing MySQL database to it, but slowly realized that it was probably just easier to upload the whole thing. I added my finished script to the Python Cookbook as Recipe 576548: Converting MySQL database to an Amazon Simple DB database.

It worked for me, it will probably work for others. With a little TLC, it could probably be a bit easier to run. It’s really more of a rumor than a recipe. Regardless, it let me put a bunch of data into the cloud quickly. Sort of quickly. It turns out, SimpleDB is pretty slow for uploading. My first version of the script was single threaded, it took forever. I threw together a simple thread doo-dad, and it seems to have sped up. Although I didn’t time it.

This whole cloud thing has me sort of giddy. When I finish what I’m working on, I’ll write it up.

Phew…good summer folks. So good, that I haven’t worked on a single side project since June. ChiPy ignored, Tastebud ignored…just about everything, unless it was work or named Wil & Camri, totally, epically, ignored.

Now that the temperature is dropping faster than the leaves, I’ve turned my attention to messing around with cool toys again. My Dad and I have been playing a bunch of Eve Online, growing our fancy Mining corporation to around 300 million isk in assets. It’s the nerdiest thing I’ve ever done, but it’s ok, because I’m doing it with my Dad. We don’t fish, we mine pretend ore and sell it for pretend money.

At the same time, I’ve been paying attention to some stuff that Harper Reed has been doing. If I were to boil his recent work down to a theme, he’s connecting lots of services together in new and interesting ways. It used to be called “mashing up” but now its really just more of “connecting the wires.” You don’t have to do a whole bunch of screen scraping and reverse engineering anymore, it’s all pretty much out there for the connecting. The idea is appealing to me, as I’m largely lazy, and don’t like to pay attention to a project for more than a few hours after the boy goes to sleep.

Now that my Dad and I are sitting on a huge asset (pun intentional), we’ve been managing our business with a handful of shared Google Spreadsheets. There’s a site called Eve Central that publishes an XML API of market data. We pulled that data into our spreadsheet to figure out profit margins on the 10 or so industrial goods we manufacture to supplement our mining.

The mineral data we get from Eve Central is good, but it’s based on universe wide prices, rather then the prices in the region we call home. Also, we’re totally risk adverse, so we never buy minerals in low security space, only in high security. The prices on Eve Central ended up being low, artificially inflating our margins. When we stepped back and did the math by hand, we realized production was actually losing us money.

After retooling our business to focus on ore mining and mineral refining (production will catch up one of these days) I took a stab at cleaning up the spreadsheet. I wanted to filter the data on Eve Central, to only show prices in our region, and only in high security space. However, like I said, I’m lazy. The thought of putting together a Django app, just to do some filtering, seemed like wicked overkill. So I sort of left the project sit.

Then I read this great entry on Data Scraping Wikipedia with Google Spreadsheets. He uses Yahoo Pipes! I totally forgot they existed. So I sat down for a bit with the Pipes docs, and ended up pulling together a simple filter for the Eve Central feed, which gives you a nice RSS feed. After dinking around with it for a while, I found out that you can also grab a CSV file by changed the _render option to ‘csv’. I pulled the filtered feeds into spreadsheet and saw that we’re losing isk across the board.

I have a couple of other ideas in the queue, based on this new style of easy-peasy mashup. As long as I can make significant headway between 7:30 and 9:30 pm, I’ll probably get them accomplished.