Monthly Archive for October, 2008

Migrating MySQL to Amazon SimpleDB

I started playing around with Amazon’s SimpleDB this week. I knew I wanted to move portions of an existing MySQL database to it, but slowly realized that it was probably just easier to upload the whole thing. I added my finished script to the Python Cookbook as Recipe 576548: Converting MySQL database to an Amazon Simple DB database.

It worked for me, it will probably work for others. With a little TLC, it could probably be a bit easier to run. It’s really more of a rumor than a recipe. Regardless, it let me put a bunch of data into the cloud quickly. Sort of quickly. It turns out, SimpleDB is pretty slow for uploading. My first version of the script was single threaded, it took forever. I threw together a simple thread doo-dad, and it seems to have sped up. Although I didn’t time it.

This whole cloud thing has me sort of giddy. When I finish what I’m working on, I’ll write it up.

Shaking Off The Summer

Phew…good summer folks. So good, that I haven’t worked on a single side project since June. ChiPy ignored, Tastebud ignored…just about everything, unless it was work or named Wil & Camri, totally, epically, ignored.

Now that the temperature is dropping faster than the leaves, I’ve turned my attention to messing around with cool toys again. My Dad and I have been playing a bunch of Eve Online, growing our fancy Mining corporation to around 300 million isk in assets. It’s the nerdiest thing I’ve ever done, but it’s ok, because I’m doing it with my Dad. We don’t fish, we mine pretend ore and sell it for pretend money.

At the same time, I’ve been paying attention to some stuff that Harper Reed has been doing. If I were to boil his recent work down to a theme, he’s connecting lots of services together in new and interesting ways. It used to be called “mashing up” but now its really just more of “connecting the wires.” You don’t have to do a whole bunch of screen scraping and reverse engineering anymore, it’s all pretty much out there for the connecting. The idea is appealing to me, as I’m largely lazy, and don’t like to pay attention to a project for more than a few hours after the boy goes to sleep.

Now that my Dad and I are sitting on a huge asset (pun intentional), we’ve been managing our business with a handful of shared Google Spreadsheets. There’s a site called Eve Central that publishes an XML API of market data. We pulled that data into our spreadsheet to figure out profit margins on the 10 or so industrial goods we manufacture to supplement our mining.

The mineral data we get from Eve Central is good, but it’s based on universe wide prices, rather then the prices in the region we call home. Also, we’re totally risk adverse, so we never buy minerals in low security space, only in high security. The prices on Eve Central ended up being low, artificially inflating our margins. When we stepped back and did the math by hand, we realized production was actually losing us money.

After retooling our business to focus on ore mining and mineral refining (production will catch up one of these days) I took a stab at cleaning up the spreadsheet. I wanted to filter the data on Eve Central, to only show prices in our region, and only in high security space. However, like I said, I’m lazy. The thought of putting together a Django app, just to do some filtering, seemed like wicked overkill. So I sort of left the project sit.

Then I read this great entry on Data Scraping Wikipedia with Google Spreadsheets. He uses Yahoo Pipes! I totally forgot they existed. So I sat down for a bit with the Pipes docs, and ended up pulling together a simple filter for the Eve Central feed, which gives you a nice RSS feed. After dinking around with it for a while, I found out that you can also grab a CSV file by changed the _render option to ‘csv’. I pulled the filtered feeds into spreadsheet and saw that we’re losing isk across the board.

I have a couple of other ideas in the queue, based on this new style of easy-peasy mashup. As long as I can make significant headway between 7:30 and 9:30 pm, I’ll probably get them accomplished.




Proxy surf mp3 site mp3 sites