Twitter Mosaic and EC2

We’ve played with Amazon’s Elastic Compute Cloud service before – principally as a low-cost proof-of-concept to demonstrate Pixenate‘s ability to scale across multiple servers in a clustered environment (Some of our larger customers have several instances of Pixenate running on multiple servers).

Recently I took another look at EC2 – this time for a different reason…

Our Twitter Mosaic service can generate large mosaic images on the fly which can be printed on to T-Shirts, Mugs and Bags.
The service pulls 100’s of user avatar images from twitter and creates a large mosaic from the image collection. This turned out to be a much slower process than anticipated – not because of the image processing but because of how Twitter stores it’s images. Twitter uses Amazon’s S3 cloud storage service – a very low-cost effective storage system. The only problem with S3 right now is that access can be slow. Our service was pulling 100’s of images at a time from S3 – that’s 100’s of HTTP requests – most of which could take anywhere between 0.05 seconds and 5.00 seconds to complete. The result was that our service could take up to 2 whole minutes to create mosaic for use on a product. Clearly not a good shopping experience.

One of the overlooked advantages of EC2 is that if you run software on an EC2 server, S3 is blazing fast. If you’re writing software that does any heavy lifting with S3 data – do it on EC2. It’s like working in zero gravity.

We quickly changed the service so that a key part (the retrieval and composition of S3 images) moved to EC2 and a single HTTP request could be made to the EC2 server. By doing so, we reduced the time it took to create a mosaic from 2 minutes to a couple of seconds. A big win.

Advertisements
Tagged , , , , ,

4 thoughts on “Twitter Mosaic and EC2

  1. Totally agree. There is a cost issue in terms of pulling and pushing from EC2 as well vs. external vendors.

  2. […] So if you’re hosting your data on S3, EC2 is a great way to crunch on it efficiently. Update: Walter weighs in with a great demonstration of this on the backend for his Twitter Mosaic […]

  3. […] years ago. Needless to say, these APIs are being battered at the moment. Walter has more details on how to get things running quickly too and the end products seem to be reviewed very […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: