Amazon Web Services Blog

It is finally summer here in Seattle and I'm trying to get out of the office as early as possible today. Here are a few cool things that have recently landed in my inbox:

Don MacAskill wrote to tell me about his new product, SmugVault. This new service extends the existing image storage capabilities provided by SmugMug, allowing users to upload a wide variety of image files, as well as files of other sorts, for safekeeping. Don used Amazon's DevPay system to implement a usage-based fee structure -- $1 per month and 22 cents per GB of storage, along with fees to transfer data in and out. The entire contents of a 2 GB memory card can be stored for just 44 cents per month. SmugVault can create finished products from raw video and image data, and it can also bundle together alternate formats of an image (GIF, RAW, and so forth).

Animoto For Business adds professional features to Animoto's existing music video creation service. This version of the service can be used by businesses of all sorts -- sports teams, real estate agents, vacation resorts, and trade shows -- to produce DVD-quality videos in MP4 and ISO formats. Business users have access to a library of music that has been pre-licensed for commercial use. The product is brand-new but there are already some good success stories.

There's a new release of Cloud Studio. It now incorporates an S3 browser!

ElasticFox now includes support for Firefox 3.0, as does the S3 Firefox Organizer.

Jungle Dave wrote to tell me that he's released version 2.0 of Jungle Disk Desktop. The new version includes a revised user interface with a setup wizard, a configuration dialog, and a backup preview dialog; renaming and encryption of files and directories; support for European S3 buckets; bandwidth limiting, and much more. There's also a beta version of the workgroup version (details on it can be found here). This version allows multiple users to share a single Amazon S3 account without sharing account credentials, and also allows for per-user control of access to S3 buckets.

And with that, I am out of here!

-- Jeff;

From time to time, potential users of AWS ask me about the best way to set up a highly scalable architecture using Amazon EC2, S3, SimpleDB, and SQS. I'd like to challenge readers of this blog to document their AWS-powered architectures in a blog post, preferably with a diagram, and to leave comments with a link back to their posts. I'll collect them all up in a future post.

Here are a few that I have already:

Doug Kaye described the architecture behind GigaVox Audio Lite in his post, Amazon for Infrastructure-on-Demand. Doug used EC2, S3, and SQS to build the highly scalable podcast processing system behind The Conversations Network.

Doug's implementation regulates the number of EC2 instances in use by tracking the amount of time it takes to process each work item in the queues which drive the Transcoding and Assembly processes.

Don MacAskill described SmugMug's master controller (SkyNet) in SkyNet Lives! (aka EC2 @ SmugMug). Don's post doesn't include a block diagram, but it does include a cool usage graph (included at right).

Don's master controller watches 30 to 50 factors in order to make high quality scaling decisions. It was called RubberBand until it became sentient and attempted to take over the world launch several hundred Extra-Large EC2 instances simultaneously. It was then renamed SkyNet.

The architecture behind online social media dating site Zoosk is described in Elastic Computing with Amazon Web Service, written by Zoosk CEO Shayan Zadeh.

Per the blog post, they use SQS to maintain a queue of uploaded photos. The photos are processed on EC2 and then uploaded to S3. The graph in the blog post indicates that they are adding approximately 4 TB of new photos every month.

The AWS Developer Connection has some worthwhile how-to articles as well. In Monster Muck Mashup - Mass Video Conversion Using AWS, Mitch Garnaat shows how to use SQS, EC2, and S3 to do video conversion in a scalable way.

The article Auto-scaling Amazon EC2 with Amazon SQS also has a whole lot of really good information.

Once again, I invite you to write an architecture post of your own and to leave a link to it in the comments. I would also like to see posts which make reference to load management tools such as Scalr, RightScale, and Elastra.

-- Jeff;