Amazon Web Services Blog

VholdR - Wearable Camcorder and Community Site

Yesterday afternoon I left the confines of the Amazon headquarters building and paid a visit to the Seattle-area offices of Twenty20 Cameras in order to learn more about their new VholdR product.

I learned that they've got a really cool piece of hardware which is tightly coupled to an Amazon-powered online service and I can't wait to become a user. This is the kind of thing that I could lend (emphasized for their benefit) to my teenagers so that I could get a first-hand view of their snowboarding antics from a safe distance while I stay as far away from Alpental as possible (I'd rather melt than freeze).

The hardware component is a small and rugged video camera suitable for use in extreme conditions -- ski slopes, competitive cycling, and so forth. The camera can record 2 full hours of video on an embedded MicroSD card. Once mounted, leveled (using a pair of lasers, no less) and aimed, a single, glove-friendly switch controls the recording process. There's also a microphone built-in.

Once the recording is complete, it can be uploaded to the VholdR site, tagged, labeled, and then shared.

The site itself uses several Amazon Web Services behind the scenes. All of the videos are stored in Amazon S3, and the entire EC2-powered uploading, transcoding, and post-processing system is driven by data stored in a couple of Amazon SQS instances. The developers noted that they made use of David Kavanagh's "lifeguard" automated server pool management as described in this Developer Connection article. Scaling should be painless for them -- as more videos are uploaded, the lifeguard will automatically add enough additional EC2 instances to ensure that they are processed in a reasonable amount of time.

Finally, they mentioned that they are hiring like crazy and asked me to note that they have a whole bunch of open positions in the Seattle area for development, user experience, sales, and marketing people.

-- Jeff;

Sonian Archive SA2 - Data Archiving Service in the Amazon Cloud

Sonian Networks has completed their early-adopter program and is about to launch their new Amazon-powered data archiving service, the Sonian Archive SA2. I spoke with their executive team last week and was very intrigued by what I heard.

With the exception of one external watchdog server, the SA2 service is totally contained within the Amazon cloud! In fact, it was designed from scratch to be a highly-scalable cloud-aware application using four of the Amazon Web Services -- Amazon S3 for storage, Amazon EC2 for processing, the Amazon Simple Queue Service for internal job control, and the Amazon SimpleDB for storage of document metadata. The automation, deployment, and self-healing aspects were all designed and built in-house.

They believe that the economies of scale which come from using AWS will allow them to offer SA2 at a very competitive price. Typical customers will retain archived data for 3, 5, or 7 years. Sonian Networks will retain all customer data, fully parsed, indexed, and searchable, for the specified retention period.

Their goal is to help mid-sized companies (those with a few hundred to a few thousand employees) do a better job of archiving internally generated digital content for storage management, electronic discovery and litigation support. They will begin by focusing on email (with integration to popular servers such as Microsoft Exchange, IBM Lotus Notes, and Novell GroupWise) and will expand over time to handle other types of managed content such as wikis and blogs.

Traditional data archiving solutions are appliance-based and suffer from capacity limitations. With the ever-increasing number of messages along with the fact that people routinely attach and store large documents and media clips, it is not surprising to learn that these systems tend to fill up far sooner than expected and that scaling up can be complex, challenging, and costly.

As you might expect I am thrilled to hear about in-the-cloud implementations like this. If you've got one, don't hesitate to send an email to awseditor@amazon.com .

-- Jeff;

AWS For Facebook Applications

Developers  have used the Amazon Web Services to create highly scalable applications for Facebook including  Haikoo Zoo, Booze Mail, iLike, We're Related, and SocialMoth Secrets.

AWS is an ideal hosting environment for these applications. Developers can start small, test and prove out their ideas, and then rapidly add processing and storage resources as they develop a following.

In order to make it even easier for developers to find the resources that they need, we've teamed with  Facebook to collect all of the resources that you need to be the next big success story in one convenient location. If you visit our new page on Building Facebook Applications on AWS, you can find step-by-step instructions for registration along with links to tutorials, a pre-built EC2 AMI, direct access to the relevant developer forums and interviews with the creators of some of the applications mentioned above.

-- Jeff;

PS - If you are an AWS developer and you are on Facebook, you should check out the AWS and Cloud Computing groups too.

Proto.in Startup Event in Second Life This Week (Updated)

I will be speaking (via Second Life) at the Proto.in conference on Friday night and I hope to see you there. I will be speaking at 11:59 PM Friday (this time has changed several times) and will also be hanging out on the island on Friday as time permits.

You can read more about this on the Proto.in blog and in another great post by Balaji (who's also done some cool integration between Second Life and the Amazon Mechanical Turk).

Note: The original plan was to livecast the event into Second Life. This has been cancelled due to technical issues related to the internet connection at the host site. My talk will still proceed as scheduled.

-- Jeff;

Increasing Amazon S3 Data Transfer Performance

The Amazon S3 team is now beta-testing support for an important low-level networking feature which has the potential to significantly increase the performance of large data transfers to and from S3, particularly (but not limited to) for long distance data transfers.

In particular, there is a new beta endpoint which supports the RFC 1323 model for TCP window scaling. With this option in effect, a larger amount of data can be in transit across the network at any given time, reducing the impact of speed-of-light delays as data is broken down in to chunks, sent across the internet, verified, and acknowledged. When the two nodes in question are far apart, the time that it takes for the data to travel from sender to receiver (and for the acknowledgments to travel back) turns out to be just as important as the raw data transfer rate.

Per the thread in the Amazon S3 Forum, early results from the beta testers are quite good with reported speedups of 4x to 18x! If you are moving large amounts of data into or out of S3 then you will definitely want to implement this feature.

Note that we are supporting this through a beta endpoint. The endpoint will be present for the duration of the test and will then vanish. Of course you won't want to embed this endpoint in any shrink-wrapped software.

You will need to dig deep into your operating system's networking setting in order to take advantage of this new feature. For Linux systems the information here will get you started (there's also some information for other operating systems in there too). Windows 2000 and Windows 2003 users can start at this Knowledge Base article. I know nothing about the Mac except that my youngest daughter keeps asking me for one.

I would also recommend that you read this very informative article by Brian Tierney for further background information and some formulas and recommendations on how to calculate and set the optimal window size.

-- Jeff;

PS - Did you know that Amazon.com sells actual windows?

New Zealand and Australia

Are you a developer in Australia or New Zealand? In February I'll be visiting New Zealand and Australia on the first-ever Amazon Web Services evangelism tour. I'm really excited about the trip--especially because I lived in New Zealand briefly many years ago, and have not had the opportunity to visit since.

So far there is a really solid lineup of user groups--ranging from Linux to .NET developers. There are still some daytime slots open on my calendar if you'd like a private meeting, school class presentation, etc.--or if you just want to have tea or coffee. In fact, one Monday evening is still available in Melbourne if you run a user group. Take a look at our Evangelism Wiki and request a meeting if you're interested.

We've made all sorts of recent announcements about new services such as Amazon SimpleDB, so there's lots for us to have a conversation about!

-- Mike