Amazon Web Services Blog

New ECS Release

A new release of ECS 4.0 was rolled out earlier today. There are two new features:

• A new response group, OfferListings, was added for all locales. This response group returns somewhat lighter version of the data provided by the Offers group.
• Data for the French locale now includes the French waste tax.

-- Jeff;

Precious - Ruby, Gems and AWS

Ruby got a new face-lift with the new Rails framework. I was always fascinated with the Ruby language. 25 lines of Java code gets shrunk to 4 lines of equivalent Ruby code plus its more readable. And with Rails, it gets a new kick. It is always impressive to hear people say "I am a full-time Ruby on Rails Developer".

There are plenty of resources for those who would like to get started on this technology:

There are lots of Amazon S3 libraries in Ruby: Some are full fledged libraries like AWS:S3 (Marcel Molina of 37signals talks about his library and why he chose Amazon S3 here - worth a read!), while some libraries are getting started types so that you can customize as per your needs. Some are command shell utilities like rSh3ll, while some are gems like S33r and ruby-s3 for easy installation.

Zachary Holt created a powerful library for Amazon SQS in Ruby. He has added some cool utility functions such as ability to forcibly delete a queue etc.

If you would like to get started with Ruby/Rails on Amazon EC2 instance, all you have to do is fire up an instance and type "yum install ruby" and you are all set!. To get Rails, follow two additional steps mentioned here.

Ruby on Rails reduces your development time and gets you up and running in minutes. There are several apps which have been built using Ruby and Rails framework. Some that caught my eye were:

• All new! SmilePooling - SmilePooling connects Amazon's "smily boxes" (read sidetrack below) and Amazon ECS's WishLists with the car-pooling concept. It is a web application to order items from Amazon stores for you and your friends and colleagues, so that you can split the shipping costs and save. Nice idea, isn't it ?
• Competitio.us - simple handy tool to track your competition. Competitio.us is powered by the Alexa Web Information Service and the Alexa Thumbnail service. It gives all sorts of traffic data and other information for your competitor's website.

Are there any other cool Ruby on Rails applications powered by AWS ?

--Jinesh

Sidetrack: Have you guys noticed the amazon.com logo and wondered what the arrow signifies? I did not know until now. The arrow means Amazon has everything from A to Z and it also represents the smile it brings to the customer's face - Quite deep, isn't it?

update: Also, one more  -  Ruby/Amazon - Amazon ECS library in Ruby.

Sent To Smugmug

Over in the "practice what you preach" department, I've spent the last month or so uploading over 18,000 of my family pictures to Smugmug. Even though I store the photos on a RAID drive in my house and move regular DVD backups offsite, I am still relieved that someone else is worrying about my photos.

Smugmug is very easy to use. There are some nice third-party tools for uploading (via the Smugmug API). I used a powerful tool called Send to Smugmug.

Once the pictures are uploaded it is very easy to group, caption, and rotate them. The site scales down gigantic, high-resolution photos for display, but the originals are just a click or two away.

It is good to know that my family pictures are now smug as a bug in a rug, as they say.

 

-- Jeff;

Sheep Market Thesis

Aaron Koblin, developer of The Sheep Market, has published his thesis as a Word document: The Sheep Market: Two Cents Worth. It is fun to read, with some very far-ranging references and some entertaining diversions. It is interesting to read about Aaron's hesitancy to be just a cog in a machine, then to see him use the Mechanical Turk to actually rotoscope a movie frame illustrating Charlie Chaplin as just such a cog, and finally to the use of the Turk to get 10,000 people to draw a sheep. I didn't know that some of the Turkers had actually taken issue with Aaron's decision to offer the sheep for sale.

After having given many, many AWS presentations in the last 4 years, I do have to say that The Sheep Market is one of those applications which always gets people to pay attention. The productivity factor of "11 sheep per hour" always gets a good laugh from the audience.

-- Jeff;

Technology Evangelism Conference

If you are interested in the art of evangelism, there's an event coming up that you should attend. It's called GNoTECon (Global Network of Technology Evangelists Conference), and will be held in Santa Clara, CA on December 4th. Microsoft's Anand Iyer blogged about it recently, so I won't repeat everything here. In fact, this is a really short heads-up post to all of you who are evangelists, or passionate about technology. You don’t have to wear the title on your business card to be an evangelist; many of us spread the word about our passion in formal and informal venues.

Worth noting who will be speaking. In addition to our own Jeff Barr, Guy Kawasaki will keynote.

Hope to see you there...

Mike

S3 for Static Web Content

In Userscript.org and Amazon S3, Jesse Andrews describes how he moved all of his static content from his own server over to S3. Jesse set up the virtual hosting, moved over all of the content (using S3 Fox no less), and had everything up and running inside of 15 minutes. As he notes "it took longer to write this post than it did to do the conversion."

Userscripts.org isn't some quiet site with little or no traffic. As the primary repository for Greasemonkey scripts, it is quite busy -- Jesse says that they handled 24 million requests for 307GB of static content in the last two months alone.

Jesse asked his friends to try out the new version side-by-side with the old one. In this unscientific and somewhat subjective test, the new version (using S3) was unanimously judged to be faster.

By the way, be sure to browse the items tagged with Amazon on the site. You'll find my old favorite Book Burro (also written by Jesse) and lots of other cool stuff.

 

But wait, there's more!

Casey Muller wrote a really nice guide, including code for an s3commit script (to copy assets to S3) on Serving the Rails Public Directory out of S3. Casey is one of the developers of jamglue, currently in beta and serving up its content from S3.

-- Jeff;

EC2 Rock and Roll

There's so much happening with Amazon EC2 that I can hardly read it all. Here's a sampling:

• The Atlantis Computing blog talks about Amazin Amazon or why EC2 is the bee's knees. Instead of investing between 30 and 40% of their seed capital on hardware, they have implemented it on top of EC2. As the post notes, the monetary savings is important, but it is not the only benefit to them. They can now focus on their core competency of building their application (Thewebtop).
• One-time Amazonian Greg Linden, author of the Geeking with Greg blog, talks discusses the use of Hadoop on Amazon EC2. Greg saved me the trouble of linking to the Hadoop on EC2 page.
• Speaking of Hadoop on EC2, Scott Delap says that you can Run Your Own Google Style Computing Cluster with Hadoop and Amazon EC2.
• The Android Tech blog says that we are Leading the Charge to Web 3.0.  As they say, "Who knew?"
• David Berlind (co-organizer of Mashup Camp and Startup Camp) does the math, in Amazon’s Jeff Bezos: Honey, I Just Shrunk the Server Hosting Business. David really cuts to the chase of what makes EC2 special, when he writes "None of these back of the envelope calculations take into account what happens if you get smart about server utilization and decide to take full advantage of the Amazon APIs that, in a blink of an eye, can turn these x86 instances on and off. With dedicated hosting of the sort that we have, because of our annual contract, we're married to two servers for an entire year. Whether we're using them or not, we're paying. Not so with Amazon's EC2."

If that's not enough for you, check out the AWS Buzz items for EC2 on del.icio.us.

-- Jeff;

TalkCrunch interview with Jeff Bezos

If you've got 17 minutes to spare, this TalkCrunch interview with Jeff  Bezos is worth a listen. In addition to learning more about our Web-Scale Computing initiative, you can hear Jeff's famous laugh for yourself. There's also a summary of the talk on TechCrunch.

If that's not enough, there's another podcast interview over at Information Week, and here's the transcript.

-- Jeff;

ElasticLive - Hershey's Kisses

An important part of our mission as Web Services Evangelists is to encourage our developer community to innovate. We invite them to surprise us with new and interesting ways to consume our services, to show us the Power of Innovation !

One such innovation is running Windows Server 2003 on an Amazon EC2 instance. The power of Windows combined with the elasticity of Amazon EC2  - Just like Hershey's Kisses (Attractive on the Outside, Nourishing in the Inside).

The tech pundits at Enomaly have put 2 & 2 together by leveraging their experience with Amazon EC2 and successfully installed Windows Server 2003 on an Amazon EC2 instance using QEMU. QEMU is an open source processor emulator allowing a user to simulate a complete computer system within another one. This screenshot shows that they were able to connect to Amazon EC2 instance via remote desktop  connection. Pretty Cool!

They have also put together a small how-to article for others. It will be very interesting to see some real benchmarking on how this emulation layer affects performance.

What is ElasticLive?  ElasticLive is a value-added service on top of Amazon EC2.  Power of Amazon EC2 combined with their expertise and professional support. Pre-installed, pre-configured, automatically-updating applications on the top of web-scale platform at your disposal. What else do you need?

--Jinesh

Caption This Picture...

This photo was taken as our team was leaving the MIT Emerging Technologies Conference last month:

So far the best caption we've come up with internally is "Excuse me, but can you tell us where the physics building is? And how do we find the Mensa Club?"

If you've got a better one, leave us a comment.

-- Jeff;

Special Opportunity for Qualified AWS Developers in Seattle

I get to meet all sorts of interesting people in my position. Many of the meetings start out very casually and then evolve over time. I met Brad Hintze at the first Mashup Camp  and was immediately intrigued by his somewhat vague description of his employer's product. Since then, Brad has done his best to keep me in the loop as the product has come together within a veritable cone of silence.

So, what are they building? I'm not ready to spill all of the beans just yet, but I can say that they are working on some very intriguing tools which should simplify and streamline the process of building and deploying powerful, Ajax-style web applications and which might happen to call one or more SOAP or REST web services and/or deal with raw XML data. I've seen it, and it works as advertised.

Later this month, a few lucky Seattle-area developers will have the opportunity to see what I am talking about!

If you are an individual developer, part of a small company, or part of a small development team at at larger company and you are using one of more of Amazon's web services, then you may be eligible to partcipate. On the 28th of November, there will be a free Seattle-area training and focus group session for the product. You will receive full training on how to use the system, and you'll have plenty of opportunities to provide feedback. You will also get individualized, hands-on attention from the development team, and you will have the opportunity to qualify for free application hosting for all or part of 2007.

Sound interesting? Are you actively using AWS? Do you have the time and the inclination to participate?

If so, send an email to Brad -- his address is bradhintze@gmail.com.

-- Jeff;

Interesting way to use EC2, SQS, S3 together : Webmail.us

Backing up data to Amazon's Secure Storage S3 has been the hot-favorite-use-case for many companies. Webmail.us, the email hosting company, who is currently serving more than 27,000 businesses and hosting more than 350,000 paid mailboxes and growing by close to 10% per month, is now using amazon S3 for backing up encrypted email data.

Smart Guys at Webmail took AWS to next level by using all these services together. Its very interesting to know how creatively they used EC2, SQS and S3 together to scale out, save time and money. They used SQS to queue up their "cleanup" jobs and EC2 to process the jobs concurrently.

Webmail has a backup policy to keep user data on Amazon S3 for 14 days. After 14 days, the magic happens :

Every night, 35,000 "cleanup" jobs to-be-processed gets queued up in SQS. Jobs are fed to hungry EC2 instances one by one. Using the power of parallelism, 3,500 jobs gets processed over 5 EC2 instances in less than 40 minutes. They save 160 minutes by using 5 instances (imagine if they launch 10 EC2 instances!). Plus, there are two more advantages :

• Spawning more EC2 instances will allow them to scale out and process a higher number of jobs (in future) in the same amount of time.
• Cleanup happens at a remote location without sucking up any of their server's bandwidth

-- Jin

A New ISV Business Model

Amazon Elastic Compute Cloud continues to impress the people who matter most: software developers (in this instance, an ISV).

Genexus is an innovative software platform that enables non-technical business people to rapidly build out applications--including an online presence. Andres Aguiar, their chief architect, sent me a note yesterday to say "We have GXPortal running in EC2. It took us one day to make it work. The site is running a beta of the next GXPortal version, on Java/MySQL".

There's an exciting new business opportunity for ISVs: create software that enables businesses to run in an environment without a dedicated data center! In essence, an environment where you throw your servers away! It's a bit different than the traditional ASP model that has been around for a while now, in that the customer has total control over (and is responsible for) their own server; however they do get to eliminate all that muck known as the expense and hassle of a physical data center. It's highly unlikely that any business will be able to operate their own center for less than $0.10 per hour!

-- Mike

Watch The Buzz!

Just a quick reminder that we are now collecting and taggging articles related to AWS on del.icio.us using the user name awsbuzz.

If you visit http://del.icio.us/awsbuzz on a regular basis (or, even better, pick up the RSS feed) you will be fed with a constant stream of information about what's going on with AWS.

Can't miss articles today include Jeff Bezos' New Thing in Business Week and When Workers Turn Into Turkers in the Christian Science Monitor.

-- Jeff;

Avoiding a Success Disaster

For a while I have been using the term "success disaster" to characterize what can happen on the web all too easily. What's a success disaster? You put up a piece of content somewhere and you get ready to handle a reasonable number of downloads.

Being the creative person that you are, however, links to your content shows up on Digg and Slashdot the same day that you are written up on TechCrunch. Suddenly the whole world wants in, and unless you've been Slashdotted before, you have no idea how to respond. Your basic assumptions about server capacity, network traffic, and data transfer limitations have just been blown out the window.

Getting more hardware to address this peak demand is probably not the right thing to do. All that traffic will  go away just as quickly as it came, and you don't want to raise your monthly burn rate just to accomodate these infrequent peaks in demand.

You need a shock absorber to help you deal with this transient surge in attention.

Think of it this way. If 200 of your closest friends suddenly showed up on your doorstep and said that they'd be hanging around for a while, you probably wouldn't drop everything and build an extension to your house. Instead, you would use an on-demand resource, in this case the nearest hotel, to handle this (hopefully transient) need for more room.

The folks over at the Spanning Sync Blog found themselves in just this situation a few days ago. After putting a new video online, traffic surged and they were saturating their server's network connection. They quickly moved the video over to Amazon S3 and the downloads proceeded very smoothly. In fact they served up over 6,000 11,726 copies of the video in just one day - more info in the comments.

For more information, read their post: 5,000 Video Downloads. Time for Amazon S3.

-- Jeff;

Shopping For Real: Amazon Web Services in Second Life

A few weeks ago, as reported on my personal blog, I was interviewed within and about Second Life by a pair of reporters for the Second Life Business Magazine. The new issue was released late last night and I wasted no time downloading it and looking for the resulting article!

If you would like to read it, simply download the PDF and then skip forward to page 71.

Needless to say, I think that there are many ways for  AWS developers to bring their existing skills to this new world and I look forward to seeing even more cool examples of what can be built within Second Life.

What most developers don't know is that there's a very powerful event-driving scripting language inside of Second Life. Any object can respond to clicks and other events with just a few lines of LSL -- the Linden Scripgt Langguage. As a starting point, you may want to look at the LSL tutorial or the LSL Wiki. I've written some scripts of my own, including the Elevator Script.

Need I even remind you that you should email me if you build something cool in Second Life using AWS?

-- Jeff;