Amazon Web Services Blog

Amazon Web Services in India - December 2007

We received several requests from different parts of India for more information about the Amazon Web Services. Hence I am putting together a trip to India in early December.

Wipro's product engineering has already been ahead in the game. We love the Wipro folks and we worked together and co-presented at MySQL Conf 2007, which was well received.

We are also seeing innovative use cases like this one - Outsourcer in Chennai running an EC2 application to process 4,500 staff payroll - costs him just $75 for each run. I call this "Cost-effective Innovation".

While I am there, my presentations will be  mainly focused on:

1. How outsourcing/consulting firms can take advantage of Amazon's Infrastructure on-demand services (testing/QA/development)
2. Exciting new technologies for emerging Indian Start-Ups (Utility Computing, Web Services, Virtualization etc)

As always, we would love to hear from you if you are based in India or have a development office in India and have interest in Amazon Web Services. If you or your organization would like to meet while we are there, please feel free to schedule your appointment on our Evangelists Wiki Page.

-- Jinesh

Link Roundup for Monday, October 22, 2007

My inbox is overflowing with material for the blog! Here's what I have:

The web-based Bungee Connect development environment now includes support for the Amazon Flexible Payment Service (FPS). Using the new library (which you can read about here), it is easy to create, sign, and issue your FPS requests. You may want to read the entire post to learn how Brad used the REST interface to FPS while still taking advantage of the data types described in the FPS WSDL file.

Speaking of Bungee Labs (or more properly speaking to them), Alex Barnett and Ted Haeger of Bungee interviewed me last week as part of their podcast series. You can listen to the first part of the interview here. We talked for about an hour; the second part should be ready soon.

Paul Allen, founder of World Vital Records, is a big fan of Amazon EC2. His recently launched Related on Facebook application has attracted tens of thousands of users in its first couple days of operation. They are in the process of putting their fully scalable architecture into place. Paul notes that this will allow him to grow the business to incredible size without the need to make a major investment in servers:

Back in 1999 we had to invest millions of dollars in servers in order to handle the load. Today we are working on switching from our single beefy server to a cloud of servers on Amazon’s EC2 web service, giving us virtually infinite scalability with no cap ex expense.

Amazon Vice President Adam Selipsky was a speaker at last week's Web 2.0 Summit in San Francisco. As part of his talk, Adam unveiled the newest statistics on usage of Amazon S3 usage. Our 265,000-strong developer community has now stored over 10 billion (10,000,000,000) objects in S3 at a rate of 27,601 transactions per second. That's a lot of data and a lot of transactions!

A week or two before my most recent trip to Europe, I received a blog comment from Manoj Ranaweera, CEO of edocr.com. In a subsequent email conversation, Manoj invited me to spend a day of my trip in a city called Daresbury, and promised that it would be worth my while. I took a train ride to Daresbury and Manoj drove me to the Daresbury Innovation Centre, where I was happy to give an AWS presentation to the OpenCoffee Liverpool meetup and to meet with several startups, some of them already using one or more of our services. That's Manoj to my right in the picture.

Manoj and I met privately and I was able to learn all about edocr. Powered by EC2 and S3,  edocr simplifies and centralizes the process of putting business documents of all sorts online. Business can store and tag product literature, product updates, case studies, white papers, and newsletters in edocr. The site will generate a thumbnail for each document and it will also collect ratings, comments, and viewing statistics. Documents can be viewed within the site or downloaded for offline use. It is also easy to embed the documents in your own site or to link to the master copy on edocr.

While I was in Daresbury I also met with Mike Carter and Chris Haslam of Ixis. Ixis has built Football FanCast, an S3-powered site for UK football fans. The site includes blogs, podcasts, and forums for each team. Using S3, they are able to deal with traffic surges and to offer media downloads that would otherwise require a lot of storage, bandwidth, and other server infrastructure.

Paul Bissett of WeoGeo dropped me a note to make sure that I knew that their WeoCEO product is now available for public beta testing. The newest version includes enhancements to the product's stable IP address management feature, failure detection, and automatic scaling and load balancing. You can read more in Paul's recent blog post.

That's about all I have time for today. If this doesn't satiate your appetite for Amazon Web Services new, you may want to check out the AWS Buzz on del.icio.us. As I write this there are almost 1300 articles tagged.

-- Jeff;

New Resource Center Features: Code Search and Tech Centers

We've added two new features to the AWS Resource Center: Code Search and Tech Centers. Let's take an in-depth look at each one.

Code Search

The search box in the top right corner of each Developer Connection page now searches site content from within the Resource Center and also AWS code samples found in the Resource Center as well as on other sites including Google Code, SourceForge, and RubyForge.

Built around technology from Krugle, this feature aims to make it easier for developers to find the snippets of code or evaluate an existing project without having to download one of our monolithic .zip files.

For example, if you need to know how to use EC2's RunInstances function, you can find a myriad of snippets in a variety of languages. Search results within a code snippet are shown in context:

The Krugle technology also supports browsing through our code samples using the "Browse Code" button next to each sample.

If that's not enough, there's also an Advanced Search option for extended spelunking through code.

Finally, developers can add their own code by logging in to the Developer Connection and choosing to Submit Community Code.  This code will be indexed for use in the code search.

Tech Centers

The tech centers provide developers with an alternate, language-centric view of our developer content. We currently have centers for C#, PHP, Java, and Ruby; we'll add more based on available content and on requests from developers.

We anticipate that these new features will make it even easier for you to build really cool and innovative AWS applications. Let us know what you think!

-- Jeff;

CohesiveFT's Elastic Server

As part of my most recent trip to London I paid a visit to a suite of offices shared by LShift and CohesiveFT. I gave them an in-depth AWS presentation and we also discussed Cohesive's Elastic Server product.

Elastic Server simplifies the process of creating application stacks for use on EC2 and other virtual environments. Instead of the painstaking manual downloading, configuration, building, and installation that is otherwise required when creating a complex stack, Elastic Server provides a menu-driven selection and build process.

After choosing the desired web service environment, workflow framework, J2EE server, enterprise service bus, Java servlet container, page server, message queue and AMQP broker the user simply enters the desired virtualization parameters and initiates the build process. After a few minutes the build process spits out a a finished EC2 AMI. From there it is easy to launch one or more instances of the AMI using the EC2 Firefox Manager or another EC2 tool.

Each application stack is configured to include a copy of Cohesive's web-based Elastic Server Manager for easy management of each component.

If this sounds cool you should definitely watch the video to see how it works.

-- Jeff;

Amazon EC2 Gets More Muscle

The Amazon EC2 team just added Large and Extra Large instance types to EC2. The former "one size fits all" instance type is now known as a Small instance.

Large instances are 4 times larger in each dimension (CPU power, RAM, and disk storage) than the Small instances and cost $0.40 per hour. Extra Large instances are 8 times larger in each dimension and cost $0.80 per hour.

Both of the new instance types support 64-bit computing. While the Large Instance type offers 7.5 GB RAM, the Extra Large Instance Type offers 15 GB RAM (compared to the Small instance type and its 1.7 GB RAM). To help developer compare the new instance types, we are measuring the CPU capacity using a new term called an EC2 Compute Unit. The EC2 home page has more information about this.

When I first heard the news, I fell off my seat after reading the specs, especially '64-bit' and '15 GB RAM'. This is addressing one of the most common requests that we have heard from our developers.

With these new types of instances, developers will now be able run ravenous applications like large databases and/or compute-intensive tasks like simulations. Most importantly, they will be able to mix-and-match based on their infrastructure needs. Some Ideas that I can think of are:

• Small-scale user : 1 Small instance running the entire month (Website Hosting)
• Medium-scale user:  4 Small instances, 2 Large instances (Social Networking App)
• Compute intensive on-demand parallel user: 400 instances for 72 hours (Hadoop Cluster)
• High-perf user: 20 Extra Large instance for 14 days (Biotech Drug Synthesis or Render Farms)
• Database or file share hosting user:  8 Large instances running the entire month (Memcached-based Applications)
• Mixed large-scale user: 16 small instances, 4 large instances, 2 extra large instances, running entire month (Large Web-Scale Application)

Imagine the new possibilities!

If you have more ideas for how you would use these new instances, I would love to know.

I have also updated our AWS Simple Monthly Calculator with the new Instance Types where you can get estimate of your monthly bill based on your usage.

We are working hard to improve our products based on the feedback that you provide us. Keep the excellent feedback coming in!

-- Jinesh

Amazon S3 At Your Service

Virtually everyone on the Amazon Web Services team has occasion to interact with our developer customers from time to time. This rich source of product feedback gives us a lot of insight into ways that we can do an even better job of meeting their needs as we grow and enhance each of our web services.

Some of the developers building applications with Amazon S3 have been asking us about an SLA, or Service Level Agreement. An SLA defines the minimum acceptable level of performance from a service along with some sort of penalty for not meeting expectations. A typical SLA actually defines a performance or reliability boundary which is somewhat lower than what the system is actually designed, built, and expected to deliver.

We know that many of our customers, including a multitude of teams within Amazon, are using S3 in mission-critical ways and need a formal commitment from us in order to make commitments to their own users and customers.

After talking to many developers to make sure that we fully and precisely understood what the term "SLA" meant to them, we were able to start defining one that was appropriate for S3.

I am very happy to announce that, effective October 1, 2007, The Amazon S3 Service Level Agreement is in effect.

This SLA has been in the works for a while and we take the commitments made in this document quite seriously. We knew that S3 had to meet the very high performance and reliability goals set by our internal clients. We strongly believed that meeting this level of operational excellence would be good enough for our external users as well. Before we published our SLA, we wanted to get a better sense of how our external developers were making use of S3. With well over 5 billion objects under management, we now understand the usage patterns and properties needed to make an informed commitment.

You can read the entire document to see how this will work. Basically, we commit to 99.9% uptime, measured on a monthly basis. If an S3 call fails (by returning a ServiceUnavailable or InternalError result) this counts against the uptime. If the resulting uptime is less than 99%, you can apply for a service credit of 25% of your total S3 charges for the month. If the uptime is 99% but less than 99.9%, you can apply for a service credit of 10% of your S3 charges.

We're committed to providing a highly available service which meets the needs of current and future customers. This new SLA is our way of formalizing that commitment, letting you know what the minimum expected level of performance will be.

As is the norm with agreements like this, there's some fine print and you should definitely read it yourself to learn more.

--Jeff;

Two new locations for The Start-Up Project: San Diego and Los Angeles

Start-Up Project is an AWS-Exclusive Event for Start-Ups. It started with Seattle, then San Francisco, Silicon Valley, New York City, Boston. Now its has two new locations: Los Angeles and San Diego. This is probably your last chance this year to attend and interact with AWS Business Development Managers, Directors and Developers from Amazon Web Services.

Los Angeles Open RSVP
Tuesday, October 23rd, 2-7pm
V Lounge, 2020 Wilshire Blvd, Santa Monica, CA, 90403

San Diego Open RSVP
Monday, October 22nd, 2-7pm
Stingaree, 454 6th Ave, San Diego, CA 92101

If you own a start-up or thinking of starting one, this event is for you.

I will be also presenting at San Gabriel .Net Users Group on 17th of October. I will be in the SoCal area on 18th and 19th. Please use our Evangelists Wiki to schedule a meeting if you/your organization would like to know more about Amazon Web Services while we are in the town. I am always eager to meet with the developers 1:1 over lunch/coffee to learn how you are implementing our services and gather more feedback for our service teams. You can contact us at evangelists at amazon dot com.

--Jinesh

Amazon Web Service User Groups: A Follow-Up Report

Recently I blogged about the formation of the first AWS Users Group in Rochester, NY. This is a report about that historic meeting on September 20th, as well as a bit about several other AWS user groups.

Just before the meeting we ordered pizza for a crowd, not knowing whether or not one would materialize. No worries about that point! Thanks to the hard work of Mitch Garnaat and David Kavanagh, the turnout was outstanding. Attendees included representatives from academia, major corporations, hobbyist developers, and everything in between. A show of hands really surprised me--just over half the crowd had experience with Amazon Web Serivces. That meant the other half turned up on faith, because someone told them that "you've got to see this". Wow!

Reuven Cohen came over from Toronto to check things out in preparation for an event back home (see below). Sounds like the two groups may set up a "sister city" relationship.

At the end of the meeting we had one way to measure how successful the evening was: $RemainingPizza = 0... RawSug's next meeting will be next meeting is on October 18th at the Pittsford Library.

Rochester AWS Users Group (before the room filled)

AWSome
Innovation happens on all fronts--with AWSome being an excellent name for a user group. :) The second-ever AWS user group met one day later in Meetup format in Palo Alto, California. They met on a Friday night. That's true Valley Passion, because most people spend Friday nights unwinding.

According to a follow-up email from Sebastian Stadil, one of the organizers...

"We gave an overview of each of Amazon's web services, their use, advantages and drawbacks, and then discussed openly what we expected to get out of the meetup: meet people, and encourage the use of AWS among the startup community."

The demographics were young entrepreneurs, 20-35 years of age, having been or being involved in a startup. Most came to get informed, some to learn best practices, a few to get advice from more knowledgeable users. For all subsequent meetups we will have one or two speakers present use cases, or how they use AWS and for what purpose. This should be very interesting!

Toronto
In mid November Toronto will hold their first meeting on November 15th at 7pm, and current plans are to hold it at the Indoor Playground. Actual venue may change if the list of interested people continues to grow at its current rate. Already there are more people signed up for this group than any other that I'm aware of! The brains behind this event include Reuven Cohen of Enomaly and Kevin Thomason of AideRSS (among others).

By the way, I plan to attend this meeting--fresh off the plane from Europe.

Let us know if you'd like to form an AWS User Group in your area!

-- Mike

Second Life Developer Chat Schedule for October 2007

Here's the Second Life chat schedule for October. All of the chats will take place at 10 AM PST on the Amazon Developers Islands in Second Life. If you are new to Second Life and have trouble locating the island, simply log in 15 minutes or so before the chat and send me ("Jeffronius Batra") an Instant Message (IM). I will help you. Once you have Second Life installed you should be able to get to the right place by clicking here.

• Thursday, October 4th
• Thursday, October 18th
• Thursday, October 25th

We discuss the various Amazon Web Services, application architectures, issues, use cases, and lots more.

Hope to see you there!

-- Jeffronius;

Rememble - Create Digital Memories

One of the benefits of my team's self-serve scheduling system is that we get to meet face to face with many developers. While I was in London this past July I met with Gavin O'Carroll of Rememble.com .

Supported in part by the UK's National Endowment for Science, Technology and the Arts (NESTA), Rememble is the concrete realization of Gavin's long-term dream of creating a system to capture and organize "Digital Moments."

A Digital Moment could be a piece of text, a picture, a sound clip, or even a video. Each moment is date stamped and can be tagged, shared, and commented on. You can upload media files directly from the site, you can send them to Remeble from your cell phone using MMS (except from within the US) or SMS, from your Twitter account, or even from your collection of Flickr photos. You can use Rememble to record and chronologically organize the moments in real time, or you can upload them after the fact and set the appropriate dates yourself.

The moments are then organized into a timeline, which can be private or shared. Groups can create and collectively add to shared timelines. This could be an excellent way to organize and share pictures after a reunion or other family event.

Rememble launches today and it is cool to know that it is powered by Amazon S3 and Amazon EC2.

Congratulations to Gavin for making his dream a reality!

-- Jeff;

Start-Up Project Customer Presentations Rock!

For those who were eagerly waiting for the Start-Up Project Presentations, Here you go!

The Start-Up Project was an AWS-exclusive event that took place in Boston, New York City, San Francisco and Silicon Valley. The next locations are Los Angeles and San Diego.

These presentations have a ton of "Lessons Learned" embedded within. Each presentation is filled with Best Practices, reference-architecture diagrams, approximate monthly bill they paid to Amazon, and some simply cool strategies. These are experiences from those who are running Amazon Web Services in Production. It showcases a nice variety too - from facebook apps to video transcoding and from digital media to geo-spatial applications - a showcase of how each business/industry/vertical is leveraging AWS.

We are really thankful to all. Paul, Don, Theron, James, Nathan, Jon, Joyce, Oren, Ilya, Francis, Kevin, Brad, Sean, Jonathan, Kris, Kyle, Mark, Max, Stevie and all those who presented. Thank You for sharing your experiences with us.

WeoGeo - Organic, Self-Healing Systems For AWS

 

Geezeo - Educated Financial Decisions

Praxeon - Getting to Market with Amazon AWS

AideRSS - Taming the RSS Beast

Cruxy - Digital MarketPlace and Video Transcoding Service

Mashery - On-Demand Business and technical Infrastructure for all Web Services providers

BoozeMail - Hosting Facebook Apps on AWS

SlideShare.net - Bootstrapping SlideShare with AWS (using S3 to avoid a VC)

SmugMug - Set Amazon's Servers on Fire, Not Yours

ooyala - High-Quality Video

Animoto - Blog Post

I will add more as they become available.

-- Jinesh

"HelloWorld" Facebook Application AMI

I was quite curious to see how facebook applications can be built which use Amazon S3 and Amazon EC2 so I spent some time creating a "HelloWorld" facebook application that lists your Amazon S3 objects given the bucket name (basically integrate Amazon S3 libraries with Facebook libraries). I bundled up my code/configuration and created a Public AMI so that facebook developers can simply re-use my configuration and host their app in Amazon EC2.

AMI ID: ami-74f3161d
AMI Manifest: aws-facebook-app/image.manifest.xml

This Amazon Machine Image is pre-configured and ready-to-go for hosting your Facebook Application. There are some simple steps listed in our Resource Center Public AMIs page that will help you get started.

So now you have solid scalable infrastructure to back you up, innovative facebook platform to play with and all you need is the Killer Idea!

If you think this is helpful, let me know through comments. We could extend this AMI and build an Auto-Scaling module around the app so that we can simply "Auto-Scale Facebook Applications" out of the box and never worry about servers when you get famous overnight.

Thoughts?

--Jinesh