Amazon Web Services Blog

The Cloud as a Platform for Platforms

Of the many things I love about AWS, I will mention three of my favorites in this blog post:

• AWS does not force developers to use any particular programming model, language, or operating system.
• AWS does not force developers to use the entire suite of services - they can use any of our infrastructure services individually or in any combination.
• AWS does not limit developers to a pre-set amount of storage, bandwidth, or computing resources they can consume - they can use as much or as little as they wish, and only pay for what they use.

Our customers love this flexibility. Today, a developer can run more experiments and achieve results much faster than before. If something does not work in a particular environment, the developer can drop that idea, click a few buttons, dispose all of his infrastructure and move on to the next experiment; starting with a fresh, new environment. Developers can try out several new ideas simultaneously by running multiple projects concurrently. Once the ideas are implemented, they can be further battle-tested using more resources in the AWS cloud until they become finished products. Developers love this because they are able to convert their concept/idea into a successful finished product quickly. As a result, we are seeing tremendous innovation happening at break-neck speed. The Cloud is becoming a platform for Innovation. 

The inherent flexibility of the AWS cloud enables customers to use it as a Platform in variety of different ways, including: 

• The AWS cloud as a Platform for Collaboration
• The AWS cloud as a Platform for Computation
• The AWS cloud as a Platform for Software and Data Delivery
• The AWS cloud as a Platform for Hot and Cold Storage
• The AWS cloud as a Platform for Research Development and Experimentation

Every day, I find customers in each of categories mentioned above. Some of them share their stories and architectures with us.

It does not stop there!

Its inspiring when I see the AWS cloud being used as a Platform for Platforms.

AWS is not only a rich platform to build solutions but also a platform for building specialized platforms. Customers can choose to either use the AWS cloud directly or take advantage of these value-added platforms. Customers can also mix and match platforms from this rich ecosystem.  

In this post, we look at some of the best examples of specialized platforms built on AWS:

Ruby Platforms
Heroku, as most of you may already know, is one of the early platforms built on top of Amazon Web Services. This "Instant Ruby Platform" enables any Ruby developer to take their existing Ruby code and move it to cloud. Customers of Heroku do not have to worry about scaling or managing their server farm, in case there is a success disaster. Heroku deploys a Ruby app in a single step without changing the app or the process. They recently launched commercially and offer a similar pay-as you-go pricing for enterprises and hobbyists. They also offer a free tier to try and test your prototypes.  By offering Deployment platform-as-a-Service on the top of reliable Amazon's platform, everybody wins.  The end-user gets all the things they need to build a modern web-scale application quickly while Heroku manages the "magic" (via “Slugs” and “Dynos”) of Ruby deployment without worrying about the complexity of maintaining and managing the underlying infrastructure.

Engine Yard offers a rich and open Ruby deployment platform on Amazon Web Services. Developers can take advantage of Engine Yard's pre-configured standardized stacks which makes it easy to deploy a Ruby on Rails application. Using Engine Yard's wizard-style web interface, developers can create the entire environment including different Unix packages and Ruby Gems to install, setting frequencies of database backups. They have a nice video on How to deploy your Ruby app on EC2 in 10 minutes. 

Language-agnostic Development Platform
I stumbled upon CodeRun – Online Development Platform on Amazon EC2 – few months ago and was tracking it closely. The interface looked just like Visual Studio but in-browser. End-user can code in PHP, AJAX, ASP.NET in an in-browser IDE which is fully hosted on multiple Amazon EC2 instances and then deploy the code (by clicking on “Debug” or “Run”) using several backend services that in-turn run on various other Amazon EC2 instances (all managed). Free accounts may share instances while premium accounts (which will be available in August) will run on stand-alone instances. Developers may deploy their code to Amazon EC2 more than once for testing, debugging and production purposes. Code snippets are shared on Amazon S3 and can be shared among developers (Check out AWS Code Samples). Amazon EBS is used to store users files and data while Amazon CloudFront is used distribute static files. Logs are stored on Amazon SimpleDB. They almost have a full house!

The platform includes a custom elasticity mechanism that monitors resource usage and performs automatic scaling based on dynamic set of predefined business rules. Developers can code in existing technologies (.NET/PHP/JS) and “outsource” scalability to CodeRun. With a single-click, you can deploy your app. CodeRun leverages AWS API to completely automate the deployment process. This includes allocating resources (instances, addresses and storage), copying files, synchronizing database structure and configuring the web servers. The entire platform is not fully baked yet but, I think, it has tremendous potential.

Voice Platform
Twilio is a voice platform built on the top of AWS. As Twilio’s website suggests, developer could build innovative voice apps like sales automation systems, order inquiry lines, CRM solutions, call routing apps, appointment reminders, custom voicemail apps. Platform developers at Twilio are focusing on building a powerful telephony platform on the top of Amazon Web Services. Twilio is drop-dead simple and easy to get started (“friction-free development”). With this simplicity, I think, it won’t be too long until a developer will be able to write a phone tree app that calls up all your friends from your social network about an upcoming party and get their RSVP over phone which can then be viewed on a website. Take a look at Twilio’s presentation (from the AWS Start-Up Tour in Seattle), and you will be convinced that they are AWS experts and they know what they are doing.

At VoiceCon, Siemens Enterprise Communications pre-announced that their Voice and Unified Communications product suite will be available as-a-Service on the AWS cloud. It will be interesting to see how this platform evolves.

Bottom line
Heroku, Engine Yard, Twilio, CodeRun are all different in nature and behavior. All of them are built using different technologies and methodologies. All are targeting different market segments. All share one thing in common. They are all built on AWS. All of them are built to scale and take advantage of flexibility. Innovation thrives in an environment that permits flexibility. AWS gives them the flexibility they need along with the scalability and elasticity their customers require.

This, to me, is very inspiring. What do you think ?

- Jinesh Varia (evangelists at amazon dot com)

Scaling to the Stars

Recently I blogged about The Server Labs, a consultancy that specializes in high-performance computing – including on Amazon Web Services.

Here’s another story that I found fascinating: nominally it is about how The Server Labs uses Amazon Web Services as a scale-out solution that also implements Oracle databases; however it’s really about space exploration (or should I say “nebula computing”). It began with an email asking whether there would be a problem running up to 1,000 Amazon EC2 High-CPU Extra-Large instances.

The Server Labs is a software development/consulting group based in Spain and the UK that works closely with the European Space Agency, and they needed to prove the scalability of an application that they helped build for ESA's Gaia project. In addition to the instances, they also requested 2 large and 3 X-Large instances to host Oracle databases that coordinate the work being performed by the high-CPU instances.

Gaia’s goal is to make the largest, most precise three-dimensional map of our Galaxy by surveying an unprecedented number of stars - more than one billion. This, by the way, is less than 1% of all stars! The plan is to launch a mission in 2011, collect data until 2017; and then publish a completed catalog no later than 2019.

I had the opportunity to see a PowerPoint deck created and presented by The Server Lab’s founder, Paul Parsons, and their software architect, Alfonso Olias, who is currently assigned to this project.

The deck explained that the expected number of samples in Gaia is 1 billion stars x 80 observations x 10 readouts, which is approximately equal to 1 x 1012 samples—or as much as 42 GB per day transferred back to Earth. There’s a slide in the deck that says “Put another way, if it took 1 millisecond to process one image, the processing time for just one pass through the data on a single processor) would take 30 years.”

As the spacecraft travels, it will continuously scan the sky in 0.7 degree arcs, sending the data back to Earth. Some involved algorithms will come into play in order to process the data; and the result is a fairly complex computing architecture that is linked to an Oracle database. Scheduling the cluster of computational servers is not quite so complicated, and is based on a scheduler that is focused on keeping each machine as busy as possible.

However the amount of data to process is not steady—it will increase over time. Which means that infrastructure needs will also vary over time. And of course idle computing capacity is deadly to a budget.

The opportunity to solve large computational problems usually turns to grid computing. No difference this time either – except that as mentioned above, the required size of the grid is not constant. Because Amazon Web Services is on-demand, it’s possible to apply just enough computational resources to the problem at any given time.

In their test, The Server Labs set up an Oracle database using an AWS Large Instance running a pre-defined public AMI. Then they mounted 5 EBS volumes of 100 GB each, and mounted them to the instance.

Then they created Amazon Machine Images (AMIs) to run the actual analysis software. These images were based on large instances and included Java, Tomcat, the AGIS software and an rc.local script to self-configure an instance when it’s launched.

The requirements break down as follows:

To process 5 years of data for 2 million stars, they will need to run 24 iterations of 100 minutes each, which works out to 40 hours running a grid of 20 Amazon EC2 instances. A secondary update has to be run once and requires 30 minutes per run, or 5 hours running a grid of 20 EC2 instances.

For the full 1 billion star project numbers extrapolate out more or less as follows: They calculated that they will analyze 100 million primary stars, plus 6 years of data, which will require a total of 16,200 hours of a 20-node EC2 cluster. That’s an estimated total computing cost of 344,000 Euros. By comparison, an in-house solution would cost roughly 720,000 EUR (at today’s prices) – which doesn’t include electricity or storage or sys-admin costs. (Storage alone would be an additional 100,000 EUR.)

It’s really exciting to see the Cloud used in this manner; especially when you realize that an entire set of problem solutions that were beyond economic possibility before the Cloud became a reality.

Mike

Webinar: How to Create Secure Test and Dev Environments on the Cloud

Amazon Web Services, CohesiveFT, and RightScale will participate in a webinar titled "How to Create Secure Test and Dev Environments on the Cloud."

Along with Michael Crandell and Edward Goldberg of RightScale, Simone Brunozzi of Amazon Web Services and Patrick Kerpan of CohesiveFT will show you how you can save time and money by running your entire testing application testing infrastructure in the cloud. They will discuss creation of an agile approach to rapid prototyping, creation of a test and development environment which replicates the final deployment environment, and will also show how to build a secure VPN environment.

The webinar is free but registration is required.

-- Jeff;

AWS Management Console Support for CloudFront

The AWS Management Console has a brand new tab:

This new tab contains full support for Amazon CloudFront. You can start distributing your content in minutes. You don't need to make a long term commitment and you don't need to download a client application. It is now even easier to access CloudFront in pay-as-you-go fashion.

After selecting the tab you will see a list of all of your CloudFront distributions:

You can select any distribution to see detailed information about it. You can also create a new distribution by clicking the Create Distribution button and filling in the resulting dialog:

You can also edit the properties of any of your existing CloudFront distributions. You can enable or disable logging, add and remove CNAMEs, and even enable or disable the distribution itself.

We've also got a brand new getting started video.

This is just one of many exciting new features that we have in store for users of the console. Stay tuned to this blog for further developments.

-- Jeff;

Gowalla - Location-based iPhone 3G Application

Gowalla is a location-based iPhone game. Currently focused on Austin, Boston, New York, and San Francisco, the application runs on any iPhone 3G with the location-based services turned on. You use Gowalla to collect "virtual souvenirs" by visiting specified locations in each city, neatly combining the virtual and the real worlds.

A product of Alamofire, Gowalla runs on EC2 and makes extensive use of S3, SQS, and SimpleDB. To learn more, you can watch Alamofire founder Scott Raymond's presentation at the Scotland on Rails conference. In the video, Scott talks about the pros and cons of using AWS, how they scaled their MySQL database tier, Facebook interaction, and much more.

-- Jeff;

S3Stat - Log Analysis for Amazon CloudFront and Amazon S3

S3Stat now has the ability to analyze the logs generated by Amazon CloudFront using the venerable Webalizer package. With your permission, S3Stat will enable CloudFront logging for the distributions of your choice. It will then run Webalizer each day and deposit the report in S3 for easy browsing. Of course, as the name implies, you can also use it to analyze the logs produced by S3.

The service is free for the first month and then $5 per month thereafter.

-- Jeff;

ScienceResearch Deep Web Search Engine - Now On Amazon EC2

The new version of ScienceResearch.com was officially launched today. Earlier this year I spoke with company president Abe Lederman and learned that this deep web science search engine provides a single point of access to over 400 publicly accessible science and technology collections.

The new version is hosted on EC2 and includes advanced search features such as relevance ranking, clustering (by topic, author, publication, or date), and exporting of search results in popular citation formats. The site indexes content in 15 different topic areas including agricultural sciences, astronomy & space, biology & nature, chemistry, computers & technology, defense technologies, earth & environmental sciences, energy, health & medicine, materials science, physics, and mathematics. It also indexes patents and science news.

 

Abe will be presenting a paper titled Journey to Ten Thousand Sources at the SLA (Special Libraries Association) conference this week. The paper contains a lot of interesting information about the architecture of this highly scalable federated search engine including details of the ways in which queries are partitioned into batches and then spread across multiple crawlers.

-- Jeff;

New Mechanical Turk HITS: Help McCann Erickson Find Some Favorite Picks

McCann Erickson is using the Mechanical Turk to collect information about people's favorite locations around the United States. They'd like you to be as specific as possible, about places that you have actually visited. They are looking for the places that other people don't always know about. The secret local hangout, or the place you would recommend to a friend or a relative.

The HITS come with some fairly specific requirements, so please read the instructions with care. You can get started here or by searching the HITS for the word 'favorites'.

-- Jeff;

Setting up a Load-Balanced Oracle Weblogic Cluster in Amazon EC2

Oracle recently released several middleware Amazon Machine Images (AMIs) to the community. I want to point out a detailed blog entry by Paul Parsons from The Server Labs that describes how to run a Weblogic Server on Amazon EC2, incorporating the load balancing feature inside the Monitoring, Auto Scaling and Elastic Load Balancing Service for Amazon EC2.

The following paragraphs are straight out of Paul’s full blog post, which of course you should read in its entirety if this topic is relevant to your interests.

Oracle recently made available a set of AMI images suitable for use with the Amazon EC2 cloud computing platform. I found the two images (32-bit and 64-bit) that contain Weblogic (along with Oracle Enterprise Linux 5 and JRockit) the most interesting of the lot. This article will explain how to set up a basic two-node Weblogic cluster using the 32-bit Weblogic image provided by Oracle with an Amazon Elastic Load Balancer (ELB). In future articles, I will demonstrate how to set up a more complicated cluster with Apache Web Server instances balancing the load between many weblogic cluster members.

You can set up a Weblogic cluster in EC2 in very little time which makes it great for testing complicated Weblogic setups without having to mess around on your local machine or trying to scrape together the necessary hardware. This type of configuration would also be suitable for deploying a production application, though you’d have to check the licensing implications with Oracle if you wanted to do this.

Note that this article assumes a basic level of familiarity with using Amazon web services.

Mike

AWS Management Console Support for Reserved Instances

The AWS Management Console now has support for our new Reserved Instances feature, previously announced in this very blog. You can now purchase new Reserved Instances and see your existing holdings with point-and-click ease.

The EC2 tab of the console has a new button:

You can see your existing set of Reserved Instances:

And you can purchase additional Reserved Instances:

This new feature should make it easier than ever for you to enjoy the cost benefit that comes with the use of one or more Reserved Instances.

-- Jeff;