Amazon Web Services Blog

AWS Management Console Support for Reserved Instances

The AWS Management Console now has support for our new Reserved Instances feature, previously announced in this very blog. You can now purchase new Reserved Instances and see your existing holdings with point-and-click ease.

The EC2 tab of the console has a new button:

You can see your existing set of Reserved Instances:

And you can purchase additional Reserved Instances:

This new feature should make it easier than ever for you to enjoy the cost benefit that comes with the use of one or more Reserved Instances.

-- Jeff;

Webinar: Getting Started with Amazon Elastic MapReduce

At 9:00 AM PST on Thursday, May 28th, we'll be conducting webinar to help you get started with Elastic MapReduce. In the webinar you will learn how Elastic MapReduce enables you to focus on crunching or analyzing your data without having to set up, manage, or tune a Hadoop cluster.

The webinar is free but you do need to sign up!

-- Jeff;

EC2 and Wowza Media Support Belgium's Largest Live Streaming Event

Imagine if you need to prepare the internet infrastructure needed to support a live event that:

• Will host a streaming video,
• Will start at a time that you can't control,
• Will be of an unknown duration,
• May attract a worldwide audience, and
• Happens once in a blue moon.

You can't buy the infrastructure, since you'll need it just once. Even then you wouldn't know how much to get. Traditional hosting would require you to make a long term commitment and you still wouldn't know how much to reserve. Cloud computing, once again, turns out to solve these problems and to enable hundreds of thousands of people to witness a relatively rare event -- the birth of an elephant in captivity!

On May 16th and 17th, over 350,000 unique visitors were able to watch the birth of Kai-Mook, the first elephant ever born at the Antwerp Zoo in Belgium. This amazing event was streamed live from a number of Amazon EC2 servers running the Wowza Media Server Pro product.

The statistics for this event were themselves elephantine! In advance of the event, about 50,000 people registered to receive an SMS alert when the birth was immminent. When the alert went out, the system scaled up quickly and was soon streaming live video to over 30,000 concurrent users, helped by some good media coverage including a BBC article. The users watched for an average of 1 hour and 35 minutes and the live event lasted for a total of 42 hours. Behind the scenes, 170 EC2 Large instances handled the streaming.

Note: The original version of the preceding paragraph claimed that the event pumped out 34 Gbps (gigabits per second) of data. It turns out that this was an optimistic and somewhat fuzzy estimate.

Videos, photos, and more are available at the Baby-Oliphant site, developed by interactive agency Boondoggle. CDN provider Rambla used a combination of AWS and their own infrastructure for this project. The original video is here. There's also a Flickr Photostream. Naturally enough, there's also a Kai-Mook blog and a complete genealogy. Weighing in at 80 kilograms, Kai-Mook has 12 siblings on his father's side and 3 more on his mother's.

As you can see, EC2 and Wowza Media Server Pro were able to support this event in fine style. Billing for the Wowza product is handled through Amazon DevPay so they didn't have to pay an arm and a leg (or a trunk?) for an excessive number of software licenses to support this unique event.

-- Jeff;

AWS Import/Export: Ship Us That Disk!

Many years ago, professor Andy Tanenbaum wrote the following:

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

Since station wagons and tapes are both on the verge of obsolescence, others have updated this nugget of wisdom to reference DVDs and Boeing 747s.

Hard drives are getting bigger more rapidly than internet connections are getting faster. It is now relatively easy to create a collection of data so large that it cannot be uploaded to offsite storage (e.g. Amazon S3) in a reasonable amount of time. Media files, corporate backups, data collected from scientific experiments, and potential AWS Public Data Sets are now at this point. Our customers in the scientific space routinely create terabyte data sets from individual experiments.

This isn't an issue that can be solved by getting a faster connection; even at the highest reasonable speed, some of these data sets would take weeks or months to upload. For example, it would take over 80 days to upload just 1TB of data over a T1 connection.

Customers with AWS storage requirements at the terabyte and petabyte level often ask us if they can sidestep the internet and simply send us a disk drive, or even a 747 full of such drives.

I can now say "Yes, you can!" Our new AWS Import/Export service allows you to ship your data to us. This service is now in a limited beta and you can sign up here. We'll take your storage device, load the data into a designated S3 bucket, and send your hardware back to you. The data load takes place in a secure facility with a high bandwidth, low-latency connection to Amazon S3. Once the data has been loaded in to S3, you can process it on EC2, and then store the results anywhere you would like -- back into S3, in SimpleDB, or on EBS volumes.

During the limited beta we are set up to accept devices with a USB 2.0 or eSATA connector, formatted as an FAT32, ext2, ext3, or NTFS file system. We are set up to handle devices that weigh less than 50 pounds and fit within an 8U rack. We are also happy to make special arrangements to accommodate larger and heavier devices. Last week after a conference talk one of the attendees asked me "Can we ship you our SAN?" In theory, yes, but we'd need to discuss the specifics beforehand.

It is easy to initiate a data load. Here's what you do:

1. Load the data onto your compatible storage device.
2. Create a manifest file per our specification. The file must include the name of the target S3 bucket, your AWS Access Key, and a return shipping address. You can also specify content types and S3 Access Control Lists in this file. You can also use the newest versions of third-party tools such as Bucket Explorer and S3 Fox to easily create manifest files from your S3 buckets.
3. Email the manifest file to a designated address with the subject CREATE JOB.
4. Await a return email with the subject RE: CREATE JOB.
5. Extract the JOBID value from the email.
6. Use the JOBID, manifest file, and your AWS Secret Access Key to generate and sign a signature file in the root directory of your storage device.
7. Ship your storage device, along with all necessary power and data cables, to an address that we'll provide to you. You can use one of the usual shipping company or a courier service.
8. Await further status emails.

Once we get the device we'll transport it to our data center and initiate the load process by the end of the next business day. A log file is created as part of the process; it will include the data and time of the load, and the S3 key, MD5 checksum, and size (in bytes) of each object. We'll reject unreadable files and those larger than 5GB and note them in the log. At the end of the process we'll send the device back to you at our expense.

In keeping with our model of charging for resources only as they are consumed, you will pay a fixed fee per device and a variable fee for each hour of data loading. There's no charge for data transfer between AWS Import/Export and an S3 bucket in the United States. Normal S3 Request and Storage charges apply.

Right now packages must be shipped from and returned to addresses in the United States. We do expect to be able to accept packages at a location in Europe in the near future.

Let's talk about security for a minute. You can choose to encrypt your files before you send them to us, although we don't support encrypted file systems. We track custody of your device from the time it arrives in our mailroom until it is shipped back to you. All personnel involved in the process have undergone extensive background checks.

Also, as you can probably guess from the name of the service, we have plans to let you transfer large amounts of data out of AWS as well. We will provide further information as soon as possible.

Here are some preliminary screen shots of the AWS Import/Export support in Bucket Explorer and S3 Fox:

We have also created an Import/Export Calculator:

If you have a large amount of data stored locally and you want to get it into Amazon S3 on a cost-effective and timely basis, you should definitely sign up for the beta now. You can read more about AWS Import/Export in the Developer Guide.

-- Jeff;

New SimpleDB Goodies: Enhanced Select, Larger Result Sets, Mandatory HTTPS

The Amazon SimpleDB team just cranked out a bunch of new features for the Select function, bumped up the maximum size of a batch of Select results, and also deprecated the old Query functions and the non-SSL interface. All of this new functionality is available through the 2009-04-15 version of the SimpleDB WSDL file. You can head straight to the documentation or you can keep reading to learn more.

The new functionality revolves around the Select function. Here's what's new:

• The like operator can now be used to search for a desired value embedded anywhere within an attribute value using the "%" character as a wildcard. Before this release the "%" had to be at the end of the search term, so that you could do searches such as " like 'Jeff%' ". As of this release you can do searches of the form " like '%Jeff%' ".
• You can now reference use itemNames() to refer to the current item's name in where and order by clauses.
• You can now use sorting even if the where clause contains the is null operator.
• The Select function can now return up to 2500 items at a time as long as they fit within the 1 MB result size limit. You can retrieve additional results by passing the NextToken value returned from the first call as input to the next one, and so forth.

As of this release the Query and QueryWithAttributes functions are considered deprecated and are not included in the WSDL file. Although these functions are still accessible through older versions of the SimpleDB WSDL file, it is time to start planning to migrate your code to use the Select function.

All requests to SimpleDB must now be made using HTTPS for security reasons.

-- Jeff;

New Features for Amazon EC2: Elastic Load Balancing, Auto Scaling, and Amazon CloudWatch

We are working to make it even easier for you to build sophisticated, scalable, and robust web applications using AWS. As soon as you launch some EC2 instances, you want visibility into resource utilization and overall performance. You want your application to be able to scale on demand based on traffic and system load. You want to spread the incoming traffic across multiple web servers for high availability and better performance. You want to focus on building an application that takes advantage of the powerful infrastructure available in the cloud, while avoiding system administration and operational burdens ("The Muck," as Jeff Bezos once called it).

Today, we are bringing you a lot closer to that world! The load balancing, auto scaling, and cloud monitoring features that I talked about earlier are now available. The features work together to help you to build highly scalable and highly available applications. Amazon CloudWatch monitors your Amazon EC2 capacity, Auto Scaling dynamically scales it based on demand, and Elastic Load Balancing distributes load across multiple instances in one or more Availability Zones. The measurements collected by Amazon CloudWatch provide Auto Scaling with the information needed to run enough Amazon EC2 instances to deal with the traffic load. Auto Scaling updates the Elastic Load Balancing service when new instances are launched or terminated to automatically scale the load-balanced capacity. You can instantiate, configure, and deploy these important system architecture components in seconds.

Amazon CloudWatch tracks and stores a number of per-instance performance metrics including CPU load, Disk I/O rates, and Network I/O rates. The metrics are rolled-up at one minute intervals and are retained for two weeks. Once stored, you can retrieve metrics across a number of dimensions including Availability Zone, Instance Type, AMI ID, or Auto Scaling Group. Because the metrics are measured inside Amazon EC2 you do not have to install or maintain monitoring agents on every instance that you want to monitor. You get real-time visibility into the performance of each of your Amazon EC2 instances and can quickly detect underperforming or underutilized instances.

Auto Scaling lets you define scaling policies driven by metrics collected by Amazon CloudWatch. Your Amazon EC2 instances will scale automatically based on actual system load and performance but you won't be spending money to keep idle instances running. The service maintains a detailed audit trail of all scaling operations. Auto Scaling uses a concept called an Auto Scaling Group to define what to scale, how to scale, and when to scale. Each group tracks the status of an application running across one or more EC2 instances. A set of rules or Scaling Triggers associated with each group define the system conditions under which additional EC2 instances will be launched or unneeded EC2 instances terminated. Each group includes an EC2 launch configuration to allow for specification of an AMI ID, instance type, and so forth.

Finally, the Elastic Load Balancing feature makes it easy for you to distribute web traffic across Amazon EC2 instances residing in one or more Availability Zones. You can create a new Elastic Load Balancer in minutes. Each one contains a list of EC2 instance IDs, a public-facing URL, and a port number. You will need to use a CNAME record in your site's DNS entry to associate your this URL with your application. You can use Health Checks to ascertain the health of each instance via pings and URL fetches, and stop sending traffic to unhealthy instances until they recover.

Here's how the services fit together:

All of this functionality is provided in web service and command-line form:

• You can call ListMetrics to get a list of statistics collected by Amazon CloudWatch, and then call GetMetricStatistics to retrieve them. Your call to GetMetricStatistics can include a number of parameters to specify the date range, desired metrics and statistics, metric granularity, and more. You can also use mon-list-metrics and mon-get-stats from the command line. There's a lot more info in the Developer Guide (HTML or PDF) and the Quick Reference Card.
• On the load balancing side, you start out by calling CreateLoadBalancer to create an Elastic Load Balancer, and will receive a DNS name in return. You can include a list of Availability Zones in the call or you can add them later using EnableAvailabilityZonesForLoadBalancer. From there you can add any number of health checks using ConfigureHealthCheck. A call to RegisterInstancesWithLoadBalancer will add your Amazon EC2 instances to the Elastic Load Balancer, and load balancing will commence. You can use elb-create-lb, elb-enable-zones-for-lb, elb-configure-healthcheck, and elb-register-instances-with-lb from the command line. Again, there's a lot more info in the Developer Guide (HTML or PDF) and the Quick Reference Card.
• For Auto Scaling you begin by calling CreateAutoScalingGroup, naming the group and providing the information needed to launch suitably configured Amazon EC2 instances. You then establish the scaling parameters using the CreateOrUpdateScalingTrigger function. The service will then launch Amazon EC2 instances as indicated by the scaling parameters. You can call DescribeScalingActivities at any point to fetch a list of recent scaling activities (instance launches and terminations). Command line equivalents are as-create-autoscaling-group, as-create-or-update-trigger, and as-describe-scaling-activities. Again, there's a lot ore info in the Developer Guide (HTML or PDF) and the Quick Reference Card.

If you're signed up for the Amazon EC2 service, you're already registered to use all of these new features and can begin using them via the web service APIs or Command Line tools. These new features are currently available in the U.S. region with EU region availability coming in the next few months.

You can use these services to make your AWS applications perform better without sacrificing application control freedom of development, choice of tools, speed of deployment, or any other kind of flexibility. You can be up and running with these new services in a matter of minutes. All of these new features are supported through our public forums and also through AWS Premium Support.

Morning Update:As always, a few interesting things came up after I put this post out last night:

1. Amazon CTO Werner Vogels wrote about these new features in his blog post, Automating the management of Amazon EC2 using Amazon CloudWatch, Auto Scaling and Elastic Load Balancing.
2. RightScale founder Thorsten von Eiken also wrote about them in his post, Amazon adds Load balancing, Monitoring, and Auto-Scaling..
3. There's a good discussion taking place on a Hacker News thread.

-- Jeff;

Quetzall CloudCache

Marc from Quetzall sent me a note about their new CloudCache product. CloudCache is a fast, lightweight key-value caching system designed for use within the cloud. Each key can optionally have an associated TTL (time to live). Once the TTL is reached the key and the associated value are removed from the cache.

Running on Amazon EC2 via DevPay, CloudCache is fast, with latency measured at just 1.5 ms. It can be run in multiple EC2 regions to minimize latency concerns. Customers can start small (1 cache) and grow large (1000 caches). CloudCache returns data in Ajax (JSON) or XML format. Bindings are available for Ruby, Java, PHP, and Python. Read the API documentation to learn more.

Since CloudCache is accessible via DevPay, you can sign up here and start using it right away.All pricing and usage charges are available on that page.

-- Jeff;

Cirrhus9 Cloud Adoption Webinar - May 28th

Mike from Cirrhus9 just told me that they'll be hosting a Cloud Adoption Webinar on the 28th of May.

In the webinar you will learn about the factors which influence cloud adoption rates and how a company's culture can affect the adoption curve. You'll also get to hear success stories of successful migrations of infrastructure into the cloud.

The webinar is free but you need to register!

You can also sign up for their monthly newsletter (The Cirrhus9 Cumulati) on their home page.

-- Jeff;

Amazon Simple Pay Subscriptions - Power Your Subscriptions - Special Offers and a Free Webinar

We recently announced the release of Amazon Simple Pay Subscriptions. You can use this product to charge your customers on a recurring basis after a single authorization from the customer. If you offer digital content subscriptions, collect membership dues on a periodic basis, or provide premium services on your website, this service is for you.

We launched the product in limited beta to gather feedback from businesses and developers to improve the product’s feature set and its ease of use. As a result we added a feature for businesses to manage subscriptions from the Amazon Payments website, improved the ability for merchants to cancel subscriptions on behalf of their customers and revamped the existing Amazon Simple Pay documentation.

Some of our friends who gave us invaluable feedback in the beta period and are using Amazon Simple Pay Subscriptions:

• Smugmug - One of the leading photo and video sharing and storage services. SmugMug also uses Amazon S3 and EC2.
• Social Calendar - The leading general purpose calendar on Facebook - designed to work well with the user's social graph. People use SocialCalendar to organize their social life - birthdays, anniversaries, holidays, get-togethers and to plan events and coordinate going to movies, concerts, sporting events, arts & theater, and restaurants with their friends and family.
• eJAMMING - The Collaborative Network for the World's Musicians Creating and Performing Together Online in Real Time.

To learn more about Amazon Simple Pay Subscriptions, please sign up for our Power Your Subscriptions Webinar. The webinar will be held on Thursday, May 14th at 11:00 AM PDT. If you offer digital content subscriptions, collect membership dues on a periodic basis, or provide premium services on your web site this webinar is for you. Sign up now!

In addition, we are also announcing a free payment processing offer through September 30, 2009 to businesses and organizations that begin accepting Amazon Payments using Checkout by Amazon and Amazon Simple Pay on or after April 29, 2009. Additional details and terms of the offer are available at www.amazonpayments.com/offer.

-- Jeff;

Amazon CloudFront Request Logging

We just added a handy new request logging feature to Amazon CloudFront. One line is written to a log file each time an object is accessed via CloudFront. This data (object name, access point, time of access, and so forth) can be used to generate usage reports using reporting tools.

Once enabled for a particular distribution (there's complete information in the Developer Guide), CloudFront will deposit new log files into the S3 bucket of your choice every hour or so. The logs are generated in the W3C Extended Log File Format, making it possible for you to process them using existing server log analysis tools. As you can see from the screen shot at right, the log includes a field for the edge location. You can use this field to analyze and understand the geographic distribution of your user base.

 

The free Cloudberry S3 Explorer makes it easy for you to control CloudFront logging. You need only enable the feature, enter the desired prefix for the log files, and choose an S3 bucket. This very handy program has an impressive feature list and should be of value to anyone using S3 and/or CloudFront. Andy from CloudBerry Lab also sent a picture of their new reporting feature, made possible by the CloudFront logs:

 

Good Data has created an on-demand service to analyze and report on CloudFront performance and traffic. The new service takes advantage of their ability to load, analyze, and report on large amounts of data. You can access your logs using a pre-built dashboard of reports, do ad hoc analysis, and correlate it with other internal or external data. You can embed the finished reports and charts in your corporate portal or wiki if you'd like. There's a free trial and an introductory video too.

 

We have also published a really cool code sample. This sample uses Amazon Elastic MapReduce and the Cascading framework to show how to process the new CloudFront log files in a scalable fashion. The sample reads any number of log files from a given S3 bucket, processes the files which represent a specified date range, and generates up to four report families: Client IP addresses, Object Popularity, Overall Volume, and Edge Locations. The reports are written to another S3 bucket. Here's one of the Edge Location reports, request counts by edge location:

 

I'm confident that additional tool and analysis support will emerge before too long. When it does, I'll cover them in a new post. We'll add them to the CloudFront Solutions Catalog.

Update: BucketExplorer has also added support for logging. Per their announcement, they have also added support for creating and managing distributions, the ability to map CNAMEs to distributions, batch-mode ACL updates (very handy for large S3 buckets), automated bucket backup, and lots more.

-- Jeff;