Amazon Web Services Blog

AWS Weeks in Review - December 17 & 24, 2012

Due to the holidays, today's report covers the past two weeks!

Monday,  December 17	You can now copy an EBS Snapshot between AWS Regions.
Tuesday, December 18	We added an Analytics Dashboard and real-time comparison of rank expressions to Amazon CloudSearch. We added environment resource support and an updated PHP runtime to Elastic Beanstalk.
Thursday, December 20	We announced that Amazon ElastiCache now supports the Virtual Private Cloud.
Friday, December 21	We launched the new EC2 High Storage instance family. We announced that the AWS Data Pipeline is now ready for use. Elastic MapReduce now supports IAM Roles. We announced CloudFormation template editors for Visual Studio and Eclipse. We announced even more features for AWS GovCloud (US).
Thursday, December 27	We introduced Root Domain Website Hosting for Amazon S3. We announced that AWS CloudFormation now supports EC2 block device mapping.
Friday, December 28	Amazon Elastic MapReduce can now make use of the High Storage (hs1.8xl) instances in US East (Northern Virginia).

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

-- Jeff;

Root Domain Website Hosting for Amazon S3

As you may already know,  you can host your static website on Amazon S3, giving you the ability to sustain any conceivable level of traffic, at a very modest cost, without the need to set up, monitor, scale, or manage any web servers. With static hosting, you pay only for the storage and bandwidth that you actually consume.

S3's website hosting feature has proven to be very popular with our customers. Today we are adding two new options to give you even more control over the user experience:

• You can now host your website at the root of your domain (e.g. http://mysite.com).
• You can now use redirection rules to redirect website traffic to another domain.

Root Domain Hosting
Your website can now be accessed without specifying the “www” in the web address. Previously, you needed to use a proxy server to redirect requests for your root domain to your Amazon S3 hosted website. This introduced additional costs, extra work, and another potential point of failure. Now, you can take advantage of S3’s high availability and scalability for both “www” and root domain addresses. In order to do this, you must use Amazon Route 53 to host the DNS data for your domain.

Follow along as I set this up using the AWS Management Console:

1. In the Amazon S3 Management Console, create an S3 bucket with the same name as your www subdomain, e.g. www.mysite.com. Go to the tab labeled Static Website Hosting and choose the option labeled Enable website hosting. Specify an index document (I use index.html) and upload all of your website content to this bucket.
   
   
2. Create another S3 bucket with the name of the root domain, e.g. mysite.com . Go to the tab labeled Static Website Hosting, choose the option labeled Redirect all requests to another host name, and enter the bucket name from step 1:
   
   
3. In the Amazon Route 53 Management Console, create two records for your domain. Create an A (alias) record in the domain's DNS hosted zone, mark it as an Alias, then choose the value that corresponds to your root domain name:
   
   Create an Alias (A) record and set the value to the S3 website endpoint for the first bucket (the one starting with www).

Redirection Rules
We're also enhancing our website redirection functionality. You can now associate a set of redirection rules to automatically redirect requests. The rules can be used to smooth things over when you make changes to the logical structure of your site. You can also use them to switch a page or a related group of pages from static to dynamic hosting (on EC2 or elsewhere) as your site evolves and your needs change.

Amazon CTO Werner Vogels has already started using root domain support for his blog. Check out his post for more information. Our walkthrough  on setting up a static website using Amazon S3, and see the Amazon S3 Developer Guide contains even more information.

If you are looking for some sites to help you build and maintain a static web site, you may enjoy Mick Gardner's recent post, An Introduction to Static Site Generators.

-- Jeff;

http://www.npr.org/rss/podcast/podcast_directory.php

Even More Features for AWS GovCloud (US)

What's GovCloud?

AWS GovCloud (US) is a gated community cloud designed to support the compliance needs of customer workloads with direct or indirect ties to U.S. Government functions, services, or regulations. The AWS GovCloud (US) framework adheres to U.S. International Traffic in Arms Regulations (ITAR) requirements.

Hot on the heels of my last GovCloud post, I've got even more good news today. Two more AWS services are now available for use in AWS GovCloud (US).

If you are part of the US Federal Government, a US state or local government, if you provide services to them, or if you need to follow Federal regulations on Controlled Unclassified Information (CUI) data protection, you can now make use of the following services:

Amazon Relational Database Service (RDS) - Launch database instances running MySQL, Oracle Database, or Microsoft SQL Server and let RDS take care of all of the common database administration and scaling tasks for you.

Amazon DynamoDB - Use this fully managed NoSQL database to build highly scalable dynamic applications while taking advantage of fast, SSD-based storage and automatic replication across three Availability Zones.

As I mentioned before, we recently added support for High Performance Computing, Elastic Load Balancing, Auto Scaling, CloudWatch Alarms, the Simple Notification Service, the Simple Queue Service, and the ElasticWolf management application.

Putting it all together, you now have access to a very broad and powerful set of services in AWS GovCloud (US). Here are some of the ways that our other AWS customers have been using GovCloud and AWS:

BioSense 2.0
The US Centers for Disease Control and Prevention hosts the BioSense 2.0 application on AWS. Watch this video to learn how they use Amazon S3, Amazon EC2, Elastic MapReduce, and the Amazon Simple Email Service (for push notifications) to build a system that meets their needs for privacy, security, and availability while saving money:

The BioSense application gives local, state, and federal officials the power to monitor and assess a set of symptoms that, taken together, characterize a disease or other threat to public health.

NIH Microbiome Cloud
http://www.ncats.nih.gov was established to speed up the basic scientific discovery for patient therapies and treatments.  A new website was created on AWS GovCloud (US) to help catalyze the generation of innovative methods and technologies that will enhance the development, testing and implementation of diagnostics and therapeutics across a wide range of human diseases and conditions.

For more information about the AWS GovCloud (US), visit the GovCloud site or contact us.

-- Jeff;

AWS CloudFormation Template Editors for Visual Studio and Eclipse

AWS CloudFormation gives you the power to launch an entire stack of AWS resources using a declarative template. The templates are JSON-formatted text files that define all of the necessary resources, their attributes, and the relationships between them.

Today we are introducing a new CloudFormation editor to make it even easier for you to create the templates using Visual Studio or Eclipse. You can:

• Browse the list of available AWS resources to insert into your template.
• Insert definition blocks for AWS resources that let you fill in the blanks for config.
• Create and update your CloudFormation stacks straight from your template in Visual Studio or Eclipse.

The CloudFormation template editor understands the template format and provides intelligent assistance as you're editing.  You can use code completion with inline descriptions to quickly find and insert definition blocks for different AWS resources.  Then you simply configure your specific details into the placeholders. The editor will also validate your template as you make changes to identify any problems in the file.  Once you define your templates, you can use them to create and update your stack of AWS resources right from within Visual Studio and Eclipse.

Here are some screen shots of the editor in action, first editing a template:

Second, deploying the finished template:

The editor is available as part of the AWS Toolkit for Visual Studio and the AWS Toolkit for Eclipse. Install the appropriate toolkit now and give it a whirl.

-- Jeff;

AWS Data Pipeline - Now Ready for Use!

The AWS Data Pipeline is now ready for use and you can sign up here.

As I described in my initial blog post, the AWS Data Pipeline gives you the power to automate the movement and processing of any amount of data using data-driven workflows and built-in dependency checking. You can access it from the command line, the APIs, or the AWS Management Console.

Today l'd like to show you how to use the AWS Management Console to create your own pipeline definition. Start by opening up the console and chosing Data Pipeline from the Services menu:

You'll see the main page of the Data Pipeline console:

Click on Create Pipeline to get started, then fill in the form:

With that out of the way, you now have access to the actual Pipeline Editor:

At this point you have two options. You can build the entire pipeline scratch or you can use one of the pre-defined templates as a starting point:

I'm going to use the first template, Export DynamoDB to S3. The pipeline is shown on the left side of the screen:

Clicking on an item to select it will show its attributes on the right side:

The ExportSchedule (an item of type Schedule) specifies how often the pipeline should be run, and over what time interval. Here I've specified that it should run every 12 hours for the first 6 months of 2013:

The ExportCluster (a Resource) specifies that an Elastic MapReduce cluster will be used to move the data:

MyDynamoData and MyS3Data specify the data source (a DynamoDB table) and the destination (an S3 bucket):

An Amazon SNS topic is used to provide notification of successes and failures (not shown):

Finally, MyExportJob (an Activity) pulls it all together:

Once the pipeline is ready, it can be saved and then activated:

Here are some other resources to help you get started with the AWS Data Pipeline:

• AWS Data Pipeline Home Page
• AWS Data Pipeline Developer Guide
• AWS Data Pipeline API Reference
• AWS Data Pipeline FAQ

And there you have it! Give it a try and let me know what you think.

-- Jeff;

 

The New EC2 High Storage Instance Family

In our continuing quest to ensure that Amazon EC2 is applicable to an incredibly broad array of use cases, we are launching a new instance family today.

The High Storage Eight Extra Large (hs1.8xlarge) instances are a great fit for applications that require high storage depth and high sequential I/O performance. Each instance includes 117 GiB of RAM, 16 virtual cores (providing 35 ECU of compute performance), and 48 TB of instance storage across 24 hard disk drives capable of delivering up to 2.4 GB per second of I/O performance.

This instance family is designed for data-intensive applications that require high storage density and high sequential I/O -- data warehousing, log processing, and seismic analysis (to name a few). We know that these applications can generate or consume tremendous amounts of data and that you want to be able to run them on EC2.

The storage on this instance family is local, and has a lifetime equal to that of the instance. You should think of these instances as building blocks that you can use to build a complete storage system. You should build a degree of redundancy into your storage architecture (e.g. RAID 1, 5, or 6) and you should use a fault-tolerant file system like HDFS or Gluster. Of course, you should also back up your data to Amazon S3 for increased durability.

Here's some nmon output with all of the disks visible:

You can launch multiple High Storage Eight Extra Large instances in a placement group for high bandwidth low latency networking between the instances.

High Storage instances are available now in the US East (Northern Virginia) Region and will be made available in other AWS Regions in the coming months. On-Demand pricing is $4.60 per hour in US East (Northern Virginia). You can also purchase one and three year Reserved Instances (Light, Medium, or Heavy). See the EC2 pricing page for more information.

-- Jeff;

The AWS Report - Jeff Lawson of Twilio

In the latest episode of The AWS Report, I spoke with Jeff Lawson of Twilio to learn more about what they do and how they use AWS:

As Jeff noted, Twilio has a presence in six AWS Regions and their service is available in 40 countries. He told me that none of their developers or admins are allowed to log in to their production systems since this is a source of human error.

-- Jeff;

PS - I wrote about Twilio for the first time way back in 2008. It was wonderful to hear how they'd been able to grow over the last 4 years, aided by AWS.

 

AWS Elastic Beanstalk - Environment Resource Support + Updated PHP Runtime

AWS Elastic Beanstalk can now provision and configure AWS resources to power your application. In conjunction with a new version of our PHP runtime, you get more control and more flexibility with less code.

In October, we announced the Elastic Beanstalk configuration files and talked about how they can help you configure EC2 instances without creating and maintaining custom AMIs. We have extended these configuration files to allow you to provision and configure resources such as SQS queues and DynamoDB tables for your Elastic Beanstalk application. The configuration file format is YAML-based, and you can find the details in the AWS Elastic Beanstalk Developer Guide.

We are also announcing an updated PHP runtime that supports configuration files as well as the seamless integration with Amazon RDS and Amazon VPC - see my previous blog post for details. The PHP runtime supports 5.3 and 5.4. You can also install dependencies using Composer. Visit the Elastic Beanstalk Developer Guide to learn more about the updated PHP runtime and for walkthroughs of how to deploy CakePHP and Symfony2.

I'd lke to show you how simple and powerful it is to use Elastic Beanstalk to create and configure the new environment resources. I’ll use the updated PHP runtime and show you how to configure DynamoDB backed session management. You can find the full example in the AWS Elastic Beanstalk Developer Guide.

Here's what you need to do:

• Launch an Elastic Beanstalk PHP environment (see the AWS Elastic Beanstalk Developer Guide).
• Create an IAM role with enough permission to read and write to the DynamoDB table (let’s call it “SessionRole”).
• Add an .ebextensions directory to your application.
• In that directory, create a configuration file called dynamodb.config with the following content:
  Resources:
    SessionTable:
      Type: AWS::DynamoDB::Table
      Properties:
        KeySchema:
          HashKeyElement:
            AttributeName: Index
            AttributeType: S
        ProvisionedThroughput:
          ReadCapacityUnits: 1
          WriteCapacityUnits: 1
• Add the role to the Elastic Beanstalk environment (this and all of the subsequent snippets also belong in the dynamodb.config file):
  AWSEBAutoScalingLaunchConfiguration:
      Properties:
        IamInstanceProfile:
          Fn::GetOptionSetting:
            OptionName: IAMInstanceProfileARN
            DefaultValue: "arn:aws:iam::123456789123:instance-profile/SessionRole"
• Pass the details of the DynamoDB table to the EC2 instances in your environment:
  files:
    "/var/app/sessiontable":
      mode: "000444"
      content: |
        `{"Ref" : "SessionTable"}`
        `{"Ref" : "AWS::Region"}`
• Add a CloudWatch Alarm so that you are alerted when requests to DynamoDB get throttled:
  SessionThrottledRequestsAlarm:
      Type: AWS::CloudWatch::Alarm
      Properties:
        AlarmDescription: { "Fn::Join" : ["", [{ "Ref" : "AWSEBEnvironmentName" }, ": requests are being throttled." ]]}
        Namespace: AWS/DynamoDB
        MetricName: ThrottledRequests
        Dimensions:
          - Name: TableName
            Value: { "Ref" : "SessionTable" }
        Statistic: Sum
        Period: 300
        EvaluationPeriods: 1
        Threshold:
          Fn::GetOptionSetting:
            OptionName: SessionThrottledRequestsThreshold
            DefaultValue: 1
        ComparisonOperator: GreaterThanThreshold
        AlarmActions:
          - Ref: SessionAlarmTopic
        InsufficientDataActions:
          - Ref: SessionAlarmTopic
• Make sure to include a require for the AWS SDK for PHP 2 in your composer.json file:
  "require": {
             "aws/aws-sdk-php": "*"
          }

• Finally, add the following code to your PHP application to connect it to the configuration file:
  <?php
  
  // Include the SDK using the Composer autoloader
  require 'vendor/autoload.php';
  
  use Aws\DynamoDb\DynamoDbClient;
  
  // Grab the session table name and region from the configuration file
  list($tableName, $region) = file(__DIR__ . '/../sessiontable');
  
  // Create a DynamoDB client and register the table as the session handler
  $dynamodb = DynamoDbClient::factory(array('region' => $region));
  $handler = $dynamodb->registerSessionHandler(array('table_name' => $tablename, 'hash_key' => 'Index'));
  
  ?>

As you can see, you can use this new Elastic Beanstalk feature to set up and configure the AWS resources needed by your application.

-- Jeff, with lots of help from Saad

EBS Snapshot Copy (Between Regions)

To make it even easier for you to build AWS applications that span regions, we're introducing a new EBS Snapshot Copy feature today. You can now copy EBS snapshots between EC2 Regions.

Why Copy?

So, why would you want to copy an EBS Snapshot from one AWS Region to another? Here are some of the more common use cases:

• Geographic Expansion - You want to be able to launch your application in a new Region.
• Migration - You want to be able to migrate your application from one Region to another. 
• Disaster Recovery - You want to back up your data and your log files across different geographical locations at regular intervals to minimize data loss and recovery time.

EBS Snapshot Copy simplifies each of these use cases by simplifying the copy process.

Ok, How do I Copy Snapshots?
You can now copy EBS Snapshots from one AWS Region to another. You can copy any accessible Snapshots that are in the "completed" status. This includes Snapshots that you created, Snapshots that were shared with you, and also Snapshots from the AWS Marketplace, VM Import/Export, and Storage Gateway. If you copy a Marketplace product to a new Region, you need to make sure that the product is supported in the destination Region.

You can initiate copies from the AWS Management Console or from the command line. You can also use the new CopySnapshot function from your own code. Here's how you initiate a copy operation from the Console:

After you choose the Copy Snapshot operation, the Console will ask you where you would like to copy the snapshot:

While the copy is underway, you can watch the progress by switching to the destination Region:

What Else Do I Need to Know?
Here are a few important facts about this new feature:

• The AWS Management Console shows the progress of a Snapshot copy in progress, you can check the percentage complete there.
• You can initiate multiple Snapshot Copy commands simultaneously either by selecting and copying multiple Snapshots to the same region, or by copying a snapshot to multiple regions in parallel.  The in-progress copies do not affect the performance of the associated EBS Volumes.
• The console-based interface is push-based; you log in to the source Region and tell the console where you'd like the Snapshot to end up. The API and the command line are, by contrast, pull-based and you must run them within the destination Region.

Pricing
You will be charged the regular data transfer prices when you copy snapshots between Regions. Once you have created a new Snapshot, you'll pay the regular price to store it in the destination Region.

Coming Soon
As you can see, EBS Snapshot Copy, will let you simplify your geographic expansion, data center migration, and disaster recovery. As a next step, we will implement copying of AMIs (Amazon Machine Images) between Regions. You'll be able to copy public and custom-created AMIs to the Region of your choice.

-- Jeff;

AWS Week in Review - December 10, 2012

Let's take a quick look at what happened in AWS-land last week:

Monday, December 10	We announced the results of the first EC2 Spotathon. Congratulations to the winners!
Wednesday, December 12	We added a number of new customer case studies: BioSense 2.0 from the US Centers for Disease Control and Prevention, MPS Technologies (creation and management of digital content), NASDAQ FinQloud,  Shire Pharmaceuticals, and Berkeley Amp Lab (machine learning and big data for genomics and cancer research).
Thursday, December 13	We announced AWS Detailed Billing Reports. We announced that AWS Auto Scaling now makes use of EC2 status checks.
Friday, December 14	This week AWS Marketplace added a variety of new products from TurnKey Linux and Bitnami including Spree, BambooInvoice, SiT! Support Incident Tracker and others.

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

-- Jeff;