AWS OpsWorks - Flexible Application Management in the Cloud Using Chef

It’s been over six years since we launched Amazon EC2. After that launch, we’ve delivered several solutions that make it easier for you to deploy and manage applications.

Two years ago we launched AWS CloudFormation to provide an easy way to create a collection of related AWS resources and provision them in an orderly and predictable fashion and AWS Elastic Beanstalk that allows users to quickly deploy and manage their applications in the AWS cloud. As our customers run more applications on AWS they are asking for more sophisticated tools to manage their AWS resources and automate how they deploy applications.

AWS OpsWorks features an integrated management experience for the entire application lifecycle including resource provisioning, configuration management, application deployment, monitoring, and access control. It will work with applications of any level of complexity and is independent of any particular architectural pattern.

AWS OpsWorks was designed to simplify the process of managing the application lifecycle without imposing arbitrary limits or forcing you to work within an overly constrained model. You have the freedom to design your application stack as you see fit.

You can use Chef Recipes to make system-level configuration changes and to install tools, utilities, libraries, and application code on the EC2 instance within your application. By making use of the AWS OpsWorks event model, you can activate the recipes of your choice at critical points in the lifecycle of your application. AWS OpsWorks has the power to install code from a wide variety of source code repositories.

There is no additional charge for AWS OpsWorks. You pay only for the AWS resources (EC2 instances, EBS volumes, and so forth) that your application uses.

AWS OpsWorks is available today and you can start using it now!

AWS OpsWorks Concepts
Let's start out by taking a look at the most important AWS OpsWorks concepts.

An AWS OpsWorks Stack contains a set of Amazon EC2 instances and instance blueprints (which OpsWorks calls Layers) that are used to launch and manage the instances. Each Stack hosts one or more Applications. Stacks also serve as a container for the other user permissions and resources associated with the Apps. A Stack can also contain references to any number of Chef Cookbooks.

Each Stack contains a number of Layers. Each Layer specifies the setup and configuration of a set of EC2 instances and related AWS resources such as EBS Volumes and Elastic IP Addresses. We've included Layers for a number of common technologies including Ruby, PHP, Node.js, HAProxy, Memcached, and MySQL. You can extend these Layers for your own use, or you can create custom Layers from scratch. You can also activate the Chef Recipes of your choice in response to specific events (Setup, Configure, Deploy, Undeploy, and Shutdown).

AWS OpsWorks installs Applications on the EC2 instances by pulling the code from one or more code repositories. You can indicate that the code is to be pulled from a Git or Subversion repository, fetched via an HTTP request, or downloaded from an Amazon S3 bucket.

After you have defined a Stack, its Layers, and its Applications, you can create EC2 instances and assign them to specific Layers. You can launch the instances manually, or you can define scaling based on load or by time. Either way, you have full control over the instance type, Availability Zone, Security Group(s), and operating system. As the instances launch, they will be configured to your specifications using the Recipes that you defined for the Layer that contains the instance.

AWS OpsWorks will monitor your instances and report metrics to Amazon CloudWatch (you can also use  Ganglia if you'd like). It will automatically replaced failed instances with fully configured fresh ones.

You can create and manage your AWS OpsWorks Stacks using the AWS Management Console]. You can also use it via the OpsWorks API.

AWS OpsWorks in Action
Let's take a quick look at the AWS OpsWorks Console. The welcome page provides you with a handy overview of the steps you'll take to get started:

Start out by adding a Stack. You have full control of the Region and the default Availability Zone. You can specify a naming scheme for the EC2 instances in the Stack and you can even select a color to help you distinguish your Stacks:

AWS OpsWorks will assign a name to each EC2 instance that it launches. You can select one of the following themes for the names:

You can add references to one or more Chef Cookbooks:

Now you can add Layers to your Stack:

You can choose one of the predefined Layer types or you can create your own custom Layer using community cookbooks for software like PostgreSQL, Solr, and more:

When you add a Layer of a predefined type, you have the opportunity to customize the settings as appropriate. For example, here's what you can customize if you choose to use the Ruby on Rails Layer:

You can add custom Chef Recipes and any additional software packages to your Layer. You can also ask for EBS Volumes or Elastic IP Addresses and you can configure RAID mount points:

You can see the built-in Chef recipes that AWS OpsWorks includes. You can also add your own Chef Recipes for use at various points in the application lifecycle. Here are the built-in Recipes included with the PHP Layer:

Here's what my Stack looks like after adding four layers (yours will look different, depending on the number and type of Layers you choose to add):

Applications are your code. You can add one or more Applications to the Stack like this. Your options (and this screen) will vary based on the type of application that you add:

With the Stack, the Layers, and the Applications defined, it is time to add EC2 instances to each Layer. As I noted earlier, you can add a fixed number of instances to a Layer or you can use time or load-based scaling, as appropriate for your application:

You can define time-based scaling in a very flexible manner. You can use the same scaling pattern for each day of the week or you can define patterns for particular days, or you can mix and match:

With everything defined, you can start all of the instances with a single click (you can also control them individually if you'd like):

Your instances will be up and running before long:

Then you can deploy your Applications to the instances:

You have the ability to control exactly which instances receive each deployment. AWS OpsWorks pulls the code from your repository and runs the deploy recipes on each of the selected instances to configure all the layers of your app. Here’s how it works. When you deploy an app, you might have a recipe on your database Layer that performs a specific configuration task, such as creating a new table. The recipes on your Layers let you to simplify the configuration steps across all the resources in your application with a single action.

Of course, there are times that you may need to get onto the instances. AWS OpsWorks helps here, too. You can configure SSH keys for each IAM user as well as configure which IAM users can use sudo.

At the top level of AWS OpsWorks, you can manage the entire Stack with a couple of clicks:

I hope that you have enjoyed this tour of AWS OpsWorks. I've shown you the highlights; there's even more functionality that you'll discover over time as you get to know the product.

AWS OpsWorks Demo
Watch this short (5 minute) video to see a demo of AWS OpsWorks:

Getting Started
As always, we have plenty of resources to get you going:

• AWS OpsWorks User Guide
• AWS OpsWorks Walkthrough
• AWS OpsWorks Home Page

Looking Ahead
Today, OpsWorks provides full control of software on EC2 instances and lets you use Chef recipes to integrate with AWS resources such as S3 and Amazon RDS. Moving forward, we plan to introduce integrated support for additional AWS resource types to simplify the management of layers that don’t require deep control. As always, your feedback will help us to prioritize our development effort.

What Do You Think?
I invite you to give AWS OpsWorks a whirl and let me know what you think. Feel free to leave a comment!

-- Jeff;

 PS - Don't forget to read Werner's take on OpsWorks - Expanding the Cloud - Introducing AWS OpsWorks, a Powerful Application Management Solution.

Comments

[Nane Kratzke]

Sounds so great. Your pace of releasing new services is amazing ... give your customers a chance to keep the pace ;-)

[Jongleberry]

How does this compare with nodejitsu and heroku, specifically for node.js deployments? Would love some sort of comparison. Thank you!

[new AWS customer]

It's everything I ever wanted as a developer.

[David Mytton]

The UI looks to be particularly interesting - typically AWS functionality has been launched as API only but since OpsWorks is all about helping management, it makes sense to provide a graphical way to work with the tool. This perhaps shows a focus more towards enterprise users with the core services like compute and storage already defined and available to engineers, this will help larger users define their infrastructure much easier. Providing it for free is a clever move too.

[Julien Duponchelle]

Very interresting,

How can we deploy background job like Heroku workers? I imagine at this time we need to use custom cookbook.

[Ian]

This looks amazing! Quick question: is there a WSDL for the API anywhere? I can't seem to find it on the docs...

[James Toyer]

This does look good, and will take some of the pain out of using Chef as well as managing full application stacks.

I notice that right now it only supports Linux instances. Are there any plans to roll this out in the future to Windows instances?

[Chris Barclay]

@James Toyer: Although Windows isn't supported in this initial release, we're listening closely to what customers need and will be releasing frequent updates this year. Thanks for your feedback!

[Samuel Hale]

Any chance we'll see VPC support in the future?

It would be great if we could automate some of the tasks of assigning/managing/deploying various subnet groups, routing, and IP addresses.

[Chris Barclay]

@Samuel Hale: Thanks for the feedback on VPC support. We're listening closely to what customers need to prioritize our roadmap.

[Josh Wheatly]

Any plans to support puppet?

[Simon Beckett]

Second on VPC support, also in addition more OS choices or ideally tell us what is required to configure our own base ami that will work with ops works.

[Chris Barclay]

@Ian: We don't have a WSDL for the API at this time. We appreciate and collect all customer feedback.

[Sarang]

Sounds interesting....Any plans to support other app layer like Python

[Ali]

Any plans for puppet support?

[Eamonn O'Neill]

Hi - really like this direction. I was hoping to see the ability to launch any of the instances or AMIs I've already defined elsewhere. The limited choice of OS's at the minute means this can't be used for SAP software, and we're desperately trying to figure out the best management software for SAP customers. Any idea on release schedules for this service? Thanks.

[Gnep]

Eamonn, what OS is required to run SAP in your case?

[Chris Barclay]

@Simon Becket: Thanks for the feedback. What OS choices would you like to see? You can modify the base OpsWorks AMIs by adding packages to the Layer or Chef recipes.

[Chris Barclay]

@Sarang: Thanks for the feedback. Let us know what additional app layers you would like to see. We're listening closely to what customers need to prioritize our roadmap.

[Chris Barclay]

@Ali: We have no immediate plans for puppet support, but we're listening closely to what customers need to prioritize our roadmap.

[Simon Beckett]

@Chris Barclay: In a pinch, redhat, although ideally a Centos6 AMI. Given that everyone has their own preference, the ability to create your own opsworks compatible with your OS of choice would be excellent.

[Andrew Morley]

This looks great, but until it supports custom AMIs it isn't an option for us. Our puppet builds can take up to 30 minutes, and provisioning software as an instance comes online just won't give us the ability to scale as we need. We currently build AMIs in a dedicated environment and then push them out to our live environment. Only minor configuration changes are then required to be installed onto the new instance to make it functional.

When are we likely to see support for custom AMIs with opsworks?

[Harish Ganesan]

+1 for VPC

[Michael]

We currently do some of this via Puppet so this will help greatly once Puppet is supported.

[Srinivasa]

Any support for Play2 Framework?

[BenjaminRRR]

This is awesome stuff - just want to reiterate earlier comments, will be great to see Windows support. Will help us close the loop on a lot of things.

[Aldo]

Great stuff! it goes with the AWScourse I'll take. Thanks.