New Amazon EC2 Instance Type - The Cluster Compute Instance

A number of AWS users have been using Amazon EC2 to solve a variety of computationally intensive problems. Here's a sampling:

• Atbrox and Lingit use Elastic MapReduce to build data sets that help individuals with dyslexia to improve their reading and writing skills.
• Systems integrator Cycle Computing helps Varian to run compute-intensive Monte Carlo simulations.
• Harvard Medical School's Laboratory for Personalized Medicine creates innovative genetic testing models.
• Pathwork Diagnostics runs tens of thousands of models to help oncologists to diagnose hard-to-identify cancer tumors.
• Razorfish processes huge datasets on a very compressed timescale.
• The Server Labs helps the European Space Agency to build the operations infrastructure for the Gaia project.

Some of these problems are examples of what are called “embarrassingly parallel” computing.  Others leverage the Hadoop framework for data-intensive computing, spreading the workload across a large number of EC2 instances.

Our customers have also asked us about the ability to run even larger and more computationally complex workloads in the cloud.

It is clear that people are now figuring out that they can do HPC (High-Performance Computing) in the cloud. We want to make it even easier and more efficient for them to do so!

Our new Cluster Compute Instances will fit the bill. With Cluster Compute Instances, you can now run many types of large-scale network-intensive jobs without losing the core advantages of EC2: a pay-as-you-go pricing model and the ability to scale up and down to meet your needs.

Each Cluster Compute Instance consists of a pair of quad-core Intel "Nehalem" X5570 processors with a total of 33.5 ECU (EC2 Compute Units), 23 GB of RAM, and 1690 GB of local instance storage, all for $1.60 per hour.

Because many HPC applications and other network-bound applications make heavy use of network communication, Cluster Compute Instances are connected using a 10 Gbps network. Within this network you can create one or more placement groups of type "cluster" and then launch Cluster Compute Instances within each group. Instances within each placement group of this type benefit from non-blocking bandwidth and low latency node to node communication.

The EC2 API's, the command-line tools, and the AWS Management Console have all been updated to support the creation and use of placement groups. For example, the following pair of commands creates a placement group called biocluster and then launches 8 Cluster Compute Instances inside of the group:

$ ec2-create-placement-group biocluster -s cluster

$ ec2-run-instances ami-2de43f55 --type cc1.4xlarge --placement-group biocluster -n 8

The new instance type is now available for Linux/UNIX use in a single Availability Zone in the US East (Northern Virginia) region. We'll support it in additional zones and regions in the future. You can purchase individual Reserved Instances for a one or a three year term, but you can't buy them within specific cluster placement groups just yet. There is a default usage limit for this instance type of 8 instances (providing 64 cores). If you wish to run more than 8 instances, you can request a higher limit using the Amazon EC2 instance request form.

The Cluster Compute Instances use hardware-assisted (HVM) virtualization instead of the paravirtualization used by the other instance types and requires booting from EBS, so you will need to create a new AMI in order to use them. We suggest that you use our Centos-based AMI as a base for your own AMIs for optimal performance. See the EC2 User Guide or the EC2 Developer Guide for more information.

The only way to know if this is a genuine HPC setup is to benchmark it, and we've just finished doing so. We ran the gold-standard High Performance Linpack benchmark on 880 Cluster Compute instances (7040 cores) and measured the overall performance at 41.82 TeraFLOPS using Intel's MPI (Message Passing Interface) and MKL (Math Kernel Library) libraries, along with their compiler suite. This result places us at position 146 on the Top500 list of supercomputers. The input file for the benchmark is here and the output file is here.

Putting this all together, I think that we have put together a true fire-breathing dragon of an offering. You can now get world-class compute and network performance on an economical, pay-as-you-go basis.  The individual instances perform really well, and you can tie a bunch of them together using a fast network to attack large-scale problems. I'm fairly certain that you can't get this much compute power so fast or so economically anywhere else.

I'm looking forward to writing up and sharing some of the success stories from the customers who've been helping us to test the Cluster Compute instances during our private beta test. Feel free to share your own success stories with me once you've had a chance to give them a try.

Update - Here's some additional info:

• Werner Vogels: Expanding the Cloud - Cluster Compute Instances for Amazon EC2.
• TechCunchIT: AWS Launches Cluster Compute EC2 Instances For High Performance Applications.
• ZDNet: Amazon Web Services tackles high performance computing instances.
• And our own HPC Applications page.
  

-- Jeff;

Comments

[Jonathan Marston]

Coupling this instance type with the Relational Data Services would be a powerful combination.

One complication: the MySQL version would probably need to be based on PerconaDB or MySQL 5.5 to take advantage of the 8 cores.

http://broadcast.oreilly.com/2009/05/mysql-faster-better-and-still.html

[Chris Dagdigian]

Fantastic news, glad we can finally start testing the production release.

One question though -- the public Amazon CentOS AMI that boots off of EBS and runs wtih HVM says "Centos 5.4" in /etc/motd and in the descriptions but if you look at /etc/issue the OS claims to be Centos 5.5 based. Which one is it?

[Alex Kilpatrick]

Is there a plan to release a windows version of a cluster compute instance?

[Darren]

Do these instances have a multicast NIC enabled for UDP multicast networking between them only?

[Loweyj]

This is interesting, and could be very useful for smaller sites doing limited HPC work, I would argue that it is still cost prohibitive for sites doing 3-5 million compute hours annually, especially when considering bandwidth charges if dealing with large data sets. But would be very interesting for one-off or prototyping.

[AWS Evangelist]

Chris - The CCI reference AMI was originally created using CentOS 5.4. Since that time, CentOS has released 5.5, and invoking 'yum update' will automatically update from 5.4 to 5.5.

[Curious Ellie]

That term "Embarrassingly Parallel" sounds like it should be in the "Statistically Improbable" class....

[AWS Fan]

What is the largest number of instances that can be placed into a single Placement Group with full bisection bandwidth at 10GB?

[شات صوتي]

That term "Embarrassingly Parallel" sounds like it should be in the "Statistically Improbable" class....