New - Custom Metrics for Amazon CloudWatch

You can now store your business and application metrics in Amazon CloudWatch. You can view graphs, set alarms, and initiate automated actions based on these metrics, just as you can for the metrics that CloudWatch already stores for your AWS resources.

This new custom metrics feature can be used in two different ways:

1. You can add to the metrics collected for Amazon EC2 Instances, EBS Volumes, Elastic Load Balancers, and Relational Database Service DB Instances. The metrics that you store can be technical (system performance indicators) or business-related (user activity over the monitoring period).
2. You can store metrics for any generic resource. You can use CloudWatch to create a single, integrated storage and aggregation point for all of the metrics that you want to watch and to monitor.

You don't have to pre-define the new metrics and you don't have to pre-allocate storage space. Instead, you simply store new values and the new metric will be created automatically. You can monitor resources that run on EC2, or anywhere else on the Internet.

You can access all of your metrics using the ListMetrics function. Any graphing or visualization tools that have the ability to pull existing data from CloudWatch should be able to access user-defined metrics with little or no change.

Metric values can be stored using the new PutMetricData function or the mon-put-data command-line tool.

CloudWatch metrics are scoped within namespaces, and can be further qualified by up to 10 dimensions. For example, latency could be tracked for a pair of applications ("App1" and "App2") while keeping the values isolated from each other:

$ mon-put-data -namespace App1 -metric-name Latency -value 104
$ mon-put-data -namespace App2 -metric-name Latency -value 120

Dimensions are name-value pairs that can be used to expand on a metric's identity. For example, latency could be tracked on a per-host basis by adding a dimension to the example above:

$ mon-put-data -namespace App1 -metric-name Latency -dimensions "Host=host1" -value 104
$ mon-put-data -namespace App1 -metric-name Latency -dimensions "Host=host2" -value 180
$ mon-put-data -namespace App1 -metric-name Latency -dimensions "Host=host3" -value 90

Metrics that have different namespaces and dimensions are treated as discrete metrics. In this example, the application-level metrics are isolated from the host-level metrics. In other words, be sure to Put the metrics in the same format that you want to Get them.

CloudWatch lets you provide metric data from multiple sources. You can monitor scalable applications comprised of hundreds or even thousands of EC2 instances. New hosts can contribute to existing application-level metrics as they join the fleet. You can then retrieve the data for individual hosts or for the entire fleet.

Here's an example of the kinds of insights that can be gained when you use custom metrics. The following chart shows the amount of free memory (the jagged line) and the application's page latency:

As you can see, page latency increases as free memory decreases (looks like there's some garbage collection going on as well).

You can choose from among a list of standard units (Seconds, Bits, Bytes, Percent, Count, Bytes/Second, Bits/Second, and Count/Second) when you store metrics. Metrics with different units will not be included in CloudWatch's statistical calculations. All metrics are retained for a period of two weeks.

Once your metrics are stored in CloudWatch, you can treat them as if they were metrics on AWS resources. You can use the GetMetricStatistics function to calculate the Average, Min, Max, Sum, and Count for any time range and you can also graph the metrics in the AWS Management Console.

You can also create CloudWatch Alarms on your metrics. For example, if you monitor latency, you can set up an SNS notification to email you when average latency remains elevated for 15 minutes.

You can use any of the metrics to AutoScale your fleet of Amazon EC2 instances.This means that you could scale up and down on metrics that are only indirectly related to system load or performance, but that may be better indicators of application performance or customer experience: number of registered users, number of active sessions, page load times, or error rates.

The Custom Metrics are priced on a per-metric basis. You'll pay $0.50 per metric per month regardless of the number of values that you store for the metric. We are also reducing the price of EC2 Detailed Monitoring to $3.50 per month (7 metrics * $0.50 / metric). You can store 10 metrics per month at no charge; these can be used for both Custom Metrics and EC2 Detailed Monitoring metrics.

The AWS SDKs for .Net, Java, and PHP now support this new feature.

The HP SiteScope product now includes support for CloudWatch's new user-defined metrics. You can configure SiteScope to report any desired metrics to CloudWatch, and then use them to drive scaling decisions, raise alarms, or simply chart them. Integration is as easy as editing the new function in the SiteScope configuration file and creating a new SiteScope tag to define the data to be routed to CloudWatch. You can also embed HP diagnostics probes in your AMIs so that newly launched EC2 instances appear in the HP Diagnostics UI. Read more about both features in the HP APM (Application Performance Management) blog.

Some of our customers started to use this new feature during the private beta. Here's what they had to say:

Loggly.com provides log management and analysis for customers such as about.me, an AOL Inc. company that ties together multiple social networking site profiles into a unified personal homepage. Kord Campbell (Founder and CEO) told us:

Loggly makes it easy for businesses to handle the large amounts of log data they generate every day, and now customers like AOL’s about.me are asking us to deliver actionable alerts on subsets of the log data “About.me needs to know right away when response times from the social networking sites slow down or errors arise. By integrating with Amazon CloudWatch alarms, we can now automatically deliver this information to about.me and retain our focus on the core logging service.

 

Bizo is a business-to-business marketing platform operated as an online service. According to Larry Ogrodnek (Bizo's Director of Engineering):

Bizo is hyper-focused on the performance of our systems and data that our B2B marketing customers count on every day. The new functionality in Amazon CloudWatch enables us to monitor the availability of our back-end services without managing our own monitoring system. We can now gain insight into the health and performance of our systems to a level of detail that simply wasn’t cost-effective before.

 

BitRock provides simplified deployment of open-source applications through its BitNami.org website. Daniel Lopez Ridruejo (BitRock's Founder) told us that:

BitNami Cloud Hosting helps businesses deploy and manage applications on AWS, by automatically integrating those applications with Amazon CloudWatch, we’re able to easily help our customers monitor their Amazon EC2 instances and Amazon EBS volumes and scale up and down based on application-level metrics such as the number of simultaneous connections or the internal load of the database.

 

By the way, the CloudWatch and EC2 teams are hiring:

• CloudWatch Jobs
• EC2 Jobs

-- Jeff;

Comments

[scientastic]

Can we use these custom metrics to monitor things like disk space utilization? (I assume the answer is yes, but what I really want to know is, has someone done this and can they set me up quickly with how to do it?)

Thanks!

[Haim Shuvali]

Please update the HP SiteScope reference link to: http://h30499.www3.hp.com/t5/Application-Performance/HP-SiteScope-integration-with-Amazon-CloudWatch-Auto-Scaling-AWS/ba-p/2408860