Recent AWS Customer Success Stories & Videos

More AWS Customer Success Stories...

« Amazon S3 - More Than 449 Billion Objects | Main | AWS Summer Startups: Mediology »

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c534853ef014e89fef166970d

Listed below are links to weblogs that reference Additional CloudWatch Metrics for Amazon SQS and Amazon SNS:

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Susan

Jeff,

this all sounds great in theory but doesn't work that well in practice.

Two problems:

1. SQS basically guarantees out of order message arrival. That may or may not be important but for certain use cases, that IS important. And then you're stuck mitigating out of order arrival in your application. It'd be nice if there were a parameter that can be set to delay delivery of a message until the one prior arrives. I understand it's a trade-off between reliability and delivery speed but not having this configurable sucks.

2. AutoScale has no knowledge of which servers to kill. As a result, it is suited for the most basic applications only, like stateless HTTPD. Anything more complex and it fails entirely. For example, it's not suited for media transcoding because a server that's actually doing work might get killed off, while an idle instance is left alone. Not cool. So, again, one is stuck doing this logic in a custom app some place.

AkashAgrawal

I missed this service for SQS badly, that's why we wrote our own solution to auto-scale based on SQS length. Please read here: http://tech-queries.blogspot.com/2011/04/suicide-workers.html

"So when our SQS length is more than a threshold, we know that our current conversion fleet is not enough to handle increased load."

Susan

Akash,

your solution is the first that comes to mind but it's crude and not very elegant. (I can say this because we also went and abandoned this idea.:) )

The issue is that looking at absolute metric such as SQS queue depth tells you nothing about the TREND. And it's the trend you're after.

Consider this example. Let's say your threshold is 5. And queue depth metrics go 1, 3, 5, 200, 199, 198, 197, etc...

What happens is at 200 you spawn a server. You spawn another one at 199. By the time you get to 190, you have enough servers to ensure a steady downward trend on the queue depth. BUT! Your solution will keep launching servers, to the point where you end up with 100s of EC2 machines but your queue depth is 6.

A much better way is to look at moving avg to smooth out the spikes. Even better still is to look at rate of change of moving averages. You don't need to launch machines on a consistent downward trend.

AkashAgrawal

@Susan: We are actually seeing if the SQS persist above this threshold for 30 mins then only it will spawn a new instance. Also as soon as a instance is launched, SQS length should drop significantly.

@Jeff: I couldn't find the pricing information about SQS monitoring on any product page. Can you please let me know the pricing info?

The comments to this entry are closed.

Featured Events

The AWS Report


Brought to You By

Jeff Barr (@jeffbarr):



Jinesh Varia (@jinman):


Email Subscription

Enter your email address:

Delivered by FeedBurner

April 2014

Sun Mon Tue Wed Thu Fri Sat
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30