EC2 Instance Status Monitoring

We have been hard at work on a set of features to help you (and the AWS management tools that you use) have better visibility into the status of your AWS resources. We will be releasing this functionality in stages, starting today. Today's release gives you visibility into scheduled operational activities that might affect your EC2 Instances. A scheduled operational activity is an action that we must take on your instance.

Today, there are three types of activities that we might need to undertake on your instance: System Reboot, Instance Reboot, or Retirement. We do all of these things today, but have only been able to tell you about them via email notifications. Making the events available to you through our APIs and the AWS Management Console will allow you to review and respond to them either manually or programmatically.

System Reboot - We will schedule a System Reboot when we need to do maintenance on the hardware or software supporting your instance. While we are able to perform many upgrades and repairs to our fleet without scheduling reboots, certain operations can only be done by rebooting the hardware supporting an instance. We won’t schedule System Reboots very often, and only when absolutely necessary. When an instance is scheduled for a System Reboot, you will be given a time window during which the reboot will occur. During that time window, you should expect your instance to be cleanly rebooted.  This generally takes about 2-10 minutes but depends on the configuration of your instance. If your instance is scheduled for a System Reboot, you can consider replacing it before the scheduled time to reduce impact to your software or you may wish to check on it after it has been rebooted. 

Instance Reboot - An Instance Reboot is similar to a System Reboot except that a reboot of your instance is required rather than the underlying system. Because of this, you have the option of performing the reboot yourself.  You may choose to perform the reboot yourself to have more control and better integrate the reboot into your operational practices. When an instance is scheduled for an Instance Reboot, you can choose to issue an EC2 reboot (via the AWS Management Console, the EC2 APIs or using other management tools) before the scheduled time and the Instance Reboot will be completed at that time. If you do not reboot the instance via an EC2 reboot, your instance will be automatically rebooted during the scheduled time.

Retirement - We will schedule an instance for retirement when it can no longer be maintained. This can happen when the underlying hardware has failed in a way that we cannot repair without damaging the instance. An instance that is scheduled for retirement will be terminated on or after the scheduled time. If you no longer need the instance, you can terminate it before its retirement date. If it is an EBS-backed instance, you can stop and restart the instance and it will be migrated to different hardware. You should stop/restart or replace any instance that is scheduled for retirement before the scheduled retirement date to avoid interruption to your application.

API Calls
The new DescribeInstanceStatus function returns information about the scheduled events for some or all of your instances in a particular Region or Availability Zone. The following information is returned for each instance:

• Instance State - The intended state of the instance (pending, running, stopped, or terminated).
• Rebooting Status – An indication of whether or not the instance has been scheduled for reboot, including the scheduled date and time (if applicable).
• Retiring Status - An indication of whether or not the instance has been scheduled for retirement, including the scheduled date and time (if applicable).

We'll continue to send "degraded instance" notices to our customers via email to notify you of retiring instances.

Console Support
You can view upcoming events that are scheduled for your instances in the EC2 tab of the AWS Management Console. The EC2 Console Dashboard contains summary information on scheduled events:

You can also click on the Scheduled Events link to view detailed information:

The instance list displays a new "event" icon next to any instance that has a scheduled event:

I expect existing monitoring and management tools to make use of these new APIs in the near future. If you've done this integration, drop me a note and I'll update this post.

-- Jeff;

 

Comments

[Velaskec]

Hi
Great job!
Can we have this kind of notification before spot instance terminated?

[Jonathan Krengel]

Concerning this statement:
"If it is an EBS-backed instance, you can start and restart the instance and it will be migrated to different hardware."

Does this only apply when the underlying hardware is marked for retirement?

[Hrish]

This is available as part of the API

http://docs.amazonwebservices.com/AWSEC2/2011-11-01/APIReference/index.html?ApiReference-query-DescribeInstanceStatus.html

but I could not find an equivalent call in the AWS Java SDK. Is the SDK updated yet?

[Niall]

Do AWS have plans to send a simple email alert when a customers instance is earmarked for any of the reboot events you outlined above?

The alternative at the moment would be to pay for some additional 3rd part monitoring which isn't ideal.

Also, if an EC2 instance is scheduled for a system or retirement reboot then will a reboot invoked by a customer cause the instance to boot up on different hardware? And then will the schedulef reboot be disgarded?

But this is a good improvement anyways.
Thanks

[Matt]

To answer part of Niall's post, I can confirm that the scheduled shutdown by AWS is indeed canceled after a Stop and Start of your instance. Just don't forget to re-associate your Elastic IP on startup.

[imet]

but the latest JAVA API can't get the function

[Bernd Eckenfels]

What I would like to see from the API is a "soft-stop" which will schedule a stop/terminate on the next tick, so I can set a flag that my instance is no longer needed but it can run till the end of the hour. I can understand Amazon does not want this in order to increase half-used hours, however you can also get some positive aspects if a instance is marked for shutdown on the next hour, you can calculate with it becoming available. The alternative is a management portal polling the instances and shutting them down on the same clock tick. Just with additional load, more hazzles and less visibility for AWS.