Amazon S3 - Object Expiration

Amazon S3 is a great way to store files for the short or for the long term.

If you use S3 to store log files or other files that have a limited lifetime, you probably had to build some sort of mechanism in-house to track object ages and to initiate a bulk deletion process from time to time. Although our new Multi-Object deletion function will help you to make this process faster and easier, we want to go ever farther.

S3's new Object Expiration function allows you to define rules to schedule the removal of your objects after a pre-defined time period. The rules are specified in the Lifecycle Configuration policy that you apply to a bucket. You can update this policy through the S3 API or from the AWS Management Console.

Each rule has the following attributes:

• Prefix - Initial part of the key name, (e.g. “logs/”), or the entire key name. Any object in the bucket with a matching prefix will be subject to this expiration rule. An empty prefix will match all objects in the bucket.
• Status - Either Enabled or Disabled. You can choose to enable rules from time to time to perform deletion or garbage collection on your buckets, and leave the rules disabled at other times.
• Expiration - Specifies an expiration period for the objects that are subject to the rule, as a number of days from the object's creation date.
• Id - Optional, gives a name to the rule.

You can define up to 100 expiration rules for each of your Amazon S3 buckets; however, the rules must specify distinct prefixes to avoid ambiguity. After an Object Expiration rule is added, the rule is applied to objects that already exist in the bucket as well as any new objects added to the bucket after the rule is created. We calculate the expiration date for an object by adding that object's creation time to the expiration period and rounding off the resulting time to midnight of that day. If you make a GET or a HEAD request on an object that has been scheduled for expiration, the response will include an x-amz-expiration header that includes this expiration date and the corresponding rule Id.

We evaluate the expiration rules once each day. During this time, based on their expiration dates, any object found to be expired will be queued for removal. You will not be billed for any associated storage for those objects on or after their expiration date. If server access logging has been enabled for that S3 bucket, an S3.EXPIRE.OBJECT record will be generated when an object is removed.

You can use the Object Expiration feature on buckets that are stored using Standard or Reduced Redundancy Storage. You cannot, however, use it in conjunction with S3 Versioning (this is, as they say, for your own protection). You will have to delete all expiration rules for the bucket before enabling versioning on that bucket.

Using Object Expiration rules to schedule periodic removal of objects can help you avoid having to implement processes to perform repetitive delete operations. We recommend that you use Object Expiration for performing recurring deletions that can be scheduled, and use Multi-Object Delete for efficient one-time deletions.

You can use this feature to expire objects that you create, or objects that AWS has created on your behalf, including S3 logs, CloudFront logs, and data created by AWS Import/Export.

For more information on the use of Object Expiration, please see the Object Expiration topic in the Amazon S3 Developer Guide.

-- Jeff;

Comments

[twitter.com/cloudberryman]

The new version of CloudBerry Explorer freeware for Amazon S3 is available. The new version comes with the full support for Object Expiration .
More info on this feature on our blog at http://blog.cloudberrylab.com/2011/12/how-to-setup-object-expiration-with.html

[Spinningmatt.wordpress.com]

http://spinningmatt.wordpress.com/2011/12/28/amazon-s3-object-expiration-what-about-instance-expiration/

It would be great to have an EC2 instance expiration feature. One that could be (0) assigned per instance and (1) adjusted while the instance exists. Bonus if can also be (2) adjusted from within the instance without credentials. Think leases.

[Dchancogne]

If object expiration can be applied to any S3 object and EBS snapshots are stored on S3, can I set up an expiration date for my EBS snapshots? If yes, how?

[Jeff Barr]

@Dchancogne - You can't expire EBS snapshots using this method, but it is a good idea and one that I will pass along.

[S3 Browser Team]

New version of S3 Browser Freeware also comes with full support for Object Expiration.

[Ronak]

Nice Blog..Looks really good.
New version of Bucket Explorer is coming soon with full support of Object Expiration.
You can start from here:
http://www.bucketexplorer.com/be-download.html

[Denis]

In AWS SDK for Java 1.3.4 setExpirationTime method is added in ObjectMetadata class. Does it mean I can set object expiration time by putting an object in S3 using the API?

[Socialkik]

he new version of CloudBerry Explorer freeware for Amazon S3 is available. The new version comes with the full support for Object Expiration .so this is really needed this weblog super amazing website