New Feature: Amazon S3 now supports Object Versioning

We've added beta support for Versioning across all Amazon S3 Regions.

Versioning provides an additional layer of protection for your S3 objects. You can easily recover from unintended user errors or application failures. You can also use Versioning for data retention and archiving. Once you have enabled Versioning for a particular S3 bucket, any operation that would have overwritten an S3 object (PUT, POST, COPY, and DELETE) retains the old version of the object. Here's a simple diagram of Versioning in action:

Each version of the object is assigned a version id. For example, each version of Robot.png has its own version id:

The actual version ids are long strings; I've used v1, v2, and v3 to simplify the picture. You can retrieve the most recent version of an object by making a default GET request or you can retrieve any version (current or former) by making a version-aware request and including a version id. In effect, the complete key for an S3 object in a versioned bucket now consists of the bucket name, the object name, and the version id.

S3's DELETE operation works in a new way when applied to a versioned object. Once an object has been deleted, subsequent default requests will no longer retrieve it. However, the previous version of the object will be preserved and can be retrieved by using the version id. Only the owner of an S3 bucket can permanently delete a version.

Normal S3 pricing applies to each version of an object. You can store any number of versions of the same object, so you may want to implement some expiration and deletion logic if you plan to make use of this feature.

Enabling Versioning's MFA Delete setting on your bucket provides even more protection. Once enabled, you will need to supply two forms of authentication in order to permanently delete a version from your bucket: your AWS account credentials and the six-digit code and serial number from an MFA (Multi-Factor Authentication) device in your possession.

You can read more about Versioning in the S3 documentation.

-- Jeff;

Comments

[Saurabh]

Jeff,

Greetings - We have released a new version of Bucket Explorer with versioning support.

http://support.bucketexplorer.com/topic934.html

ThankS
Saurabh

[Ed]

Thats actually a very interesting feature.
It would be nice to know about your roadmap. For example: Will you also implement expiration and deletion logic instead of having each user implement own logic for this common task?

Also: What about de-duplication and efficient storage?
It's probably good for Amazon if the storage is "inefficient" because our bills get bigger and bigger, but ultimately it would be good to store the data very efficiently and de-dupe it.
Another VERY interesting feature would be "traffic efficiency". You could implement a feature in the API that would allow a user to provide the checksum of a file and then check all S3 buckets for this checksum. If it is available, just copy a "reference" into the S3 bucket of the user instead requiring the user to upload the file.

If the file gets deleted from the "other" bucket, the reference of course has to be replaced with the original file. This way you could store very very efficiently and your users would also benefit dramatically (less storage needed and especially less traffic).

Of course this has to be done in a very secure way in order to prevent users from "guessing" checksums and thus accessing private files, but I am sure that you guys will find a solution to implement this in a stellar and secure way ;-)

[James Stanley]

How does the new S3 versioning affect CloudFront? I presume CloudFront will always get the latest version; does this affect the rate at which CloudFront will detect a change in a source file in S3 and update the edge cache?

[Eribium]

Are the versions stored as binary diffs, or full objects?
If it's the latter, you could end up storing a lot more data than you need too.

[Ron Wolf]

Ed says, "It would be nice to know about your roadmap. For example: Will you also implement expiration and deletion logic...."

Well, actually, for those of us interested in creating AWS add-ons (as service products or as corporate users), its ESSENTIAL to know the road map for this.

In certain situations, versioning can a very useful addition to the AWS feature set. And without management, it opens a nice opportunity for add on developers to provide additional management capabilities. But, of course, we won't go to the trouble if its Amazon's intentions to add these features. And not knowing makes it impossible for us to plan.

All along these sort of product roadmap mysteries have made it hard for the 3rd party market (Amazon's Camp as Regis McKenna would say) to plan and create effective businesses.

Is this just poor product management? Or Amazon wanting to keep all of their cards close? Or am I off base???

[Ed]

@Eribium:
Exactly my thoughts in my earlier post.
Any clarification from Amazon would be really welcome here.
The fact that you asked the same question seems to prove that it is of importance :-)

@Ron:
Exactly!
But even when not working in the "ad-on business" it's important.
I'd hate to allocate our developers on the "build a deletion logic" task, when they could instead work on some "real features" for OUR users - instead of wasting time on some "background logic" thats "superseded" by Amazons own (and hopefully better) in a few weeks/months/whatever period.

Unfortunately it looks like this "blog" is not a blog, but rather a one-way street for press releases :-(

[Dilip Krishnan]

I'm curious, In the API design, whats the reason behind the decision to pass the version id in the uri?

Instead of

GET /myObject?versionId=3/L4kqtJlcpXroDTDmpUMLUo HTTP/1.1
Host: bucket.s3.amazonaws.com
Date: Wed, 28 Oct 2009 22:32:00 GMT
Authorization: AWS 02236Q3V0WHVSRW0EXG2:0RQf4/cRonhpaBX5sCYVf1bNRuU=

Why not just do something like so?

GET /myObject HTTP/1.1
Host: bucket.s3.amazonaws.com
Date: Wed, 28 Oct 2009 22:32:00 GMT
Authorization: AWS 02236Q3V0WHVSRW0EXG2:0RQf4/cRonhpaBX5sCYVf1bNRuU=
x-amz-version-id: 3/L4kqtJlcpXroDTDmJ+rmSpXd3QBpUMLUo
....

[Patsi Blackford-Bailey]

Thanks all you people at Amazon for a brilliant quick service , i am new to Amazon , I will be using you again soon , best wishes, Patsi Blackford-Bailey x