Recent AWS Customer Success Stories & Videos

More AWS Customer Success Stories...

« Amazon RDS - Multi-AZ Deployments For Enhanced Availability & Reliability | Main | enStratus Adds Support for Amazon SNS and Amazon RDS »

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c534853ef01347fe63205970c

Listed below are links to weblogs that reference New: Amazon S3 Reduced Redundancy Storage (RRS):

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Sylvain Hellegouarch

> If Amazon S3 detects that an object has been lost any subsequent requests for that object will return the HTTP 405 ("Method Not Allowed") status code.

Wouldn't "410 Gone" have been more logical since you know it did exist but not any longer.

Sergey

This is already supported in S3 Backup, BTW. http://s3bk.com/

Dirkdk

sounds cool!

is the API the same as for S3? In other words, is it just a configuration to switch from S3 to RSS?

greets, dirk

-dan

Seems to me like you are setting up a case in which you will 'lose' an item ever year to get people to upgrade.

David Zuelke

I agree with Sylvain. "405 Method Not Allowed" is definitely the wrong status code. Please change it to "410 Gone", that's exactly what it is for. You'll need 405 at some point in the future to indicate something else (let's say when you have a read-only item, then you want to return a 405 error when someone DELETEs or PUTs to it).

Dan

Shame there's no way of getting notification when an object is lost, short of regularly looking at all objects and checking the status. Is this something that's impossible because of the S3 architecture, or could it later be added as a feature?

mamund

I think 405 is not the right response for content missing on the server. 405 tells clients "stop performing that request, this server doesn't support it." Instead I suggest using 422 (http://www.webdav.org/specs/rfc4918.html#STATUS_422) or maybe 424 (http://www.webdav.org/specs/rfc4918.html#STATUS_424) from the WebDAV specs. Both of these response code relate to the condition of the data on the server, not the protocol method used by the client.

John Haugeland

The standard suggests that the correct status is 410 Gone, not 405 Method Not Allowed. Gone is for when a resource is missing, cannot be found elsewhere, and should be considered permanently missing. 405 is for when the resource is available, but the method (post, get, push, etc) is not allowed for this one specific resource.

This will have a significant impact on proxies. Note that you are not returning a list of valid methods in the Allow header as the standard requires (and cannot, because there will be no allowed methods, meaning that 405 is an impossible response to make legal.)

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

Quoting the standard:

---------------------------

10.4.6 405 Method Not Allowed

The method specified in the Request-Line is not allowed for the resource identified by the Request-URI. The response MUST include an Allow header containing a list of valid methods for the requested resource.

---------------------------

10.4.11 410 Gone

The requested resource is no longer available at the server and no forwarding address is known. This condition is expected to be considered permanent. Clients with link editing capabilities SHOULD delete references to the Request-URI after user approval. If the server does not know, or has no facility to determine, whether or not the condition is permanent, the status code 404 (Not Found) SHOULD be used instead. This response is cacheable unless indicated otherwise.

The 410 response is primarily intended to assist the task of web maintenance by notifying the recipient that the resource is intentionally unavailable and that the server owners desire that remote links to that resource be removed. Such an event is common for limited-time, promotional services and for resources belonging to individuals no longer working at the server's site. It is not necessary to mark all permanently unavailable resources as "gone" or to keep the mark for any length of time -- that is left to the discretion of the server owner.

---------------------------

David and Sylvain are correct. Please fix.

Eric

I agree with those wanting 410 gone. The WebDAV errors that mamund suggested are not good because they suggest that the entry still exists or that it can possibly recover. 410 maps exactly to the problem: your file is gone, sorry.

John Haugeland

Dan: disks fail. No amount of redundancy can prevent that more than one disk could fail simultaneously. One loss in ten million every ten million years - eleven nines - is ridiculously stable. If someone advertises 100% to you, they are lying through their teeth.

Amazon is a large industrial company. They cannot tell lies like that.

If you were to move the entirety of WordPress.com - the one that hosts tens of thousands of blogs - to this storage service, it's quite likely that you'd still lose less than one item (posts, comments, settings fields, et cetera) every ten thousand years.

So, y'know, a 20% chance that one thing would have been lost since we turned over Year 0.

That's not setting up for upgrades. That's just facing facts: electronics aren't perfect. Things fail, and even with extreme redundancy, it is possible that all the disks carrying copies of your data will fail before any may be replaced.

If one in ten million since the time of the dinosaurs isn't good enough for you, you might want to have hosting that isn't mass market value oriented.

It's more than good enough for almost anything legitimately imaginable, though. Short of if you're a bank or a nuclear power plant, eleven nines should be just fine.

Besides, statistically, the thing you'll end up losing is going to be some form of spam. That's pretty much the only thing the internet generates anymore, besides pornography.

John Haugeland

The standard suggests that the correct status is 410 Gone, not 405 Method Not Allowed. Gone is for when a resource is missing, cannot be found elsewhere, and should be considered permanently missing. 405 is for when the resource is available, but the method (post, get, push, etc) is not allowed for this one specific resource.

This will have a significant impact on proxies. Note that you are not returning a list of valid methods in the Allow header as the standard requires (and cannot, because there will be no allowed methods, meaning that 405 is an impossible response to make legal.)

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

Quoting the standard:

---------------------------

10.4.6 405 Method Not Allowed

The method specified in the Request-Line is not allowed for the resource identified by the Request-URI. The response MUST include an Allow header containing a list of valid methods for the requested resource.

---------------------------

10.4.11 410 Gone

The requested resource is no longer available at the server and no forwarding address is known. This condition is expected to be considered permanent. Clients with link editing capabilities SHOULD delete references to the Request-URI after user approval. If the server does not know, or has no facility to determine, whether or not the condition is permanent, the status code 404 (Not Found) SHOULD be used instead. This response is cacheable unless indicated otherwise.

The 410 response is primarily intended to assist the task of web maintenance by notifying the recipient that the resource is intentionally unavailable and that the server owners desire that remote links to that resource be removed. Such an event is common for limited-time, promotional services and for resources belonging to individuals no longer working at the server's site. It is not necessary to mark all permanently unavailable resources as "gone" or to keep the mark for any length of time -- that is left to the discretion of the server owner.

---------------------------

David and Sylvain are correct. Please fix.

Kshitij Lauria

Sorry, that's not how probabilities work.

Using your given definition, if the durability is d, then there is a probability of (1-d) that any given object will be lost. For simplicity, and because from your use of the term "object" it seems the concept is atomic, I will take their probabilities of loss to be independent. This is blatantly false if there is exactly one hard disk that can fail, and comes closer and closer to the truth as you guys devise better and better methods. If I store N objects with you, the probability that you lose M of them after one year is given by sampling the binomial distribution: C(N,M) * (1-d)^M * d^(N-M).

If I stay in business with you for more than 1 year because of how awesome your data storage service is, say T years, then I will want to estimate the probability that you lose a total of M objects over the course of those years. I won't go into the details, but you want the product of the binomial probability given above for each year, for each partition of those M objects over those T years. You might want to try your hand at coming up with a (somewhat) closed form solution.

Why bring this up? Because the following two statements:

"Let's define durability (with respect to an object stored in S3) as the probability that the object will remain intact and accessible after a period of one year. 100% durability would mean that there's no possible way for the object to be lost, 90% durability would mean that there's a 1-in-10 chance, and so forth."

AND

"durability [...] is 99.999999999%. If you store 10,000 objects with us, on average we may lose one of them every 10 million years or so."

are incompatible, unless you have strong reason to believe that your failures occur in an idiosyncratic way such that multiple failures over time and object are interconnected in a very specific way that I, for the moment, wouldn't want to work out mathematically.

It is not inconceivable that you guys might end up putting these figures in some legally binding contract. Don't get burned by math.

pwb

Are you going to *ever* rollout HTTPS support on CloudFront? What the heck is taking so long for this obviously critical capability.

Matt

99.999999999% is purely through technical failure, right? I reckon the odds of Amazon going broke (or similar improbable happenings) would be more likely than that...

Reduced redundancy at a cheaper rate is a nice addition.

Sewook Wee

I was looking for "eleven 9s durability" and this post serves great! Still I want one more detail. If the durability is defined per object, shouldn't it vary by the size of an object? For example, 1kB object would be lost in the same probability as ~2GB object? If AWS's data loss granularity is per disk, and any given object should not be distributed across multiple disks, it makes sense that each object has same probability, though. Otherwise, will it be possible to know the durability as a function of object size? Thanks.

Andrew S

Pwb: Take a look at http://developer.amazonwebservices.com/connect/thread.jspa?threadID=45378&start=15&tstart=405

Looks like Cloudfront HTTPS support is coming in the next few weeks.

The comments to this entry are closed.

Featured Events

The AWS Report


Brought to You By

Jeff Barr (@jeffbarr):



Jinesh Varia (@jinman):


Email Subscription

Enter your email address:

Delivered by FeedBurner

April 2014

Sun Mon Tue Wed Thu Fri Sat
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30