Amazon Glacier: Archival Storage for One Penny Per GB Per Month

You Need Glacier
I’m going to bet that you (or your organization) spend a lot of time and a lot of money archiving mission-critical data. No matter whether you’re currently using disk, optical media or tape-based storage, it’s probably a more complicated and expensive process than you’d like which has you spending time maintaining hardware, planning capacity, negotiating with vendors and managing facilities.

True?

If so, then you are going to find our newest service, Amazon Glacier, very interesting. With Glacier, you can store any amount of data with high durability at a cost that will allow you to get rid of your tape libraries and robots and all the operational complexity and overhead that have been part and parcel of data archiving for decades.

Glacier provides – at a cost as low as $0.01 (one US penny, one one-hundredth of a dollar) per Gigabyte, per month – extremely low cost archive storage. You can store a little bit, or you can store a lot (Terabytes, Petabytes, and beyond). There's no upfront fee and you pay only for the storage that you use. You don't have to worry about capacity planning and you will never run out of storage space. Glacier removes the problems associated with under or over-provisioning archival storage, maintaining geographically distinct facilities and verifying hardware or data integrity, irrespective of the length of your retention periods. 

Tell me More
We introduced Amazon S3 in March of 2006. S3 growth over the past 6+ years has been strong and steady, and it now stores over one trillion objects. Glacier builds on S3's reputation for durability and dependability with a new access model that was designed to be able to allow us to offer archival storage to you at an extremely low cost.

To store data in Glacier, you start by creating a named vault. You can have up to 1000 vaults per region in your AWS account. Once you have created the vault, you simply upload your data (an archive in Glacier terminology). Each archive can contain up to 40 Terabytes of data and you can use multipart uploading or AWS Import/Export to optimize the upload process. Glacier will encrypt your data using AES-256 and will store it durably in an immutable form. Glacier will acknowledge your storage request as soon as your data has been stored in multiple facilities.

Creating a vault in Amazon Glacier.

Glacier will store your data with high durability (the service is designed to provide average annual durability of 99.999999999% per archive). Behind the scenes, Glacier performs systematic data integrity checks and heals itself as necessary with no intervention on your part. There's plenty of redundancy and Glacier can sustain the concurrent loss of data in two facilities.

At this point you may be thinking that this sounds just like Amazon S3, but Amazon Glacier differs from S3 in two crucial ways.

First, S3 is optimized for rapid retrieval (generally tens to hundreds of milliseconds per request). Glacier is not (we didn't call it Glacier for nothing). With Glacier, your retrieval requests are queued up and honored at a somewhat leisurely pace. Your archive will be available for downloading in 3 to 5 hours.

Each retrieval request that you make to Glacier is a called a job. You can poll Glacier to see if your data is available, or you can ask it to send a notification to the Amazon SNS topic of your choice when the data is available. You can then access the data via HTTP GET requests, including byte range requests. The data will remain available to you for 24 hours.

Retrieval requests are priced differently, too.  You can retrieve up to 5% of your average monthly storage, pro-rated daily, for free each month. Beyond that, you are charged a retrieval fee starting at $0.01 per Gigabyte (see the pricing page for details).  So for data that you’ll need to retrieve in greater volume more frequently, S3 may be a more cost-effective service. 

Notifications for retrieval jobs.

Secondly, S3 allows you to assign the name of your choice to each object. In order to keep costs as low as possible, Glacier will assign a unique id to each of your archives at upload time.

Glacier In Action
I'm sure that you already have some uses in mind for Glacier. If not, here are some to get you started:

• If you are part of an enterprise IT department, you can store email, corporate file shares, legal records, and business documents. The kind of stuff that you need to keep around for years or decades with little or no reason to access it.
• If you work in digital media, you can archive your books, movies, images, music, news footage, and so forth. These assets can easily grow to tens of Petabytes and are generally accessed very infrequently.
• If you generate and collect scientific or research data, you can store it in Glacier just in case you need to get it back later.

Get Started Now
Glacier is available for use today in the US-East (N. Virginia), US-West (N. California), US-West (Oregon), Asia Pacific (Tokyo) and EU-West (Ireland) Regions.

Watch our new video to see how to get started:

You can access Glacier from the AWS Management Console or through the Glacier APIs. We have added Glacier support to the AWS SDKs and there's also plenty of Glacier documentation.

If you'd like to know even more about Glacier, please join us for an online seminar on September 19th.

And there you have it. What do you think?

 -- Jeff;

PS. If you are an engineer or engineering manager with an interest in massive scale distributed storage systems we’d love to hear from you. Please send your resume to glacier-jobs@amazon.com.

 

Comments

[Tim Chiu]

very cool! literally! :)

[Tim Chiu]

Is there an option to move archives from glacier to standard S3 via an API call? And will Import/Export support burning from glacier directly?

[Simon Willison]

I'd really like to be able to pre-pay for Glacier. One of my fears when using AWS is that one day my credit card might bounce, I'll be off on a one-month long holiday and by the time I return all of my personal data stored on S3 will be deleted. I'd love to be able to pay some cash up front in return for knowing that my data stored on Glacier is safe for the next 50 years.

[Jshantz4]

Unbelievable. ~$10/month to store a terabyte vs. $128/month with S3. This is a game changer. Pretty sure I need to quit my PhD and go work at Amazon because you guys are innovating at lightning speed.

[Dirk Krause]

As far as I can see, there is no Python support yet for Glacier ... is this planned? in Boto, maybe?

[Abodera]

Great job Jeff!

Brilliant. With any decent internet connectivity and client-side encryption, this means an end to ye olde' tapes, external HDDs, BR, backup arrays and other hardware.

It's also a very important news for individuals, because it enables a new way of thinking about storage and gigabytes (disk space) in general. It's a perfect companion to recent cloud-computing mindset and plethora of cloud-oriented devices (like ultrabooks with small, fast SSD storage, netbooks with little or no storage, smartphones). In many scenarios (for professionals and house holds alike) it potentially retires a lot of hardware solutions.

I believe Glacier is a disruptive innovation. Because of slow access times it would not be fit for some use cases (like media content storage for creative audio/video editing, which sometimes necessitates random access to clips). But for everything else, and everything with "backup" in its name, it's a true game changer!

[David Mytton]

What are the plans for supporting this in the AWS Storage Gateway product?

[Djmatus23.wordpress.com]

Do you guys have any third party tools available from partners, e.g. cloudberry, yet?

[Peter]

This sounds really nice. One concern is lack of either SLA or back-end details. Will my data still be there in 10 years? Or is it going on a decaying tape deck somewhere? Is there ever verification/replication?

[Ali]

Great addition to AWS. We just ran a quick cost forecast in http://PlanForCloud.com and it's interesting:

If you start with 100GB then add 10GB/month, it would cost $102.60 after 3 years on AWS Glacier vs $1,282.50 on AWS S3!

[Andrew Chilton]

Hi Jeff and AWS,

Support for Glacier has been added to AwsSum, the Node.js library which talks to 26 Amazon Web Services.

* https://github.com/appsattic/node-awssum

It has also made my code for the AWS Signature v4 better too! :)

Cheers,
Andy

[Pete Chown]

This looks interesting, but I don't feel I understand the pricing for excess retrieval requests. First of all I don't understand how the 'peak hour' figure is calculated. Second, is the charge imposed when I ask for data to be brought back from secondary storage, or when I actually download it?

(IMHO, complicated pricing is a problem with a lot of the AWS product range. A good example is the charge per I/O request for EBS volumes. It's hard to know how I/O requests will be batched up by the operating system, so it's hard to predict what you'll actually be asked to pay.)

[Javier]

"Data tranfer out" is a bit pricey. In many scenarios it will represent the real cost of the service, as retrieving 100GB would be as expensive as mantaining them for ten months.

[Entegral]

Really awesome - AWS never fails to surprise with brilliant new products.
I think this could spark a few new products / startups at a cost of about x10 cheaper than S3.

[Juan Fernandez]

I wonder if this service can act as a replacement for the typical enterprise data backups. It seems to be a cost-effective solution for keeping that kind of data, but my concers arise on the worst case.

What should the restore / recovery procedure of that big data be? I mean, it's not a problem uploading 12TB of data, as we have plenty of time to do it while our systems are working. But then, in case of an emergency, as if one of our data servers breaks with some 4TB of data in it, and we have to restore the system, we need a fast, reliable and not expensive procedure of recovering the previous state.

Is there any offline procedure in place?

Thanks!

[sysadmin]

Is this service HIPPA/SOX/etc compliant?

[John]

There is no archiving system in the world that requires you to use java in order to upload or download an archive. You guys have such great ideas, but you mess it up by not providing user friendly tools to operate it. I guess I have to wait until someone builds a client app because there is not way I'm going to mess around with java just to upload an archive.
The days of black screens with green lettering is over, my friends.;-)

[Eric Mesa]

Sounds pretty awesome. Is there an ability to mount my Glacier onto a Unix computer for transparent backups?

[Wouter]

Wait what? A data-storage service for gigs and gigs of data and offering only HTTP GET-requests? Whats up with that??? I would love to use your service if you offer FTP (maybe sftp or something?) as a way to upload and download. Also mounting the volume on unix/windows/mac would come in handy. Additional question: is there an API available?

[Sergey]

If anyone wants to try uploading to Glacier over FTP, I'll be deploying that support soon. See http://blog.cloudgates.net/
-Sergey

[Dan]

Eric Mesa - I dont want to troll, but did you even read/watch the video/article? or did you just try to "be first"

[S3Browser]

We are working on Windows Client for Amazon Glacier.

FastGlacier: http://fastglacier.com

The first public version will be released in the near future.

[Rayson (Open Grid Scheduler)]

Boto will be getting Glacier support: https://groups.google.com/forum/?fromgroups=#!topic/boto-users/0yAXxh4qNiA

Of course, the maintainer of Boto is also busy with many other things.

[Michael]

I like the service and pricing, but as a minor point of feedback, I question the name of the product. "Glacial" can mean very very cold, but it can also mean very very slow, which isn't really a connotation you want associated with a web service.

[Brian McMichael]

Hi everyone. I made an uploader for Glacier last night. It's one-way right now, but I can add more features later.
http://simpleglacieruploader.brianmcmichael.com/

[S3Browser]

The first public version of FastGlacier is just released: http://fastglacier.com

[Chris Gent]

Are there any plans for an EBS volume archive to Glacier api function? I like the idea of incremental daily snapshots to s3 for fast data recovery and then full weekly/monthly/yearly backups to Glacier.

[twitter.com/cloudberryman]

CloudBerry Lab has just released the version of CloudBerry Backup the Windows application that automates data backup to cloud storage, with Amazon Glacier support. Download your copy at http://www.cloudberrylab.com/backup/

[twitter.com/cloudberryman]

CloudBerry Lab has also released the newer version of CloudBerry S3 Explorer freeware, the tool that helps move files to the Glacier with FTP like interface. http://www.cloudberrylab.com/free
In addition CloudBerry Cloud Migrator allows migration data to Glacier from FTP sites, Amazon S3 and other cloud providers https://sync.cloudberrylab.com/ The first 6 GB of data transfer per month is free

[Ashok Kumar]

Can you explain, how can we download data from Glacier.

[Jeff Barr]

Hi Ashok, you will need to use the Glacier APIs or a Glacier GUI tool such as FastGlacier.

[Thomas]

Try WinGlacier for uploading and downloading your files to/from Amazon Glacier. Few features...
•archive metadata in your own db on the cloud, so your metadata doesn't get lost when you switch machines
•Zip Compression, password encryption
•Fail-safe automatic resumption of file transfers between application shutdowns
•Multiple concurrent file transfers
Check us out at www.winglacier.com

Thank you