Amazon Web Services Blog

Amazon S3 Copy API Ready for Testing

A few weeks ago we asked our developer community for feedback on a proposed Copy feature for Amazon S3. The feedback was both voluminous and helpful to us as we finalized our plans and designed our implementation.

This feature is now available for beta use; you can find full documentation here (be sure to follow the links to the detailed information on the use of this feature via SOAP and REST). Copy requests are billed at the same rate as PUT requests: $.01 for 1000 in the US, and $.012 for 1000 in Europe.

In addition to the obvious use for this feature -- creating a new S3 object from an existing one -- you can also use it to rename an object within a bucket or to move an object to a new bucket. You can also update the metadata for an object by copying it to itself while supplying new metadata.

Still on the drawing board is support for copying between US and Europe, and a possible conditional copy feature. Both of these items surfaced as a result of developer feedback.

Tool and library support for this new feature is already starting to appear; read more about that in this discussion board thread.

-- Jeff;

Your Input Please: Amazon S3 Copy

There are any number of references on the Web to Amazon’s focus on being the “earth’s most customer centric company”; that passion and determination comes straight from Jeff Bezos.

It's in this traditon that the product team posted a design spec for enhancements to Amazon S3. As Alyssa Henry says in the forum post, we've received a significant amount of customer feedback asking that we add support to Amazon S3 for copying an object, as well as related operations such as move and rename. The post points to a design document and asks for community input. (The document is not final--don't design code around it!)

And the input is coming in! I encourage each of you to read the spec and send us your thoughts on whether the team's work meets your needs. If not, please take the time to provide suggestions on changes you'd like to see in the spec. We're serious about making certain our Web Services are the best on earth for you, the customer.

Mike

Zmanda Webinar

At 10 AM on Wednesday, February 13th, Dmitri Joukovski from Zmanda will explain how backup to Amazon S3 complements traditional backup to disk and tape. He will also demonstrate how easy it is to configure Amanda Enterprise for backup and recovery to Amazon S3.

Dmitri will talk about the benefits of using S3 for off-site backup and archiving. He will also list the advantages of using Amanda Enterprise with S3 vs online backup services. The webinar will show how easy it is to configure and backup to S3 using Amanda Enterprise.

The webinar is free but pre-registration is a must.

-- Jeff;

Increasing Amazon S3 Data Transfer Performance

The Amazon S3 team is now beta-testing support for an important low-level networking feature which has the potential to significantly increase the performance of large data transfers to and from S3, particularly (but not limited to) for long distance data transfers.

In particular, there is a new beta endpoint which supports the RFC 1323 model for TCP window scaling. With this option in effect, a larger amount of data can be in transit across the network at any given time, reducing the impact of speed-of-light delays as data is broken down in to chunks, sent across the internet, verified, and acknowledged. When the two nodes in question are far apart, the time that it takes for the data to travel from sender to receiver (and for the acknowledgments to travel back) turns out to be just as important as the raw data transfer rate.

Per the thread in the Amazon S3 Forum, early results from the beta testers are quite good with reported speedups of 4x to 18x! If you are moving large amounts of data into or out of S3 then you will definitely want to implement this feature.

Note that we are supporting this through a beta endpoint. The endpoint will be present for the duration of the test and will then vanish. Of course you won't want to embed this endpoint in any shrink-wrapped software.

You will need to dig deep into your operating system's networking setting in order to take advantage of this new feature. For Linux systems the information here will get you started (there's also some information for other operating systems in there too). Windows 2000 and Windows 2003 users can start at this Knowledge Base article. I know nothing about the Mac except that my youngest daughter keeps asking me for one.

I would also recommend that you read this very informative article by Brian Tierney for further background information and some formulas and recommendations on how to calculate and set the optimal window size.

-- Jeff;

PS - Did you know that Amazon.com sells actual windows?

Amazon S3 POST Support

The Amazon S3 team is looking for feedback on their proposal to support browser-based file uploads. This feature will allow developers to build web-based applications which support upload of local files to Amazon S3 without the need for an intermediate server.

If you are interested in helping to shape the definition this feature, you should definitely read the proposal and then visit this thread to let them know what you think.

-- Jeff;

Notes From Berlin

Another post from the road… In this case the “road” is a three-week trip to Ireland, Germany, Portugal, and Canada before returning to Seattle. This week I’m in Berlin for Web 2.0 Expo and Interop; both are under one roof.

Today’s announcement that we’re launching Amazon S3 in Europe has me really excited, given where I’m posting this entry from. You can guess what the #1 request from developers has been on this trip: “When will Amazon S3 host data in the EU?”. Developers in the United States were also asking for S3 in Europe, because they were looking for lower latency. Of course the requests for hosting in the EU are as old as the service itself—and checking off a top developer request makes the Amazon S3 product team as happy as the customers who asked.

To support Europe we enhanced Amazon S3’s API to include a new parameter for locality. The name of the parameter is LocationConstraint, and if you use it then buckets are created in the location you specify. Existing applications will not be affected, and will create buckets in the United States, because that’s the “default” behavior. Pricing is slightly different in Europe; although not but much. Use the AWS Price calculator if you want to estimate what your costs will be.

ISVs Already Support Amazon S3 in Europe
Existing ISVs who built products on top of Amazon S3 have been effusive. For example, Saurabh Dani from Bucket Explorer said "We built Bucket Explorer as a simple and easy to use, yet powerful Interface for S3. The majority of our customers represent three segments: people looking to use Amazon S3 as an online collaboration tool, as a reliable store for off-site backups, or as a content delivery network. And our EU customers from all three segments expressed the need for an EU node of S3 to keep their data confined to the EU. With the integration of Bucket Explorer with S3/EU, we are very excited to be able to address to our customer needs." The latest version of Bucket Explorer can be downloaded at www.bucketexplorer.com/be-download.html.

Bucket Explorer and a popular Firefox add-in known as S3 Organizer (available at www.rjonna.com) have enhanced their products to enable users to specify whether buckets are created in North America or Europe. Of course you’ll need to download the latest version in order to take advantage of this new feature.

Real-World S3 Examples
The launch of S3 in Europe is one entry in a long list of milestones. To me, real-world implementations are always the most interesting. One really interesting example that demonstrates the power of scale on demand is in a recent blog post by software developers who work for the New York Times.

The post is about their project to convert old newspaper articles to PDF format for access online (check this out by searching for ”Titanic iceberg” on the front page of the Times, and choose the 1851-1980 archive in the drop-down box). From the blog: “[I] determined that if I used only four machines, it could take some time to generate all 11 million article PDFs. But thanks to the swell people at Amazon, I got access to a few more machines and churned through all 11 million articles in just under 24 hours using 100 EC2 instances, and generated another 1.5TB of data to store in S3. (In fact, it worked so well that we ran it twice, since after we were done we noticed an error in the PDFs.)”

Wow…

Mike

Cool Real Estate Application Using Amazon S3

The other day I blogged that we're looking for applications that use Amazon Web Services, and that were developed on the other side of the Atlantic... Had some interesting replies; and want to tell you about one of them.

Metropix is based in England, and is the leading supplier of floor plans to the UK real estate industry. They've built a server-based engine that converts 2D floor plans into 3D views of homes and other buildings. Of course, the interior as rendered makes a few assumptions regarding decorating and furnishing, but the point of doing this is to help people understand the space that might be their new home, and offer them a glimpse without having to visit the property.

Max Christian is technical director for MetroPix, and emailed me the following validation of the hard work our product team put into building the Amazon S3 web service platform.

"We're in production and have around 17,000 users in any one week. As well as basic floor plans, we use S3 to host 3D floor plans and are starting to use it to store Google Earth KMZ files of 3D models of houses for sale. "

"The company started in 2004 and as the customer base grew we responded by adding additional web servers. Although using additional servers to cope with demand was reasonably straightforward, the storage needed for the thousands of 3D Floor Plans and 3D Walkthroughs hosted on each server was also growing rapidly. It became increasingly impractical to keep expanding the storage capacity of the web servers, which was consuming valuable technical resources that could be better spent on product development. In early 2007 we switched to Amazon's S3 service for storage, shuttling images and video backwards and forwards between S3 and our servers so that only recently-demanded content need be kept locally. This proved so successful, and so reliable, that in March 2007 we went one step further and are now serving floor plans directly from S3 to all major UK real estate portals. As well as vastly improving the scalability of our systems, S3 is already saving us over a thousand pounds a month, as Amazon's data transfer charges are 5% of what we paid previously and have proved to be just as reliable."

Max pointed me to a KMZ file for a house that's for sale in Southern England. After downloading the 12 MB file, Google Earth did its usual zoom in to the location of the house, and then rendered the building. As you can see below, I was able to check and uncheck "layers" to add and remove floors in the building. Really neat way to bring a 2D floor plan to life!

Initially both floors were displayed, showing a 3D rendition of the interior of this home.

By unclicking the upper floor layer, the ground floor was revealed

Sounds like entire towns might be next -- evidence that content is still king on the Net.

-- Mike

Serving KML, KMZ files from Amazon S3

Map-based Mashups are not new. Overlaying your data on maps are also not new. But serving your Google Earth's Keyhole Markup Language (KML) and its zipped (KMZ) files right off from Amazon S3 is new and innovative.

Few months ago, I blogged about Microsoft's MapCruncher tool that generates requisite tiles/files of your map mashed with Microsoft Virtual Earth and publishes them to Amazon S3 using the "Publish to S3" button in the tool. My Lake Roosevelt and Grand Coulee Dam crunched map is available here.

Likewise, some smart developers utilized Google Earth Network Links to publish files on Amazon S3.

Google Earth has the ability to display data imported from outside sources. Users can create KML, an XML syntax used to create data files, dynamically and overlay the information on Google Earth.

Sam Curren,  the creator of ActiveTrails.com, a site that let’s you upload GPS data of your hikes to create trail maps on Google Earth, wrote a nice article - Wicked Fast Google Earth Network Links using KML, Zip files, and Amazon S3. In the article, he nicely explains how he used all these technologies together to create a meaningful app that serves files real fast from the cloud. The article is nice read even though you are not a Google Earth user.

Arc2Earth V2 converts and publishes your ArcGIS data to view in Google Earth, Google Maps or Virtual Earth with a click of a button, right from the app. In the article, they mention what files they publish on Amazon S3 and how.

Likewise, Ogle Earth's Brian talks about a similar app that uses Amazon S3 for Google Earth's Network regions.

Of course, my hot favorite is WeoGeo - one-stop marketplace for mapping content. WeoGeo is a killer Ruby on Rails app, built and hosted on Amazon EC2 and uses Amazon S3 to host their KMZ files that you purchase off their website. These high-resolution professional mapping content can be terabytes in size and because Amazon S3 scales as you go, WeoGeo is able to reach to a larger audience while keeping their infrastructure costs down. Do checkout WeoGeo's cool, very intuitive, iPod-style interface, if you haven't already done so.

-- Jinesh

More Choices - All backed by Amazon S3

It is always great to have more choices:

DigitalBucket - Your Windows Explorer On the Web

DigitalBucket is now Live - really cool windows-explorer-style online file management, sharing and publishing backed by Amazon S3. Users can sign up for an account with 1GB storage and 3GB bandwidth for free.

Greg Hacobian, one of the developers, in his own words:

AWS gave us the opportunity to start a business on top of S3 without worrying about buying Terabytes of storage. Running this kind of business in the past was impossible without raising funds from venture capitals. Now, digitalbucket.net is privately funded and has 2 employees.  It was like dream come true when we were first introduced to S3.  The opportunities are endless with AWS.

What I liked about digitalbucket was its intuitive UI and its Windows-Explorer-Look-and-Feel that has zero-learning curve for any windows users. Its packed with AJAX, optimized by JSON, built on ASP.NET with MS SQL Server to store user metadata and relationships and uses Amazon S3 as primary storage for files. It has all the features that you can imagine - rename, move, copy, share, publish, view videos, slideshows, subscribe to RSS feeds, edit office documents online (aha!), zipping and more. It also has a "Sub Accounts" feature - great for startups and small business firms where employees can share documents in one master "company account".

File123 - Fax-in, Email-in, Scan-in your files to appear online automatically

Users can simply choose to email their files as attachments to specified email address or choose to fax their documents to 888 number or install a small utility and choose to scan and upload files with one click. All the files then appear automatically in their own online secure storage vault. Of course they can also upload digital files manually, manage and share documents using the online file manager too. It automatically gets indexed for searching. All documents are stored on Amazon S3.

Matt Dusig, one of the co-founders, File123 says:

You can store tax returns, wills, business leases, insurance claims, auto registrations, contracts, pay stubs, bills, receipts and other documents you deem important.  You can also backup all your digital photos or video files.  File123 makes organizing and backing up your life easy.

Users can also sign up for an File123 account with different plans (inlcuding free 1GB storage plan).

-- Jinesh

Amazon S3 - the new Asprin

As more and more people now move their production apps on Amazon S3, we are getting emails from CEOs and CTOs about their success and how Amazon S3 helped them sleep better at night. Last week I blogged about live-blogging backed by Amazon S3 and their 2-hour-$10-scaling app. This week its Pictogame. As Louis Choquel, President of zSlide (of Podmailing fame) puts it in his own words:

- with S3 we could sleep better, spend totally cool week-ends watching our Digg score climb in total serenity.

- without S3 we would have spent much more money but slept badly anyhow, had nightmares, and actually seen our server crashed, probably at the worst time of night as these things usually do.

Pictogame launched on May 25 - its one of those user-created game widgets that you can create dynamically and embed it on your MySpace blogs. Technically, the app consists of several general purpose SWF flash files (game loader, skin, game template), user media files (currently only pictures: jpeg, gif, png), an XML file describing how to mix all that into a customized game. All of these are stored in a single bucket on Amazon S3. SWF files are stored once as they are common to all, user media are first uploaded to their servers for processing then pushed to Amazon S3. XML files are uploaded to Amazon S3 each time a new game is created. A background process in PHP stores all these files asynchronously on Amazon S3 while they keep the Amazon S3 key in their database for later editing and deletion purpose.

Their stats are all the more impressive: Since launch (in 3 weeks), they got "dugg" several times and were also there in Top Ten for few hours. Overall, over 75GB were downloaded and 1 million widgets were served (Each game widget weights an average of 150KB) and Total costs were less than $20 (for storage and bandwidth).

Now since the widgets get directly relayed from Amazon S3, they don't have to worry about scaling. No more post-production headaches!

Here's the game - Try building the famous "AWS building block" (My best time : 400sec)

Put this game on your blog   Create your own   More games

Have Fun!

-- Jin

Live Blogging Experiment Results - Sitening.com

I blogged about Sitening.com's Live Blogging Experiment yesterday. They were trying to capture the live feeds from Steve Job's Keynote at WWDC 2007 Event. For the past several years, any site which live-blogged the event would become hopelessly buried in traffic.

They built a small admin tool that periodically encapsulates the content every minute or so and posts it on Amazon S3 Bucket and lets Amazon S3 handle the load. In the followup blog post, they tell us more about their tool and share the results of their 2-hour experiment. The Blog post is a good read.

• 7 pictures
• 130 text posts
• 20k visits
• 50k pageviews to the page
• 5M requests - this includes the 15 second content refresh and the images.
• 47GB of transfer
• content was 17k, images were 200k.

Cost for the event: $10.

This demonstrates the true power of Web-scale: A massive scalable infrastructure to back them up when they need it and pay for only what they consume. This model is validated and affirmed during these types of "live" and "spiky" events.

Now I can't wait to see their next experiment "Live Video Blogging"

Nice work Sitening.com!

-- Jin

Apple's WWDC Keynote updates - Powered by Amazon S3

If you are following the hype of iPhone and Leopard, then you would know that WWDC's Keynote is so inevitable. Jon Henshaw and Tyler Hall from Sitening is blogging live from Keynote room (10:00AM) as I type this post up. Its powered by Amazon S3.

You can get Steve Jobs' Keynote updates here

This demonstrates the true Power of Scale, as he mentions on his blog post, he does not have to worry about how many people will log in or how many times he updates the page, It will simply auto-scale.

This is going to let us reach a ton of people without worrying about bandwidth, infrastructure, etc.

Currently we can see that they have photos and text, It would have been even cooler if it would have been live video updates. It looks like they have created a cool utility that automatically updates an Amazon S3 page and also sends signals to Twitter Account.

May be Jon or Tyler can tell us about their cool little "Live Blogging" app.

-- Jin