Amazon S3 Shoots, Scores, Soars

Developers have been pouring data into Amazon's Simple Storage Service (S3) at a prodigious rate. In fact, there are now over 800 million discrete objects stored in S3.

Earlier today we announced some of our successes in a press release. Some of the highlights include:

• The aforementioned 800 million stored objects.
• Argentinian newspaper La Nacion used Amazon S3 for banner ad storage, allowing them to withstand the onslaught of traffic brought on by the World Cup.
• Microsoft is using Amazon S3 to make software available to students as part of the MSDN Direct Student Download program.
• Photo storage and sharing site SmugMug is using Amazon S3 to deal with the scaling requirements brought on by success, adding 10 terabytes of new photos each month without investing in new hardware.
• Online storage provider Elephant Drive has doubled its user base and quadrupled the amount of data stored since switching to Amazon S3.
• Backup vendor Jungle Disk was up and running with Amazon S3 less than 30 days after they started development.
• Media Silo is using Amazon S3 to store digital media assets for video professionals.

-- Jeff;

Comments

[Brad Neuberg]

Hi there! S3 is amazing and leading to lots of very cool innovations.

Two things that would make S3 even more useful and change the game:

* Some of us have been experimenting with hosting our Ajax and DHTML apps completely through S3. This is a natural complement, since an advanced Ajax application can be written in JavaScript, CSS, HTML, etc. that is simply served from the S3 store, and can then talk back to S3 using XmlHttpRequest; you can create sophisticated apps that are a single delivery of JavaScript, HTML, etc. and not have to maintain a web server. Remote data can be saved into S3 to handle each user of your Ajax application. Even better, these Ajax apps can script web-services on other domains on the client side using one of the various tools to do so, such as JSON-P, Julien Couvreur's FlashHttpRequest, etc.; Yahoo is already exposing many of it's web services in a secure, client-side invokable cross-site manner. When you combine this with the work I've been doing with Dojo Storage, which gives you a local client-side store that can hold megabytes of data (with the user's permission) and works across 97% of the existing net, you have the possibility of creating an entire explosion of new apps that don't require server maintenence and which can store data remotely with S3, script other services, and store data locally. One big problem: since all these apps are served from the same domain name, they can all read and spy on each other's cookies, local stored data using Dojo Storage, etc. The solution is to setup Dynamic DNS to have a third domain name that can be registered on s3, such as myapp.s3.amazon.com, wordprocessor.s3.amazon.com, etc. as part of the S3 process. This would mean that cookies, Dojo Storage, etc. would be locked to this domain, which would make this approach much more realistic.

* The second big part of changing how apps are deployed on the web (and vastly opening up the ability to do innovative work and create innovate web apps, lowering the cost of entrepreneurialism) is to have a standard querying service. Why don't you take the OpenSearch service and use it to somehow do relational-like querying of S3 data? This is a bit of an unsolved problem, but I don't think it would be too hard. Think of something like map/reduce or a search-engine like API above semi-structured information stored in S3, perhaps against XML. Maybe it would simply be XPath you could use against an XML file or set of XML files. This is an open area where you could make alot of impact, like S3; hosting and 'feeding' relational databases is not easy and is damn expensive (and they don't really scale to millions of people very well at the end of the day), and is the other flip-side after basic data storage like S3 handles.

When you put this together with the work me, Julien Couvreur, and others have been doing with offline access (using things like Dojo Storage), plus S3 and an Amazon standardized-querying API over semi-structured information, I think we can blow open the kinds of apps possible on the web.

Finally, make sure that S3 has a giant list of mime-types configured, so that if you host OPML on there, for example, it gets served under the XML mime type; I think that one in particular might not get served right, but I don't remember correctly.

Best,
Brad Neuberg
bkn3@columbia.edu
http://codinginparadise.org