Amazon Web Services Blog

AWS Short Takes for Friday, April 18, 2008

Time for an Inbox cleanup...

I met developer Chris Richardson in Philadelphia last month. Chris is a seasoned Java developer and the author of POJOs in Action.

He told me that he had just released CloudTools. This is a set of tools for deploying and testing Java EE applications on Amazon EC2. It consists of AMIs that are configured to work with Apache Tomcat and to work with EC2Deploy, the EC2Deploy core framework (described here), and a Maven plugin which uses EC2Deploy. Once the plugin has been configured, one command will launch the appropriate number of EC2 instances, configure a master MySQL database, populate it with data, configure zero or more MySQL slaves, configure one or more Tomcat servers, deploy the actual web application, and then configure an Apache instance which will load balance across all of the Tomcat instances (whew!).

Tarzan is an set of PHP 5 modules which provide straightforward, object-oriented access to Amazon S3, EC2, SQS, and the Amazon Associates Web Service, with planned support for SimpleDB as well. Currently in late-beta, most of the code is accompanied by unit tests. It requires access to a few common PHP extensions such as SimpleXML and PEAR HTTP Request. Tarzan was used to build Warpshare. The author is seeking patches, new code, and bug reports from the community.

We have just published the code for a Javascript-powered EC2 scratchpad in our Resource Center.

You can load this code on to the server of your choice, or you can simply unpack it into a directory on your desktop machine. Once loaded, simply open up index.html, enter your AWS keys, and start exploring the EC2 APIs.

The scratchpad provides a form-based interface to each of the function calls. It also computes and displays the complete signed URLs needed to actually make the calls into the EC2 cloud. The scratchpad can be used to learn about how to make calls to EC2 or as a cross-check for your own service invocation and request signing code. The return value from each call is displayed in raw XML form.

In a new Business Week cover story, Amazon.com founder and CEO Jeff Bezos discusses his approach to innovation, thinking for the long term, our focus on the needs of customers, and our company culture.

This focus on the needs of our customers is very real and permeates everything that we do at. The information that we gather from meeting with developers, from reading commentary on blog posts (many of which show up on the AWS Buzz), and from our interaction on our developer forums is an intrinsic part of our product planning process.

Speaking of customers, we've been adding a lot of new customer-generated content to our Resource Center. Mitch Garnaat wrote about his Son of Monster Muck Mash Mashup, and also contributed an EC2 AMI. A guy named Mark contributed an S3 library for the Runtime Revolution language. Brenton Simpson wrote an ActionScript 3 library for SimpleDB. I've been tracking all of this content by simply watching the RSS feed for the Documents section of the Resource Center.

That's about all I have time for right now!

-- Jeff;

Dekoh - Amazon EC2 case study

This is a great case study demonstrating how smart developers are moving their production environments over to Amazon EC2.

Last December, I met with Pramati Technologies - a well known household name when it comes to JEE Web Application Server. I learnt about their really cool Web 2.0 product: Dekoh. Dekoh can be described as 'Facebook on the desktop'. It consists of the Dekoh Desktop and the Dekoh Network. The Dekoh Desktop runtime - a JEE compliant small footprint application server - runs on the user’s desktop and converts it into a secure server. The Dekoh Network is a complete server-side solution ecosystem built on top of Amazon EC2 environment. It handles centralized user management and "presence" management.

Third-party developers can write rich applications on top of the Dekoh runtime and can take advantage of the Dekoh Network (friends, groups access permissions). All communications across firewalls is managed by the Dekoh runtime. The Dekoh Network has been built to scale on-demand so when thousands of dekoh applications access the hosted architecture, a few clicks will provision Amazon Machine Images and will dynamically join the topology. This is managed by sophisticated load balancing and dashboard monitoring also built by Pramati. I was impressed by their energy, their Java expertise and their passion.

In an email, they shared:

A virtual computing platform like Amazon’s EC2 serves as a perfect fit for Dekoh. Our experience building production configuration for the Dekoh backend components on Amazon’s EC2 platform has proved to us how quickly an application can be put in production mode without worrying about bottlenecks of the underlying server resources.

I had an opportunity to dig a little further into their hosted architecture.

Each desktop instance connects to a set of of loosely coupled components hosted in the EC2 cloud. The user access component is managed through a 'Login' instance, and the user's list of friends and groups is managed using 'Profile' instance. The content sharing component along with user availability is managed through a 'Presence' instance. Each component is pre-configured in the form of an AMI.

To ensure high-availability and failover there are multiple instances of the Login, Profile, and Presence servers. Each component has an individual load-balancing node which is a custom-built Pramati software Load Balancer. During peak loads, a new instance with its load balancer can be automatically started. Each component manages its own master-slave database replication model. I think this type of loosely coupled on-demand model is not only scalable but also easier to manage than some of the other models I've seen.

Pramati takes pride in their Java expertise on the Amazon EC2 environment and would like to help customers who are eager to move their Java-based production systems to Amazon EC2. Their automated scripts not only sets up the required instance but also updates the load-balancer configuration files to reflect the newly added node. Automated scripts also provide data for real-time monitoring of network topology (CPU/RAM usage etc).

It seemed like Pramati knows what it takes to build a large-scale production system on Amazon EC2. Hence, we invited them to share their architecture, insights and best practices with the community. Pramati is going to co-present with us at the upcoming TheServerSide Java Symposium on March 28th.

I strongly encourage you to check out Dekoh! 

-- Jinesh

New Version of Bungee Connect, with Amazon FPS Support

My friends over at Bungee Labs have just rolled out a new version of their web-based Bungee Connect development tool. New features include better ways to manipulate field, class, and function definitions, a statement completion control, and support for the Subversion revision control system.

Bungee Connect is currently in "early access" beta mode. You can sign up (for free) and start using it here.

They have also introduced a library to simplify the task of calling the Amazon Flexible Payment Service (FPS) using the Bungee Logic language. As you can read in Brad's blog post, the new library is based on the REST version of the FPS API. The library handles a number of low-level details including the actual creation, signing, and issuance of requests, along with processing of the results.

Brad wrapped each FPS function and set each one up to return a Bungee object as the function value. He used the FPS WSDL file to define all of the relevant types and then added a few of his own for good measure. All of the interesting details describing how he did this are laid out in his blog post.

Using this library it is easy to make FPS calls such as Pay, Refund, and GetTransaction. The library also supports the Amazon co-branded UI pipeline with utility functions to construct the URLs needed.

-- Jeff;

PS - Part 2 of my recent interview with them is now available in podcast form. Part 1 can be found here.

CohesiveFT's Elastic Server

As part of my most recent trip to London I paid a visit to a suite of offices shared by LShift and CohesiveFT. I gave them an in-depth AWS presentation and we also discussed Cohesive's Elastic Server product.

Elastic Server simplifies the process of creating application stacks for use on EC2 and other virtual environments. Instead of the painstaking manual downloading, configuration, building, and installation that is otherwise required when creating a complex stack, Elastic Server provides a menu-driven selection and build process.

After choosing the desired web service environment, workflow framework, J2EE server, enterprise service bus, Java servlet container, page server, message queue and AMQP broker the user simply enters the desired virtualization parameters and initiates the build process. After a few minutes the build process spits out a a finished EC2 AMI. From there it is easy to launch one or more instances of the AMI using the EC2 Firefox Manager or another EC2 tool.

Each application stack is configured to include a copy of Cohesive's web-based Elastic Server Manager for easy management of each component.

If this sounds cool you should definitely watch the video to see how it works.

-- Jeff;

Amazon Flexible Payments Support Sees Community Support

Adoption and community support are two measures that help determine whether an idea will thrive in the real world. As long as Amazon Flexible Payments Service is still in a limited private beta, it's hard to gauge success by adoption; however community support is still valid as a metric.

So Phil Burns' efforts are particularly interesting to me. Phil created a VB.NET library for FPS--something that I personally believe will be valued by the rest of the community. Experience says that VB.NET developers are highly interested in eCommerce.

As he points out in his blog posting, payments is more involved than simple "download and run". However if you are a VB.NET developer Phil has freed you from the tedium of rewriting C# code into VB.NET.

Do you know of other libraries or extensions to this service? If you do, we'd love to hear from you.

-- Mike

Ruby on Rails - Public AMIs

Paul Dowman created something really useful for the Ruby on Rails Community: a Pre-configured Ruby On Rails Stack AMI that "just simply works". Public AMIs are pre-packaged, pre-configured, compressed file-system blobs that are stored on Amazon S3 which can be instantiated by any Amazon EC2 user.

Ruby on Rails cuts down development time significantly due to all the freebies and code generators that comes with it. Now with these types of new public AMIs, it will also cut down deployment time too.

With Paul's Public AMI, developers will get all of following pre-configured and ready to go:

Features:

• Automatic backup of MySQL database to S3 every 10 minutes.
• Mongrel_cluster behind Apache 2.2, configured according to Coda Hale’s excellent guide, with /etc/init.d startup script
• Ruby on Rails 1.2.3
• Ruby 1.8.5
• MySQL 5
• Ubuntu 7.04 Feisty with Xen versions of standard libs (libc6-xen package).
• All EC2 command-line tools installed
• MySQL and Apache configured to write logs to /mnt/log so you don’t fill up EC2’s small root filesystem
• Hostname set correctly to public hostname
• NTP
• A script to re-bundle, save and register your own copy of this image in one step (if you want to).

In the Blog Post, Paul has a nice "Instructions Manual" to get you started. There was significant confusion among the Rails community whether to use HAProxy, Apache mod_proxy_balancer, nginx and so many other alternatives. Paul was smart enough to just put the right ingredients in his recipe.

I can't wait to see the entire network topology from load balancer to MySQL master-master replication model and a cluster monitoring solution in one single AMI that we can instantiate by invoking simple commands using Parameterized Launches feature of Amazon EC2.

Imagine a Zero-configuration world, where we simply reuse other developer's once-done configuration and even optimizations!.

Thanks to Paul for taking lead on this and sharing with the world! Travis Reeder shares his expertise with an example (in the comments section of his blog post) on how to normalize your data fields in a database by storing big blobs onto Amazon S3 thereby making the database less heavy and easier to manage.

This is what I like about this "Social computing" era - willingness to share and collaborate. Hats off! to all the developers who share their development experiences so that others don't have to go through the same pain of configuring, tweaking, patching components.

-- Jinesh

p.s. we are encouraging people to put their Public AMIs in our Resource Center "Public AMIs" bucket so other developers can find them easily.

Estimate your costs - AWS Simple Monthly Calculator

We created a JavaScript-based tool that allows you to calculate your monthly cost for using Amazon S3, Amazon EC2, and Amazon SQS.

This tool incorporates the latest pricing changes including the tiered pricing model for download bandwidth.

Use this tool to estimate your monthly bill, to determine your best and worst case scenarios (if you get Slashdotted, Dugg etc.), and identify areas of development to reduce your monthly costs and even compare it with other service providers who do not offer utility-style of billing (pay-as-you-go).

You can view the calculator here:

http://calculator.s3.amazonaws.com/calc5.html

Any feedback to improve this tool is always welcome. Please use the comments section to let us know whether this tool was helpful.

-- Jinesh

Amazon ECS Widgets

Couple of cool widgets available that enable you to earn Amazon Associate commisions.

The first is the Amazon Browser Widget, which their website describes as "The Amazon Browser widget is a mini-Amazon browser that allows you to search for items at Amazon. As a user scrolls through the displayed items, it automatically loads additional items as necessary. The widget keeps track of browsing history, and the forward/back buttons can be used to view previous searches."

The other is the Amazon List Widget, which displays Amazon lists, including wedding and baby registries, Listmania lists, and wish lists. As a user scrolls through the displayed items, it automatically loads additional items as necessary.

Both of these are available at http://www.widgizit.com/.

-- Mike

/n Software Adds Amazon Support

The community around Amazon Web Services is really on a roll! One of the communities we hear from a lot is the ISV world. For example I met the CEO of /n software, Gent Hito, at ETech a couple of months ago. Gent's company builds BizTalk adapters and SQL Server integration components. Gent was excited about their upcoming product release, which integrates Amazon Web Services. He just followed up with me to say that they are shipping!

According to /n software, their IBiz Amazon Integrator supports Amazon Web Services in several areas:

• Their BizTalk Adapters and SQL Server SSIS Tasks now support Amazon Simple Storage Service ( Amazon S3).
• The IBiz Amazon Integrator includes developer components for accessing popular Amazon Web Services such as Amazon Simple Storage Service (S3), Simple Queue Service (SQS), and E-Commerce Service (ECS).
• The new Amazon S3 Adapter for Microsoft BizTalk Server and Amazon S3 Task for SQL Server Integration Services allow for zero-code integration of Amazon S3 Storage with BizTalk or SQL Server workflows. These adapters enable their respective Microsoft products with Amazon's secure and reliable online data storage capabilities.
• In addition to the S3 offering, /n software is scheduled to release an Amazon SQS Adapter for BizTalk in the near future, enabling BizTalk with access to Amazon’s always-on distributed messaging platform.

You can read more about the product at www.nsoftware.com/ibiz/amazon/.

And if you're an ISV that supports Amazon Web Services, please tell us about your product! Ou email address is "evangelists at amazon dot com".

-- Mike

Start Crunching - Create Mashups using MapCruncher and Render on S3

Microsoft Research developed an innovative tool called MapCruncher for Microsoft Virtual Earth that takes any map-based image (JPG or PDF format) and overlays that image onto Microsoft Virtual Earth to generate requisite tiles. These tiles can then be used to display the map in Virtual Earth / Google Maps fashion (zoom in/out, pan, move etc). More info available here. Researchers recently added a new feature called "Render to S3", so now all the tiles/images/javascript that gets generated off mapcruncher can be directly uploaded to Amazon S3 right from the tool and your mashup is instantly available to the world - right hot out of S3.

Pretty neat!. So now you can not only create map-based mashups in minutes but also host them.

I could not wait to try this out. So I took the camping map from my last trip (Lake Rooselvelt and Grand Coulee Dam) and decided to crunch the NPS-published high-resolution map that shows all the campgrounds and boat launch information and overlay it on Microsoft Virtual Earth using the tool. Then I created 6 render points and let the mapcruncher algorithm do all the number crunching to figure out how may tiles I need, what tiles I need etc. I locked my images with Microsoft Virtual Earth and saved my mashup (lake.yum). It estimated that I would need 3990 tiles around 340MB of data. Now the best part, I clicked on radio button "Render to S3", it asked me for my AWS credentials and boom! it started uploading all the images/tiles to S3 with just one click of a button. After a minute or so, I got a brand new mashup of Lake Roosevelt and Coulee Dam superimposed on Microsoft Virtual Earth and hosted on Amazon S3.

Obviously, you can see that I am not a expert mapcruncher and the map got skewed to adjust with resolution. Part of the reason was that I only used 6 render points (minimum) instead of creating more for enchanced accuracy. The tool also has features to adjust transparency, which I did not get a chance to play with.

However, the thing to note here is Microsoft Researchers were able to integrate highly-scalable storage infrastructure to their application - mashup in minutes - using the power of Web Services!

Next, I am going to create my neighborhood community mashup where I will overlay all occupant information on Virtual Earth so my neighbors can find other neighbors :)

Happy Crunching!

-- Jinesh

Who's using Ruby on Rails?

If you are an Amazon EC2 or Amazon S3 aficionado and are using Ruby on Rails, we would like to hear from you. Particularly we are interested to know how you are using Ruby on Rails, your architecture, your EC2 network topology setup, your S3 hacks. Anything that can help the newcomers to ramp up fast. Ruby on Rails cuts the development time by 50%, AWS eliminates the headache by 70%, and your ideas/experience will help the new comers even more. We would love to know how you are using Amazon Web Services and Rails together. Please use the comments section to describe your product and how the Ruby on Rails framework has helped you to reach your goals.

We know about a few that are already using AWS/Ruby on Rails : RightCart, RightScale, 37Signals' BaseCamp, SlideShare, WeoGeo. In fact, Slideshare's Jon Boutelle has a cool presentation about Ruby/Amazon S3 architecture.

In my previous post, I listed various Ruby libraries that have been built to use AWS. If you have new ones or have a new version, let me know and I will add one more precious stone (ruby) in the already-shiny AWS throne. Few that caught my eye were:

• Capazon - a cool capistrano extension to manage Amazon EC2 instances
• CLI to Amazon S3 Service using Ruby - an interesting Command-Line Interface to Amazon S3 in Ruby

-- Jinesh

phps3tk - Command Line Scripts for S3 Access

I met dozens of entrepreneurs and developers at last night's Start-Up Seattle event (you can read more about it here, here and here).

Among those I met were David Busby of Edoceo.com . Through his company, David provides small and medium-sized businesses with IT solutions powered by open source technologies.

David told me that he had written an Amazon S3 access toolkit in PHP. I urged him to release it sometime soon, and promised him that I would blog about it. Apparently my arm-twisting was successful -- earlier today he announced that phps3tk was now available!

phps3tk provides an API and a command-line interface to Amazon S3. Once installed, you can use commands like "s3 ls" to list bucket contents, "s3 get" to retrieve data from an object, and "s3 put" to store data in an object.

There is also a programming interface wrapped around the S3Cache, S3Connection, and S3Response objects.

Nice work, David!

-- Jeff;

New PHP Interfaces to S3, EC2, SQS, and Mechanical Turk

Earlier today, Tyler Hall of Sitening announced that they were releasing a set of open source PHP interface classes for Amazon S3, Amazon EC2, Amazon SQS, and the Amazon Mechanical Turk. The classes are hosted in Google Code, and can be downloaded here.

Tyler also let me know that these classes were used to build Sitening's Basejumpr application.

They actually spool up EC2 instances on demand in order to do heavy duty batch data processing. As Tyler told me:

We couldn't do it with just one server, and being a small company, we can't afford to lease multiple dedicated servers which might sit idle most of the time. With EC2 we launch and shutdown instances as needed. It's typically an automatic process controlled by our web server. But we've also built a control panel so we can start and stop instances manually.

Once again, it is a great example of Web-Scale Computing in action (I'm sure you are getting tired of hearing me say that).

-- Jeff;

AWS and The Windows Presentation Foundation

WPF/E is Microsoft's next-generation presentation for creating rich web experiences. Using an XML vocabulary known as XAML, WPF/E supports both cross-browser and desktop applications which run on Windows and on the Mac.

If you would like to learn a bit about the use of WPF/E in conjunction with the Amazon E-Commerce Service, you should check out Dan Wahlin's new article,  Viewing Albums with WPF/E and ASP.NET AJAX.  Dan's been experimenting with WPF/E, JavaScript, ASP.NET AJax, and Web Services and this article pulls it all together. You can even view a video of the application in action.

-- Jeff;

Precious - Ruby, Gems and AWS

Ruby got a new face-lift with the new Rails framework. I was always fascinated with the Ruby language. 25 lines of Java code gets shrunk to 4 lines of equivalent Ruby code plus its more readable. And with Rails, it gets a new kick. It is always impressive to hear people say "I am a full-time Ruby on Rails Developer".

There are plenty of resources for those who would like to get started on this technology:

There are lots of Amazon S3 libraries in Ruby: Some are full fledged libraries like AWS:S3 (Marcel Molina of 37signals talks about his library and why he chose Amazon S3 here - worth a read!), while some libraries are getting started types so that you can customize as per your needs. Some are command shell utilities like rSh3ll, while some are gems like S33r and ruby-s3 for easy installation.

Zachary Holt created a powerful library for Amazon SQS in Ruby. He has added some cool utility functions such as ability to forcibly delete a queue etc.

If you would like to get started with Ruby/Rails on Amazon EC2 instance, all you have to do is fire up an instance and type "yum install ruby" and you are all set!. To get Rails, follow two additional steps mentioned here.

Ruby on Rails reduces your development time and gets you up and running in minutes. There are several apps which have been built using Ruby and Rails framework. Some that caught my eye were:

• All new! SmilePooling - SmilePooling connects Amazon's "smily boxes" (read sidetrack below) and Amazon ECS's WishLists with the car-pooling concept. It is a web application to order items from Amazon stores for you and your friends and colleagues, so that you can split the shipping costs and save. Nice idea, isn't it ?
• Competitio.us - simple handy tool to track your competition. Competitio.us is powered by the Alexa Web Information Service and the Alexa Thumbnail service. It gives all sorts of traffic data and other information for your competitor's website.

Are there any other cool Ruby on Rails applications powered by AWS ?

--Jinesh

Sidetrack: Have you guys noticed the amazon.com logo and wondered what the arrow signifies? I did not know until now. The arrow means Amazon has everything from A to Z and it also represents the smile it brings to the customer's face - Quite deep, isn't it?

update: Also, one more  -  Ruby/Amazon - Amazon ECS library in Ruby.

All new and improved S3Fox Organizer for Amazon S3

Earlier today Jeff barr told me that he uploaded all his photos to amazon S3 using S3Fox Organizer. S3Fox Organizer, developed by Rahul Jonna, is a firefox extension for managing your objects in Amazon S3. We had blogged about it earlier. Motivated by the number of hits he received, Rahul released a new version (v0.3) of his tool with some really cool features:

1. Synchronize folders : By mapping your local folder with remote folder (on Amazon S3), it will automatically sync up all the files in the folders.
2. Drag-n-drop files from the system (windows explorer, etc) to Amazon S3 : By clicking on the "s3fox" icon in the firefox statusbar (which will open a new panel) and simply dragging-n-dropping your files onto that panel, files will be uploaded to the remote directory you are currently in.

In this new version, he also fixed some content-type related bugs.

His slick "ACL manager" and very-useful "Copy URL to clipboard" features are also worth a look!!

Best part of this is: the tool is just a 70KB download and takes less than 5 seconds to install.

Nice Work Rahul!

--Jin

My First Post and thoughts about standards-based world

Hiya, I am Jinesh Varia!

This is my first blog post on the AWS blog. Its so cool to be an evangelist, travel around, talk about what cool things people are building and most importantly talk about what cool things YOU can build.

Nowadays, Anything and everything that has a high "coolness-factor" gets immediate attention from the crowd. This is indeed leveling the playing field. With services like Amazon S3 and Amazon EC2, I believe there will be no difference between the student in the dorm room and the executive in the board room. The one with the best idea or coolest idea will win the game. And the one who gets there fast will be rich ;-)

I hail from the standards world of IT. Prior to joining Amazon.com, I was involved in evangelising XBRL (Extensible Business Reporting Language) to FFIEC. We managed to evangelize/design/implement this standards-based technology to the Federal Government banking agencies and now over 8600 banks are reporting to FDIC/OCC/FRB every quarter in XBRL format.

Like SOAP, WSSE and other standards, XBRL is not easy. There is significant learning curve to understand the concept of taxonomies, linkbases, xbrl concepts, tuples, arcs and extended links. Normalizing Semantic Data, creating business rules using XPath and XLinks can be nerve-cracking easily.

But now, after working through various web based standards, I am feeling standards are always going complex for majority of people and they are meant to be. The elaborate and makes-me-sleepy-when-I-read nature of standards totally makes sense as it helps two entities "talk" in the same language.

So what will make these complex standards more easy-to-use and easy-to-implement? Code Samples / Starter Kits / SDKs / Software tools / Helper classes / libraries are going make the standards more "readable", more implementable. Not many people have to know or worry about what is Unique Particle Attribution Error in XSD schema specification or Exclusive canonicalization of XML transforms in WSSE Header specification. Let those who make the starter kits go through the pain and worry about that. Concentrating on your idea, using starter kits and not worrying about the muck,  will get you where-you-want-to-be faster! and remember one who gets there fast becomes rich ;-)

I will be down in Los Altos, CA at siliconvalley code camp this weekend. If you are around, please ping me and I would be more than happy to hear your comments/suggestions/criticism.

-- Jin

Using SOAPSonar to Provision Amazon Elastic Compute Cloud

Mamoon Yunus, author of the SOA Testing Blog and advisor to Crosscheck Networks sent me a link to an interesting and relevant How-To document.

Using SOAPSonar to Provision Amazon Elastic Compute Cloud covers downloading and installation of a free trial version of Crosscheck's SOAPSonar product and then shows the steps needed to load the X.509 certificate, set up SOAP authentication, and finally to call the Amazon EC2 SOAP APIs.

The document is very clear and explicit and definitely simplifies and makes obvious some steps that have proven to be difficult for some developers to master.

There's also information about how to load the EC2 and Amazon S3 WSDLs into SOAPSonar simultaneously and to use them to as the basis for a mashup.

Java.util.Map Implementation for Amazon S3

Tom White dropped me a note to tell me that he had created an implementation of java.util.Map on top of Amazon S3. As Tom notes, "Normally we strive for an implementation that is efficient in terms of speed, however in this case I was very aware of the monetary cost of each method."

Tom took care to document the number of S3 calls made by each method in his implementation, and, being a faithful S3 user, also stored the JavaDoc documentation for S3Map in S3! This table is quite interesting and should be very helpful to any developer using the class.

Way to go Tom!

-- Jeff;

Quick Picks for Friday

I use a handful of web and blog search engines which produce RSS feeds that I monitor to find interesting developments for this blog. Here are a few recent findings:

• There's a new version of Ruby/Amazon.
• Brian Krow is looking at S3 and Jungledisk, and wondering where he left his tape drive.
• David Aiken is having fun with Amazon S3 and the Windows PowerShell.
• Keith Bohanna is crying with happiness now that he's found Jungle Disk.
• Dion Hinchcliffe says that the Amazon Web Services are a good example of a WOA, or Web-Oriented Architecture.
• The Beloved Ruby Cookbook shows how to use Ruby to get Amazon data.
• Finally, Fabrice shows how to use Microsoft's upcoming LINQ technology to access Amazon data.

And that's a wrap! Have a good weekend, everyone.

-- Jeff;

Linked List - XSLT, PowerShell, .Net Sample, Hosting Site on S3

I am back from London and I have some interesting stuff queued up for you. Here goes:

• Eliotte Harold shows you how to "Debug XSL Stylesheets With xsl:message." As he says, "Echo printing is one of the oldest ways to debug a thorny problem. Nonetheless, it's still one of the simplest and quickest ways. When you aren't quite sure why a function doesn't behave as expected, print a few variables to the console with printf() or its equivalent to see just what's going on." In order to use this very handy technique to debug your ECS style sheets you would need to install an XSL processor on your own system.
• PowerShell is Microsoft's newest command processor. Originally slated to be shipped as part of Windows Vista, it is now available as a free download for earlier systems as well. Using PowerShell you can make HTTP calls to web services and you can process even process the resulting XML. Read all about this in the new article, "PowerShell and  Amazon Web Service."
• There's a new AWS sample on GotDotNet: Ajax Example using Anthem and Amazon Web Service. Anthem is a free, cross-browser open source toolkit that you can find here.
• Peter Lewis, writing from Australia, shows how to create An Amazon S3 Hosted Website using Interarchy. Interarchy is an FTP client application for the Mac.The technique illustrated in the article is not specific to this application and is quite handy - Amazon S3 is a really good web host. Of course, the article itself is on an Amazon S3 hosted website!

Openfount Queued Server

The Openfount Queued Server looks really interesting. It took me a little while to understand the architecture, but this was time well spent.

Basically, this technology interposes a set of processing queues between a web client and a backend server. The queues are apparently implemented using our S3 and SQS services. The client never talks to the server directly. Instead, it uses APIs from the Queued Server toolkit to write messages into a queue for processing by the server. The server processes messages and then writes return values into another queue, where the client picks them up and displays the results. I can't tell for sure, but I am assuming that the client has some form of event loop or notification system, making it easy to build dynamic, Ajax-style applications.

Because the server is actively polling the queue for messages, it need not have any public interface at all. No domain name, no IP address, and no vulnerability to well-known generic attacks. In fact, the server could be running behind a cable modem with a dynamic IP address and no one would be the wiser, since it simply reaches out to the world via standard HTTP requests over port 80. In this model the server never accepts incoming calls directly. It reaches out and pulls in requests, and can be very, very choosy about the requests that it accepts.

The clients and servers are isolated and protected from each other. It appears that the client need not even know the domain name or the IP address of the server!

It should be very easy to build very scalable web applications using this technique. When the server is momentarily busy the queues will simply back up a little bit. Since the server is in full control of request processing, it should never get flooded with requests. If the load becomes too much for one server to handle, it should be easy to add more servers. In either case, S3 and/or SQS provide the "elasticity" needed to allow the application to deal with an uneven traffic flow.

There's a tiered pricing model based on the number of unique users per month. Fees start at just $3 per month (100 users or less) and go up to $83 per month (one billion users or less). It isn't clear if this includes fees for S3 and SDS or not.

Update:Bill Donahue, developer of the Openfount product, has posted a must-read followup comment.

-- Jeff;

Using AWS and XSLT to create JSON

It is always interesting to see how a developer's background, experiences, and the contents of their "mental toolkit" come together in unique approaches to problem solving and creative thinking. Here's an example of what I am talking about.

Alan Taylor of Kokogiak.com has been building applications with AWS since the early days, starting out with his pioneering first version of Amazon Light in 2003. With version 3 of his site, Alan showed that it was possible to build a complete and highly functional web site using a single CSS style sheet, Amazon ECS, an XSLT transform, and a single "entry" URL. Along the way, Alan has clearly learned the ins and outs of XSLT; in fact I think that it would be fair to say that he more or less "thinks in XSLT" (that's a compliment, of course).

In his latest creative endeavor, Alan has figured out how to use an XSLT transform to convert raw Amazon ECS data into JSON objects. JSON, as you may know, is a very simple string-based data representation. Data structures of any complexity are represented as JavaScript expressions that, when evaluated, return fully-formed, easy to process JavaScript objects. This makes JSON an ideal way to feed data directly to browser-based applications.

Alan has written a comprehensive blog post on the subject, and I'll refer you there for more information. Take a look at his new article, Consuming Amazon's Web API Directly with Javascript, to learn more. The article introduces JSON and XSLT and then shows how to use them in conjunction with Amazon ECS. There are five complete examples in there, along with very complete explanations of how they work.

Anywhere, Anytime Storage and Retrieval with Amazon's S3

DevX just published a new article, Anywhere, Anytime Storage and Retrieval with Amazon's S3.

The author starts out by noting that "Unofficial testing of the service proves that it is fast enough to support applications requiring near real-time access to data. Further, the service's pricing makes it a great option for a diverse set of applications requiring fast and reliable online-storage."

The article then continues on to describe the construction of an application designed to store and retrieve information in Amazon S3 using the SOAP interface. Pricing is reviewed, noting that "it's difficult to imagine building a storage service that would cost less that could match the capacity, speed, and reliability claimed by Amazon.".

The author reviews the data model and the APIs, covers the sometimes-daunting subject of request authentication and signing, walks the user through the signup process, and concludes with an in-depth look at the sample application.

-- Jeff;

 

IBiz S3 Integrator

Gent Hito of /N Software sent me some information about their new product, the IBiz S3 Integrator. The S3 integrator (currently in beta testing) provides an easy to use developer interface for accessing Amazon S3, with full access to create, modify, and delete buckets. Includes documentation, help, and a sample S# browser application. Developers will need an S3 account in order to get started.

It is worth noting that this product is available for the .NET, ASP.NET, ActiveX, Delphi, Borland C++, Windows C++, Java desktop, Java server, and Windows CF (Compact Framework) environments. Very nice!

-- Jeff;

AmazonCommerceService.NET - Free .NET Library for ECS Access

Ed Quinn dropped me a note with information about his newest creation, AmazonCommerceService.NET.

AmazonCommerceService.NET is a set of .NET classes. It creates REST requests for ECS and then queues them up, dispatching them to Amazon at the rate of 1 request per second per the license agreement. The results are delivered asychronously as they arrive, using a delegate. Batched requests are supported.

All source code is included (C# and .NET 2.0), as is a sample web site built using the library. You can use the free Express Edition of Visual C# to develop an application using this code. You will need a GotDotNet or Passport account in order to see the code.

-- Jeff;

I'll Take Potpourri for 300, Please

This is my 300th post to this blog, and I'll celebrate by posting a veritable potpourri of interesting items.

• In a rare self-congratulatory note I will observe that we are the first entry on the Fortune 500 Business Blogging List.
• Priceheat is a one-click Amazon price-checker bookmarklet.
• Mark O'Neill talks about Security for REST Web Services, and also posted a link to a very informative presentation.
• Speaking of REST, I learned today (via the Microformats-REST list) that there is now a Simply Restful controller plugin for Ruby on Rails.
• Chris Feldman has posted the PHP5 code for a web-based S3 interface. You will need a web server, PHP5, and a couple of PEAR classes.
• The S3/Fuse project ("S3 as an infinitely large disk drive") is progressing nicely and now has a Wiki page of its own. For Linux, some assembly required!
• Jamie Wallingford is working to combine  Amazon, XML, XPath, and C# 2.0. As he says, "Another example of why it pays not to be in a hurry and to read the API."
• MyThingo helps people organize their personal possessions, keep track of what they loan to friends, and showcase their collections - "MyThingo was the result of a few guys getting tired of keeping track of their movies, music, books, hardware tools."
• Over at Codesworth, Hong Son has a couple of cool ECS and AWIS applications for free download. i-myZon is a gateway to Amazon, a .Net 2.0 application which supports searching for products and for wishlists, price alerts, and lots more. Download, screen shot. aMiner is another .Net 2.0 application, this one provides web data mining using the Alexa Web Information Service. Download and screen shot.
• Interview with Nasser Manesh, CTO of Frucall (mobile shopping search tool), previously blogged - "We are an AWS (Amazon Web Services) developer."
• The Books tab of Camaro.com has a nifty, dynamic, ECS-powered search box.
• Brad Dixon is working to add S3 support to the open source Backup Manager tool - "I've begun modifying Backup Manager to use the S3 utility as a upload target. I can now run a backup and have it stored offsite for pennies."

That's about it for this morning. Time to go write another 300 posts.

-- Jeff;

TechnoAg S3 Browser

Alan Baker released the first version of his TechnoAg S3 Browser almost a month ago, just days after the release of S3, but I just found out about it yesterday. This Windows application makes it really easy for you to view, upload, modify, and download the objects in your S3 account.

Here's what it looks like:

After downloading the application (conveniently available in both source and pre-compiled form) you need only insert your AWS Access Key and your AWS Secret Access Key into the configuration file, and then you'll be ready to go.

-- Jeff;

Paying For Web Services -- Easier Than You Think?

Amazon's new S3 service is available on a pay-as-you-go basis. This is commonly called utility computing; it is totally analogous to the way in which you pay for the water, electricity, and natural gas that you use in your home.

With this post I would like to make clear just how easy it is to use this type of web service. I'm writing this because some of the developers that I talk to seem to think that it really has to be harder than it really is, and I want to correct that notion as soon as possible.

To use services like S3 you don't need to call us, you don't need to set up a meeting, you don't need to enter into any negotiating sessions with us, you don't need to start writing custom contracts, and you don't need to send us any money up-front. We designed this service and all of the infrastructure around it to be self-serve, straightforward, and trouble-free.

Let's look at registration, service signup, reporting, and billing in turn.

On the registration side, you start by creating a free Amazon Web Services Developer Account, which you can do here. As part of this process you will be asked to read and signify your acceptance of the license agreement. When you read the agreement, you should also take care to read any of the sections which are pecular to one particular service. After you have signed up, click on the "Your Web Services Account" button to access some important information:

Clicking  "View Access Key Identifiers" to see your Access Key ID. You must pass this parameter to us as part of any request to any of our services. You will also see your Secret Access Key on the same page. You will never be asked to divulge this key, and you should never pass it to us as part of any request. Instead, you will use this key as part of a signing process; this process authenticates your requests so that we know that you (and not some other developer) are in fact making them.

At this point you are a registered AWS developer, and you have agreed to the license, but you are under no financial obligation to us.

In order to sign up for S3, you simply visit the S3 page, and then click "Sign up for Web Service":

From here you confirm your acceptance of the pricing for the service, confirm your credit card and billing address (or enter a new one), and then sign up (this picture shows me signing up for a different service; I had to do that in order to take the screen shot because I had signed up for S3 already):

Now comes the fun part, building your application using our web services APIs. As soon as you start to make calls to the services you've signed up for, we'll be tracking your usage.

Ok, so what about reporting, you ask? You can view your usage at any time and on a per-service basis using the "Usage Report" option:

After you select this option you will be prompted for some additional information. You can download the report in XML or CSV (Comma-separated variable format), and the date range.

I chose to download the data in CVS form, and then stuffed it into an Excel worksheet. Here's what it looks like:

We also put together a billing statement each month; you'll get an email reminder when we do this. Here's what the statement looks like:

And that's about it! This is what utility computing is supposed to be all about: sign up, use it, and pay, without having to go through a lot of hassles or complications to get started or to pay for the service.

-- Jeff;

Monday S3 Roundup

Developer activity around S3 continues at a rapid pace, and I already have enough material for another S3 roundup. Here goes:

• Doug Kaye, who interviewed me several years ago for IT Conversations, asks about the use of S3 as a CDN, or content delivery network. Great idea, and I will pass this post along to the S3 development team.
• Olivier Gutknecht has released the first version of an S3 browser for the Mac.
• Dave Winer makes it clear that we need to do a lot of idea sharing in order to realize the full potential of S3. There's plenty of sharing happening already on the S3 discussion forum, but there's room for more.
• Andre wants to use S3 for automated backup, and is looking for the tools needed to make this a reality.
• Jarod is trying to build an S3 application, but is having some trouble with UTF.
• There's a new release of s3DAV; it is now up to version 0.4.
• Oh yeah, and there's a Perl module, Net::Amazon::S3.

Whew, that's a lot of stuff.

-- Jeff;

Saturday Morning S3 Roundup

A few minutes of surfing, a scan of the S3 forum, a couple of Technorati and PubSub alerts, and a del.icio.us S3 tag brought all of the following cool S3 activity to my attention:

• Dave Winer wants to have a Bay Area S3 conference. That's an interesting idea, and I'll see what we can do to support it. Dave is already storing data in S3; check out the newer images on Scripting.com and you'll see that they are served up by S3.
• The Mission Data blog reports on their effort to move their data over to S3, using some modification to the Ruby samples to support streaming of data, as reported here.
• Dominic Da Silva released version 1.0 of his #Sh3ll ("Sharp Shell"), and (cleverly enough) made the downloaded bits available in S3. More details in this forum post. He's also released version 1.0 of jSh3ll, a Java version of this application.
• Matt Croydon talks about using S3 to back up Flickr photos and says "After uploading 160 or so photos to Amazon, I owe them about a penny."
• Les Orchard thought it would be cool if Amazon offered RSS feeds of bucket contents.

And that's all I've got on this fine Saturday morning!

-- Jeff;

Using S3 to Store Media Files

Adrian Holovaty is now using Amazon's S3 to store files for his Chicago Crime site. In his recent blog post, "How I'm using Amazon's S3 to store media files", he describes the entire process of moving his files over. Adrian notes that it took him less than one hour to move his files over and to modify his code.

One of the commenters to the blog was surprised to find out that the objects in S3 are URL-addressible. This is absolutely the case, and is one of the very cool aspects of S3.

In fact, the image at right is stored within S3; I put it there using the S3 Perl / Curl sample.

You can verify this by right-clicking on the  image and  inspecting its properties. The image's URL is http://s3.amazonaws.com/aws_blog/images/chicago_crime.png . In this case, aws_blog identifies the S3 bucket, and images/chicago_crime.png identifies the object (the image) within the bucket.

When I stored the image into S3 I set the content-type to "image/png" so that the browser would know that it was in fact an image. When S3 processes the HTTP GET from the browser, it returns the content type (along with any other S3 metadata attached to the object) using HTTP headers. Here's what it returned for the image:

HTTP/1.1 200 OK
x-amz-id-2: 0K7SmOorLULRBZUjPRDyQruOsnGkOYwcwIMz5nezKFjMVWsDgi099lZvY4qUCOsG
x-amz-request-id: 52F4A02FC0C51299
Date: Fri, 07 Apr 2006 18:12:23 GMT
Last-Modified: Fri, 07 Apr 2006 18:05:29 GMT
ETag: "4f81c616c6e0e65dc6c957681941a3f4"
Content-Type: image/png
Content-Length: 75860
Connection: keep-alive
Server: AmazonS3

As you can see, it would be easy to use S3's metadata facility to store exta information about each object. Once information is stored in S3, you can get it back without retrieving the entire object using a simple and efficient HTTP HEAD request.

-- Jeff;

More S3 Crunchy Goodness

I just found out about a few more cool S3 wrappers:

• S3Ajax - An Ajax Wrapper for S3
• S33r - S3 + Rails
• S3Dav - a WebDAV server for S3

-- Jeff;

S3 Roundup

There's a lot going on with Amazon's new S3 service; here's a quick roundup of some of what I've seen in the last couple of days:

• Over at InfoWorld, Jon Udell proposes S3 as a piece of politically neutral architecture.
• Mitch Garnaat has created BitBucket, a set of Python scripts for S3 access. There's even more info in the S3 discussion board thread.
• Dominic Da Silva has created jSh3ll, a Java-based command-line shell.
• Mark Hellewell wrote a similar application in Python.
• Ben Oram is now beta-testing S3 Explorer, a drag-and-drop S3 access tool for Windows.
• There's also a very clean and simple PHP wrapper for S3.

Cool stuff!

-- Jeff;

Price Comparison Add-On for Amazon Product Feeds

Users of MrRat's venerable Amazon Products Feed can now add price comparison features using the services at Price-Compare.net .

-- Jeff;

Free ISBN Web Service

From The Programmable Web comes word of ISBNdb.com, a large database of  books indexed by ISBN. Of interest to developers is the fact that there is a complete (and free) REST API. Items in the site are organized (and accessible) via book, subject, category, author, and publisher index.

Several (ok, many) ECS developers have asked me for similar services over the past year or two and it is great to see this new and powerful service in action. If you build something cool with this (alone or in conjunction with ECS) please drop me a note so that I can give it consideration for this blog.

-- Jeff;

AWS with PHP and Ajax

Ajax Magazine is running a multi-part series on the use of PHP and Ajax to create an AWS-powered application:

• Part 1 is already up. It creates a basic site with dynamic search functionality.
• Part 2 adds price  comparison across the 6 Amazon sites supported by ECS.