Amazon Web Services Blog

Start Crunching - Create Mashups using MapCruncher and Render on S3

Microsoft Research developed an innovative tool called MapCruncher for Microsoft Virtual Earth that takes any map-based image (JPG or PDF format) and overlays that image onto Microsoft Virtual Earth to generate requisite tiles. These tiles can then be used to display the map in Virtual Earth / Google Maps fashion (zoom in/out, pan, move etc). More info available here. Researchers recently added a new feature called "Render to S3", so now all the tiles/images/javascript that gets generated off mapcruncher can be directly uploaded to Amazon S3 right from the tool and your mashup is instantly available to the world - right hot out of S3.

Pretty neat!. So now you can not only create map-based mashups in minutes but also host them.

I could not wait to try this out. So I took the camping map from my last trip (Lake Rooselvelt and Grand Coulee Dam) and decided to crunch the NPS-published high-resolution map that shows all the campgrounds and boat launch information and overlay it on Microsoft Virtual Earth using the tool. Then I created 6 render points and let the mapcruncher algorithm do all the number crunching to figure out how may tiles I need, what tiles I need etc. I locked my images with Microsoft Virtual Earth and saved my mashup (lake.yum). It estimated that I would need 3990 tiles around 340MB of data. Now the best part, I clicked on radio button "Render to S3", it asked me for my AWS credentials and boom! it started uploading all the images/tiles to S3 with just one click of a button. After a minute or so, I got a brand new mashup of Lake Roosevelt and Coulee Dam superimposed on Microsoft Virtual Earth and hosted on Amazon S3.

Obviously, you can see that I am not a expert mapcruncher and the map got skewed to adjust with resolution. Part of the reason was that I only used 6 render points (minimum) instead of creating more for enchanced accuracy. The tool also has features to adjust transparency, which I did not get a chance to play with.

However, the thing to note here is Microsoft Researchers were able to integrate highly-scalable storage infrastructure to their application - mashup in minutes - using the power of Web Services!

Next, I am going to create my neighborhood community mashup where I will overlay all occupant information on Virtual Earth so my neighbors can find other neighbors :)

Happy Crunching!

-- Jinesh

Using AWS in or With Second Life? Exhibit in our Marketplace

One of the larger and more prominent features of the Amazon Developers Islands is a long, covered marketplace.

If you have ever been to South or Central America, you will see something familiar. Imagine a busy, chaotic and eclectic space, with goods of all sorts on exhibit, and lots of noise -- a full-frontal sensory assault (well, except for the smell). That's what I aim to replicate here.

If you use any of the Amazon Web Services and you have built something that runs in or with Second Life, I want to hear from you. Ditto, if you have built a "first life" application but know your way around Second Life.

For a limited time (until the space fills up or we run out of prims), you can create an exhibit in the markeplace. We've never done anything quite like this before so the rules could change, but here are the basics. First, reserve your space:

1. Find a place in the market. Plan to occupy a space no larger than 5 meters wide,  2 meters deep, and 3 meters tall. Create a landmark at your space.
2. Create a box the size and shape of your desired installation, and put your logo on it as a texture. Feel free to label it "coming soon."
3. Put the landmark in the box (open up your inventory, find the landmark, and drop it onto the box).
4. Put a notecard in the box with full information about the object's ownership and a short description of the exhibit.
5. Locate Second Life resident Amazon Link and send the box to her (she's the official owner of the islands).

Next, take some time and create your actual exibit, taking care to fit it into the shape that you reserved. I will hold the spaces for 2 weeks; this should give you enough time to make something cool.

Ideally, your exhibit will be thematically compatible with the island and with the marketplace. Don't go overboard with ugly rotating things, don't spam passers-by with notecards, and keep the scripting light. Represent your products accurately, and give the user a way to visit your own location. Be creative.

Use no more than 30 prims for the entire exhibit. Link it all together, again include the notecard and the landmark, and send it to Amazon Link.

I'm really looking forward to this new and exciting way to interact with developers, and hope that you would like to come along as well. I am open to other promotional ideas; just let me know.

Again, the rules could change as the marketplace matures, but for now this is first-come first-serve for relevant applications, with the proviso that Amazon Web Services has the final say in what we do or don't show on our land. If you have any questions send me (Jeffronius Batra) an IM.

-- Jeffronius;

Amazon Web Services Developer Chat in Second Life

Many years ago we used to hold weekly developer chats with the AWS community. These chats were fun and informative, even though we used a very limited text-based client.

Now its time to try something new and more interactive!

We will hold our first-ever Amazon Web Services Developer Chat in Second Life on Wednesday, May 30th at 10 AM SLT (Second Life Time - the same as PST). Schedules permitting, I am planning to conduct these meetings every week.

The meeting will take place in the Conference Center (pictured at right) on the new Amazon Developers Islands. If you are already a registered Second Life user, search the map for Developers and then choose the island labeled "Amazon Developers 1."

If you are not a Second Life user, come on in and join the fun! Start out by registering. Create your avatar,  learn to navigate, move around and to chat, and you'll be all set. If you have trouble finding the island, send me an IM and I will reply with a teleport request.

Please feel free to visit and to hang out on the islands. Take a ride on the tour boat, and explore all the nooks and crannies.

While you are logged in, be sure to join the Amazon Developer Island Guest group so that you can receive notification of future events.

We are currently planning to hold demos, presentations, talks, and regular chats there. Your ideas are certainly welcome. We've even got a marketplace where AWS developers can post their AWS-powered applications (more on that in a little bit).

Also, please do feel free to look me up in-world, where I go by Jeffronius Batra.

See you tomorrow!

Goplan goes AWS

Goplan - An online collaboration and project management tool - is a full fledged Ruby on Rails Application that is based on Amazon EC2 for hosting and Amazon S3 for storage. It took me less than 1 minute to get started with the app. No ugly big forms to fill out nor answer any difficult security questions like "What is your great grandfather's middle name?" The best part is signup form has 4 simple fields to fill.

Felix interviewed Goplan's developer. This blog post certainly shares some interesting insights into Goplan.org and Amazon EC2/S3.

He notes:

Tiago: The switch to EC2 came when we needed to scale the application. Not only was it cheaper to run it on EC2, it was also easier to scale during the launch. The ability to launch more instances on demand was very useful when we were featured on TechCrunch, digg, etc. Using S3 was a natural consequence of our move to EC2.

Most interesting part of the blogpost was the knowledge and experience the developer shares. Things like which S3 ruby library was used, WAL approach to data loss within PostGreSQL database, Munin to monitor the DB instances in case of failover and the best part on how a database can scale on EC2:

several ways: by changing that behaviour for read-only queries (it wouldn’t even be a day’s worth of work), by using a middleware solution (such as SQL Relay) or by addressing that at the database level (using Master-Master replication). MySQL has internal replication solutions, PostgreSQL has PGCluster (I have some experience using it) and there is always the option of using more complex (and expensive) solutions such as Oracle databases.

This shows how smart programmers are finding cool and nifty ways to architect applications and move on to solve more interesting problems.

-- Jinesh

update: Felix is the author of blog/interview

Video Bandwidth Protector

Bill Myers wrote to tell me about his Video Bandwidth Protector for Amazon S3. As Bill says:

I developed this program to make it easy to store and display videos and Camtasia movies at Amazon S3 storage, while at the same time, making it very difficult for others to steal bandwidth via unauthorized display of those videos.

Basically, this system encapsulates a video file so that it can only be played when hosted within a web page served up by a particular domain. The script uses "security via obscurity" to protect the underlying URL from prying eyes.

Bill has put together a nice tutorial video for the protector.

The system is now available free of charge, but you are certainly welcome to leave a donation to support this project.

Very cool!

-- Jeff;

 

 

Webmail's Bill Boebel talks about Amazon S3

Bill Boebel, CTO of Webmail.us and a heavy user of Amazon S3, recently spoke at the ISPCON conference.

Bill was kind enough to post the slides from his talk on this blog post. You can also read a review of Bill's talk here.

Here's my 2 minute overview of the slides (that's a direct link to the PowerPoint deck):

Founded in 1999, Webmail.us has 54 employees and over 47,000 customers. Their new custom backup solution uses Amazon S3, Amazon SQS, and Amazon EC2 to backup over 476,000 mailboxes. Because their new solution allows for incremental backups, they were able to reduce their costs from $180K per year down to just $30K. Using S3 also gave them room to grow their business, allowing them to scale by doing more backups in parallel. They use a fleet of EC2 instances, directed by a set of job requests stored in SQS queues, to delete obsolete data from S3. Bill also provides a set of best practices and recommendations for the use of S3, along with his list of desirable features for future release.

-- Jeff;

Junk I Want - Easy and Flexible Wishlist Embedding

Tom Sprows recently wrote to tell me about his new tool JunkIWant. This tool simplifies the process of embedding an Amazon Wishlist in a blog page, a MySpace page, or even a regular web site.

You have a choice of 4 different styles, along with control of colors. You can specify your wishlist by id or by email address. Once you have made your selection you will be provided with the HTML code needed to embed the JunkIWant applet on the site of your choice.

I asked Tom to tell me about some of the interesting challenges he faced while building his site. He noted that he created a proxy application so that he could call the Amazon E-Commerce Service, as well as a caching mechanism to ensure that he stayed within the Terms of Service. Tom notes that "The Amazon ESC was pretty easy to work with.  The tough part was providing the configuration and interaction with the Flash Applet."

We'll look forward to seeing more cool and useful applications from Tom; he told me that "I had alot of fun with this project and will continue to look at Amazon web services for future business opportunities."

-- Jeff;

New Simple Queue Service Release

The Amazon SQS team recently released a new version of their service. Per the SQS release notes, the new features include:

• New function to get the approximate number of messages in a queue.
• Ability to change the visibility timeout of a message.
• Ability to delete a queue even if it is not empty.
• More flexibility in queue names, including the ability to use the hyphen and underscore characters as part of the name.
• Ability to send messages up to 256KB in length via the HTTP query interface (the former limit was 8KB).

The SQS documentation and the WSDL have been updated to reflect these changes.

-- Jeff;

Phozi - Customized Photos With Amazon S3 Inside

The developer of Phozi sent me an email today, asking for some blog coverage, and I am more than happy to oblige.

Starting with an uploaded photo, a photo from a webcam, or from a mobile phone, Phozi makes it really easy to add all sorts of decorations to the photo and to post it to a social networking site like MySpace, Facebook, Bebo, Xanga, or Hi5.

The site uses Amazon S3 for all image storage and publishing, which ensures that the photos are available regardless of the status of Phizo's own server.

You can see some of what's possible from the image below:

You can also read more about Phozi, and about this entire market space, in the TechCrunch article.

-- Jeff;

Colorado Trip: Developers Wanted

Our Wiki experiment is now a standard scheduling tool! I am available for meetings in Colorado On June 25 and 26, and am willing to drive to almost any destination along the Front Range. If you would like to schedule some time, simply edit the Wiki page at http://s3.amazonaws.com/s3wiki/wiki/MikeCulverDenverJune2007. I am happy to visit with companies, individual developers, bloggers, and other folks to talk about the Amazon Web Services. Please edit the Wiki and send me a confirming email.

I'm willing to drive to almost any reasonable destination along the Front Range (or even in the hills).

-- Mike

Are You In the Dallas Area?

I'm visiting Dallas next week to present to the Dallas ASP.NET Users Group on Tuesday, May 22. If you are interested in meeting with me, having me present to your users group, etc., please email me to set something up. Am open on Wednesday (all day) as well as Thursday morning.

--Mike Culver

An Interesting Newsletter Article

A month or so ago, I attended a SOA Conference and Business Process Management Conference (there were multiple conferences in one venue), put on by the BrainStorm Group in Chicago. Lots of serious enterprise architect types attended the conference, where they were able to share their experience and learnings with others, as well as hear about the state of the art in the industry. I presented--of course--on Amazon Web Services.

Mike Rosen was one of the conference co-chairs. In addition to being co-chair for Brainstorm's SOA conference, he is also Director of Enterprise Architecture for Cutter Consortium, where he writes a weekly email advisor. Mike forwarded me a recent article that he wrote on Amazon Web Services. (By the way, Mike used to be Chief Enterprise Architect at IONA Technologies.)

It's always interesting to read what others say about a particular product or technology. You'll need to sign up for the newsletter in order to read the complete article; however in part Mike wrote "it's hard to imagine building an equivalently redundant, scalable, and available storage and compute platform at anywhere near the low cost of using Amazon".

He continued by saying what we're hearing over and over: "The real boon is for small and medium businesses that now have access to a world-class data center for less than the cost of a laptop. Imagine how this changes the cost of entry for new and startup businesses. I expect to see some dramatic examples of this in the near future. In the meantime, think about how your business might be able to make use of these services. Your competitor probably will".

The laptop analogy in interesting. Are we on the cusp of desert-island business models? Where do I sign up to beta-test running my business from a sailboat in Raratonga? :)

-- Mike

SQS: Super Queue Service

There were some questions/concerns regarding the Amazon's Simple Queue Service: What can it be used for? What has it been designed for? What types of applications can use this type of service ? and so forth.

Thomas Howe wrote an excellent post about Amazon SQS. His innovative application idea triggered me to write this post.

In this post, I will try to explain the power and potential of Amazon SQS. What makes it a "Super Queue Service" :) Hopefully, after reading this post, your "thought liquids" might be spurred enough that you will use the comments section to give me more ideas. Leverage the Power of Community!

First some theory and then we go into more juicy stuff
The key to building scalable systems is to have loosely coupled components in our architecture, components that do not have strong dependencies between each other, so that if one component were to die (fail), sleep (not respond) or remain busy (slow to respond) for some reason, the other components in the system continue to work seamlessly as if nothing has happened.

The key to building loosely coupled components in our architecture is Messaging Queues. Why? Because if a queue/buffer is used to "wire" any two components together, it can now abstract out and hide the load/demand that’s coming in or going out from components. So if one component dies or becomes unavailable, it will buffer the messages and get them processed when the component comes back up.

Now think of Amazon SQS as a "transient buffer" between producers and consumers. On one side, there are these inhumane monstrous producers that are continuously shoving-in messages/requests/jobs and on the other side there are ravenous consumers eagerly waiting and ready to consume those jobs.  Let’s analyze: What if producers get more inhumane or more producers join the gang? No problem! The Queue will expand on-demand and keep on collecting/buffering those messages. What if consumers/processors die or become unavailable? No Problem! Same answer. This is called Asynchronous Application Integration which basically means queuing leads to asynchrony which ultimately leads to a highly-scalable system.

If you are smart (like Amazonians ;)), you will probably want to use your consumers wisely to its full utilization and save some money because consumers are expensive resources. For example, Producers keep on producing messages continuously during the day and  Consumer "wakes up" for 2 hours a day, consumes/processes the messages/jobs and goes back to sleep. Or conversely, if you care for faster processing, simply add more hungry consumers. This is called Extensible/Scalable Integration which gives you the flexibility of adding/removing nodes because they are loosely coupled.

Now think of Amazon SQS as an "Unified Interface" between two components. Imagine if all the applications in the world had just two interfaces: send, receive and delete. And all the applications in the world interacts with other applications in these two interfaces (think big!, common) Because you can shove-in messages in any self-describing formats like XML/JSON, you only need to worry about is the "language" (Message format and What the message elements mean). This glues two different types of application frameworks that are running completely independent to one-another at their own will. This is called heterogeneous data integration.

Now think of Amazon SQS as a "Service Bus". Queuing allows and encourages applications to become truly service-oriented. You will be automatically making the right architecture decision of making your applications service-oriented and independent to others. Moreover, imagine if consumers' can be physically located anywhere in the world. Because queues are represented by a simple URI, Consumers can find queues dynamically and process the messages, independent of where they are located. For e.g. two consumers processing messages from US during US day time and other two consumers processing the same messages from Japan during Japan day time.

Now think of Amazon SQS as a "Workflow enabler". Imagine if all the workflow systems like WWF/WCF, jBPM, openWFE, BizTalk, Agila have Amazon SQS plug-ins. They could simply “talk” to one another. In fact, smart developers have already built plug-in for OpenWFru and Transport channel for Microsoft’s WCF

So I just thought of few ideas:
Media processing pipelines can be enabled and automated by queues.

• Document processing pipelines that format documents, perform ETL (extract, transform, load) operations and save it in different formats.
• Image processing pipelines that crop, size adjust brightness/contrast/sharpness, create thumbnails images etc.
• Video/Audio Processing pipelines that append header/footer advertisement frames in an existing video, transcode it in different formats, increase or decrease quality/size etc.
  
  Store all big documents/images/video on Amazon S3 and relay them via Amazon SQS.

Two or more disparate applications, from different parts of the world, can talk to each other:

• Billing system that’s being built and hosted in the US can talk to order processing system which is built and maintained in Africa in a secure/reliable way.
• Central Accounts Payable system in Chicago can now take requests from any department in any city
• Implement a website that runs 9AM-5PM PST and processes customers orders in china in china day time.

Get more juice out of your bureaucratic applications (applications that just take more time for processing or are just slow in responding, causing a bottleneck)

• Integrate Banking systems
• Integrate with Large computation/rendering systems
• Enable and connect various supply chain components across companies (customers, partners, distributors, suppliers)

Use the comments section to brainstorm more SQS ideas.

-- Jinesh

Meeting for Amazon Associates in Munich

On Monday, May 14th I will be at Amazon's Munich office for meetings with developers, bloggers, and the press. At 2 PM that day I will be conducting a special meeting for Amazon Associates that would like to learn more about the Amazon Web Services.

The presentation will include the following elements:

• AWS overview, with a focus on the Amazon E-Commerce Service.
• Examples of some tools that make it easy to build an associate-enabled site with ECS.
• Demonstrations of sites built by some existing associates, including "web 2.0" and social media sites.
• A review of some AWS activity that's taking place in Second Life.

If you are interested and would like to attend, please send an email to astore@amazon.de . Space is limited, so do this sooner rather than later.

Hope to see you there.

-- Jeff;

PS - If you are a DE-based developer and you have some interesting ECS-powered associates sites that would be suitable for a demo, please feel free to send me some info. I am always looking for new applications to add to my repertoire.

Search Engine Packed as an AMI?

It never hurts to try to wish a product into existence...

I received an email from an EC2 user asking me about search tools. This user runs a high traffic site on an array of EC2 instances, and is in need of a search solution. He knew that he could buy a search appliance, but this didn't fit with his company's model. As he told me:

"we don't want to do anything that involves us owning and operating a server...since we're big believers in web services."

After thinking about this for a while, I believe that one really cool solution would involve a search engine installed into an EC2 AMI (Amazon Machine Image), perhaps made available for use on a by-the-hour basis. This hypothetical AMI would incorporate all of the usual components: a crawler, data storage, and a query page for access to the actual search engine. There are bonus points for APIs for inserting and retrieving data, of course.

Perhaps the crawler runs once every 24 hours and then generates some indexed data structures which it stores in S3, where they are picked up by the engine and loaded into the instance's RAM for fast processing. Once again, I'll offer bonus points if spinning up multiple instances of the crawler makes the entire crawling and indexing process run faster.

To top it all off, the query page would be customizable and skinnable, so that this could be plugged into an existing site in a seamless fashion.

If you are doing something like this or have even thought about doing something similar, I'd like to hear from you. If you would pay to use it, same deal. Post some comments and let's see what happens.

-- Jeff;

Friday Links

've been on the road a lot this past month, including trips to Orlando, Barcelona, London, and Las Vegas.

Duringf that time I've met a lot of interesting people and have accumulated a whole bunch of relevant links:

• eWeek just published a new article, Enomalism Helps Manage Virtual Machines. Read all about their viewpoint on this new web-based management tool for EC2 instances and images.
• O'Reilly Radar just wrote about Jamglue Growth and Scaling. They spent just $1.50 per month on Amazon Web  Services during development, then $15 per month in beta, rising to $600 per month in production.
• Greg Linden wrote about Yahoo Pig, a parallel processing architecture which runs on top of Hadoop. Pig makes it possible for "regular developers" to take advantage of a large number of processors to analyze gigantic data sets. Given that Hadoop already runs on top of EC2 and S3, I suspect that it won't be long before we hear that Pig is running there as well.
• Computerworld had some words to say about MySQL and S3, including a review of Mark Atwood's new MySQL storage engine for S3. Mark has recently posted his MySQL 2007 conference presentation.
• Paul Bissett from WeoGeo has an interesting post about Building a Web 2.0 Mapping Solution. Paul runs the numbers, noting that their hardware budget was about $500K, which was reduced to around $20K with liberal application of EC2 and S3 resources.
• Coolchaser is a MySpace profile editing and styling tool, powered by EC2 and S3. You can read more about this on their blog.
  
• Blingee is another S3-powered MySpace tool, serving up hundreds of thousands of images per day.

And that's about all for today.

-- Jeff;

Who's using Ruby on Rails?

If you are an Amazon EC2 or Amazon S3 aficionado and are using Ruby on Rails, we would like to hear from you. Particularly we are interested to know how you are using Ruby on Rails, your architecture, your EC2 network topology setup, your S3 hacks. Anything that can help the newcomers to ramp up fast. Ruby on Rails cuts the development time by 50%, AWS eliminates the headache by 70%, and your ideas/experience will help the new comers even more. We would love to know how you are using Amazon Web Services and Rails together. Please use the comments section to describe your product and how the Ruby on Rails framework has helped you to reach your goals.

We know about a few that are already using AWS/Ruby on Rails : RightCart, RightScale, 37Signals' BaseCamp, SlideShare, WeoGeo. In fact, Slideshare's Jon Boutelle has a cool presentation about Ruby/Amazon S3 architecture.

In my previous post, I listed various Ruby libraries that have been built to use AWS. If you have new ones or have a new version, let me know and I will add one more precious stone (ruby) in the already-shiny AWS throne. Few that caught my eye were:

• Capazon - a cool capistrano extension to manage Amazon EC2 instances
• CLI to Amazon S3 Service using Ruby - an interesting Command-Line Interface to Amazon S3 in Ruby

-- Jinesh

Who's coming to RailsConf?

RailsConf 2007 - O'Reilly's Rails Conference is one conference that I think you should not miss. The Rails Community is growing faster than we can imagine. Amazon Web Services is going to be there at the conference to answer your Ruby/Rails questions. Check out the following places, if you want to learn more about AWS:

• My session "Auto-Scale" Rails Applications on 5/20 at 1:50PM
• Thorsten's and Gary's session Scale Rails Without Bounds on Amazon EC2, on 5/18 at 11:45AM
• "EC2 + SQS + RoR = SuperComputer" BoF Session
• "Ruby and Amazon S3" BoF Session
• Our Booth (Do stop by and say hi)

-- Jinesh

Keep Me Busy in London

In what is now becoming a standard part of our trip planning process, I am making myself available for meetings in London during the week of May 13th. If you would like to schedule some time, simply edit the Wiki page at http://s3.amazonaws.com/s3wiki/wiki/JeffBarrLondonMay2007 . I am happy to visit with companies, individual developers, bloggers, and so forth to talk about the Amazon Web Services. Please edit the Wiki and send me a confirming email.

I'm willing to hop on the tube for destinations anywhere in the greater London area, but I am not going through the Magic Roundabout.

The Amazon team in Munich owns my schedule for Monday the 14th. I understand that they still have room for a few more tech bloggers and journalists. If you are interested in attending, please send me a note (awseditor@amazon.com) and I will pass it along to them.

-- Jeff;

Come to the Lemonade Stand

I just watched a very enjoyable Motley Fool video about Amazon, where they described the Amazon Web Services as "A lemonade stand that attracts techies." I think that's pretty accurate.

They also used a rather cute graphic to promote AWS, and even talked about Amazon S3 (although they showed a screen shot of the totally unrelated s3.com for some reason).

-- Jeff;

PS - Don't take this post as investment advice or anything of the sort; I am linking to it only because I found it interesting and at least somewhat relevant.