I'm visiting Dallas next week to present to the Dallas ASP.NET Users Group on Tuesday, May 22. If you are interested in meeting with me, having me present to your users group, etc., please email me to set something up. Am open on Wednesday (all day) as well as Thursday morning.
A month or so ago, I attended a SOA Conference and Business Process Management Conference (there were multiple conferences in one venue), put on by the BrainStorm Group in Chicago. Lots of serious enterprise architect types attended the conference, where they were able to share their experience and learnings with others, as well as hear about the state of the art in the industry. I presented--of course--on Amazon Web Services.
Mike Rosen was one of the conference co-chairs. In addition to being co-chair for Brainstorm's SOA conference, he is also Director of Enterprise Architecture for Cutter Consortium, where he writes a weekly email advisor. Mike forwarded me a recent article that he wrote on Amazon Web Services. (By the way, Mike used to be Chief Enterprise Architect at IONA Technologies.)
It's always interesting to read what others say about a particular product or technology. You'll need to sign up for the newsletter in order to read the complete article; however in part Mike wrote "it's hard to imagine building an equivalently redundant, scalable, and available storage and compute platform at anywhere near the low cost of using Amazon".
He continued by saying what we're hearing over and over: "The real boon is for small and medium businesses that now have access to a world-class data center for less than the cost of a laptop. Imagine how this changes the cost of entry for new and startup businesses. I expect to see some dramatic examples of this in the near future. In the meantime, think about how your business might be able to make use of these services. Your competitor probably will".
The laptop analogy in interesting. Are we on the cusp of desert-island business models? Where do I sign up to beta-test running my business from a sailboat in Raratonga? :)
There were some questions/concerns regarding the Amazon's Simple Queue Service: What can it be used for? What has it been designed for? What types of applications can use this type of service ? and so forth.
In this post, I will try to explain the power and potential of Amazon SQS. What makes it a "Super Queue Service" :) Hopefully, after reading this post, your "thought liquids" might be spurred enough that you will use the comments section to give me more ideas. Leverage the Power of Community!
First some theory and then we go into more juicy stuff The key to building scalable systems is to have loosely coupled components in our architecture, components that do not have strong dependencies between each other, so that if one component were to die (fail), sleep (not respond) or remain busy (slow to respond) for some reason, the other components in the system continue to work seamlessly as if nothing has happened.
The key to building loosely coupled components in our architecture is Messaging Queues. Why? Because if a queue/buffer is used to "wire" any two components together, it can now abstract out and hide the load/demand that’s coming in or going out from components. So if one component dies or becomes unavailable, it will buffer the messages and get them processed when the component comes back up.
Now think of Amazon SQS as a "transient buffer" between producers and consumers. On one side, there are these inhumane monstrous producers that are continuously shoving-in messages/requests/jobs and on the other side there are ravenous consumers eagerly waiting and ready to consume those jobs. Let’s analyze: What if producers get more inhumane or more producers join the gang? No problem! The Queue will expand on-demand and keep on collecting/buffering those messages. What if consumers/processors die or become unavailable? No Problem! Same answer. This is called Asynchronous Application Integration which basically means queuing leads to asynchrony which ultimately leads to a highly-scalable system.
If you are smart (like Amazonians ;)), you will probably want to use your consumers wisely to its full utilization and save some money because consumers are expensive resources. For example, Producers keep on producing messages continuously during the day and Consumer "wakes up" for 2 hours a day, consumes/processes the messages/jobs and goes back to sleep. Or conversely, if you care for faster processing, simply add more hungry consumers. This is called Extensible/Scalable Integration which gives you the flexibility of adding/removing nodes because they are loosely coupled.
Now think of Amazon SQS as an "Unified Interface" between two components. Imagine if all the applications in the world had just two interfaces: send,receive and delete. And all the applications in the world interacts with other applications in these two interfaces (think big!, common) Because you can shove-in messages in any self-describing formats like XML/JSON, you only need to worry about is the "language" (Message format and What the message elements mean). This glues two different types of application frameworks that are running completely independent to one-another at their own will. This is called heterogeneous data integration.
Now think of Amazon SQS as a "Service Bus". Queuing allows and encourages applications to become truly service-oriented. You will be automatically making the right architecture decision of making your applications service-oriented and independent to others. Moreover, imagine if consumers' can be physically located anywhere in the world. Because queues are represented by a simple URI, Consumers can find queues dynamically and process the messages, independent of where they are located. For e.g. two consumers processing messages from US during US day time and other two consumers processing the same messages from Japan during Japan day time.
On Monday, May 14th I will be at Amazon's Munich office for meetings with developers, bloggers, and the press. At 2 PM that day I will be conducting a special meeting for Amazon Associates that would like to learn more about the Amazon Web Services.
The presentation will include the following elements:
AWS overview, with a focus on the Amazon E-Commerce Service.
Examples of some tools that make it easy to build an associate-enabled site with ECS.
Demonstrations of sites built by some existing associates, including "web 2.0" and social media sites.
A review of some AWS activity that's taking place in Second Life.
If you are interested and would like to attend, please send an email to [email protected] . Space is limited, so do this sooner rather than later.
Hope to see you there.
PS - If you are a DE-based developer and you have some interesting ECS-powered associates sites that would be suitable for a demo, please feel free to send me some info. I am always looking for new applications to add to my repertoire.
It never hurts to try to wish a product into existence...
I received an email from an EC2 user asking me about search tools. This user runs a high traffic site on an array of EC2 instances, and is in need of a search solution. He knew that he could buy a search appliance, but this didn't fit with his company's model. As he told me:
"we don't want to do anything that involves us owning and operating a
server...since we're big believers in web services."
After thinking about this for a while, I believe that one really cool solution would involve a search engine installed into an EC2 AMI (Amazon Machine Image), perhaps made available for use on a by-the-hour basis. This hypothetical AMI would incorporate all of the usual components: a crawler, data storage, and a query page for access to the actual search engine. There are bonus points for APIs for inserting and retrieving data, of course.
Perhaps the crawler runs once every 24 hours and then generates some indexed data structures which it stores in S3, where they are picked up by the engine and loaded into the instance's RAM for fast processing. Once again, I'll offer bonus points if spinning up multiple instances of the crawler makes the entire crawling and indexing process run faster.
To top it all off, the query page would be customizable and skinnable, so that this could be plugged into an existing site in a seamless fashion.
If you are doing something like this or have even thought about doing something similar, I'd like to hear from you. If you would pay to use it, same deal. Post some comments and let's see what happens.
O'Reilly Radar just wrote about Jamglue Growth and Scaling. They spent just $1.50 per month on Amazon Web Services during development, then $15 per month in beta, rising to $600 per month in production.
Greg Linden wrote about Yahoo Pig, a parallel processing architecture which runs on top of Hadoop. Pig makes it possible for "regular developers" to take advantage of a large number of processors to analyze gigantic data sets. Given that Hadoop already runs on top of EC2 and S3, I suspect that it won't be long before we hear that Pig is running there as well.
Paul Bissett from WeoGeo has an interesting post about Building a Web 2.0 Mapping Solution. Paul runs the numbers, noting that their hardware budget was about $500K, which was reduced to around $20K with liberal application of EC2 and S3 resources.
Coolchaser is a MySpace profile editing and styling tool, powered by EC2 and S3. You can read more about this on their blog.
Blingee is another S3-powered MySpace tool, serving up hundreds of thousands of images per day.
If you are an Amazon EC2 or Amazon S3aficionado
and are using Ruby on Rails, we would like to hear from you. Particularly we are interested to know how you are using Ruby on Rails, your architecture, your EC2 network topology setup, your S3 hacks. Anything that can help the newcomers to ramp up fast. Ruby on Rails cuts the development time by 50%, AWS eliminates the headache by 70%, and your ideas/experience will help the new comers even more. We would love to know
how you are using Amazon Web Services and Rails together. Please use
the comments section to describe your product and how the Ruby on Rails framework has helped you to reach your goals.
In my previous post, I listed various Ruby libraries that have been built to use AWS. If you have new ones or have a new version, let me know and I will add one more precious stone (ruby) in the already-shiny AWS throne. Few that caught my eye were:
Capazon - a cool capistrano extension to manage Amazon EC2 instances
RailsConf 2007 - O'Reilly's Rails Conference is one conference that I think you should not miss. The Rails Community
is growing faster than we can imagine. Amazon Web Services is going to be
there at the conference to answer your Ruby/Rails questions. Check out
the following places, if you want to learn more about AWS:
In what is now becoming a standard part of our trip planning process, I am making myself available for meetings in London during the week of May 13th. If you would like to schedule some time, simply edit the Wiki page at http://s3.amazonaws.com/s3wiki/wiki/JeffBarrLondonMay2007 . I am happy to visit with companies, individual developers, bloggers, and so forth to talk about the Amazon Web Services. Please edit the Wiki and send me a confirming email.
I'm willing to hop on the tube for destinations anywhere in the greater London area, but I am not going through the Magic Roundabout.
The Amazon team in Munich owns my schedule for Monday the 14th. I understand that they still have room for a few more tech bloggers and journalists. If you are interested in attending, please send me a note ([email protected]) and I will pass it along to them.