SQS: Super Queue Service

There were some questions/concerns regarding the Amazon's Simple Queue Service: What can it be used for? What has it been designed for? What types of applications can use this type of service ? and so forth.

Thomas Howe wrote an excellent post about Amazon SQS. His innovative application idea triggered me to write this post.

In this post, I will try to explain the power and potential of Amazon SQS. What makes it a "Super Queue Service" :) Hopefully, after reading this post, your "thought liquids" might be spurred enough that you will use the comments section to give me more ideas. Leverage the Power of Community!

First some theory and then we go into more juicy stuff
The key to building scalable systems is to have loosely coupled components in our architecture, components that do not have strong dependencies between each other, so that if one component were to die (fail), sleep (not respond) or remain busy (slow to respond) for some reason, the other components in the system continue to work seamlessly as if nothing has happened.

The key to building loosely coupled components in our architecture is Messaging Queues. Why? Because if a queue/buffer is used to "wire" any two components together, it can now abstract out and hide the load/demand that’s coming in or going out from components. So if one component dies or becomes unavailable, it will buffer the messages and get them processed when the component comes back up.

Now think of Amazon SQS as a "transient buffer" between producers and consumers. On one side, there are these inhumane monstrous producers that are continuously shoving-in messages/requests/jobs and on the other side there are ravenous consumers eagerly waiting and ready to consume those jobs.  Let’s analyze: What if producers get more inhumane or more producers join the gang? No problem! The Queue will expand on-demand and keep on collecting/buffering those messages. What if consumers/processors die or become unavailable? No Problem! Same answer. This is called Asynchronous Application Integration which basically means queuing leads to asynchrony which ultimately leads to a highly-scalable system.

If you are smart (like Amazonians ;)), you will probably want to use your consumers wisely to its full utilization and save some money because consumers are expensive resources. For example, Producers keep on producing messages continuously during the day and  Consumer "wakes up" for 2 hours a day, consumes/processes the messages/jobs and goes back to sleep. Or conversely, if you care for faster processing, simply add more hungry consumers. This is called Extensible/Scalable Integration which gives you the flexibility of adding/removing nodes because they are loosely coupled.

Now think of Amazon SQS as an "Unified Interface" between two components. Imagine if all the applications in the world had just two interfaces: send, receive and delete. And all the applications in the world interacts with other applications in these two interfaces (think big!, common) Because you can shove-in messages in any self-describing formats like XML/JSON, you only need to worry about is the "language" (Message format and What the message elements mean). This glues two different types of application frameworks that are running completely independent to one-another at their own will. This is called heterogeneous data integration.

Now think of Amazon SQS as a "Service Bus". Queuing allows and encourages applications to become truly service-oriented. You will be automatically making the right architecture decision of making your applications service-oriented and independent to others. Moreover, imagine if consumers' can be physically located anywhere in the world. Because queues are represented by a simple URI, Consumers can find queues dynamically and process the messages, independent of where they are located. For e.g. two consumers processing messages from US during US day time and other two consumers processing the same messages from Japan during Japan day time.

Now think of Amazon SQS as a "Workflow enabler". Imagine if all the workflow systems like WWF/WCF, jBPM, openWFE, BizTalk, Agila have Amazon SQS plug-ins. They could simply “talk” to one another. In fact, smart developers have already built plug-in for OpenWFru and Transport channel for Microsoft’s WCF

So I just thought of few ideas:
Media processing pipelines can be enabled and automated by queues.

• Document processing pipelines that format documents, perform ETL (extract, transform, load) operations and save it in different formats.
• Image processing pipelines that crop, size adjust brightness/contrast/sharpness, create thumbnails images etc.
• Video/Audio Processing pipelines that append header/footer advertisement frames in an existing video, transcode it in different formats, increase or decrease quality/size etc.
  
  Store all big documents/images/video on Amazon S3 and relay them via Amazon SQS.

Two or more disparate applications, from different parts of the world, can talk to each other:

• Billing system that’s being built and hosted in the US can talk to order processing system which is built and maintained in Africa in a secure/reliable way.
• Central Accounts Payable system in Chicago can now take requests from any department in any city
• Implement a website that runs 9AM-5PM PST and processes customers orders in china in china day time.

Get more juice out of your bureaucratic applications (applications that just take more time for processing or are just slow in responding, causing a bottleneck)

• Integrate Banking systems
• Integrate with Large computation/rendering systems
• Enable and connect various supply chain components across companies (customers, partners, distributors, suppliers)

Use the comments section to brainstorm more SQS ideas.

-- Jinesh

Comments

[Balaji Sowmyanarayanan]

SQS is neat components of AWS yet not popular enough to match its potential. In terms of fetching returns for Amazon, in the long run, SQS will be the most profitable!

Given that SQS is 'expensive', the first set of applications that will benefit the power of SQS will be in bridging online payment islands. For example, SQS can be used to to bridge payment transactions between SecondLife and Paypal. A real world product can be featured in the virtual secondlife, once the virtual product is 'bought', a message is sent to a secure queue(SQS). A client program( possibly assisted by a human) consumes the message in the queue and effects the actual real world purchase.

A bizarre application of SQS will be a SMS messaging based command line interface to a web application. Or perhaps a full EC2 instance. Thus a tiny-weeny inexpensive phone in the middle of nowhere, will have a amplified computing back-end. If SQS were 'inexpensive', SQS+SMS based back-end coupling could be put to use in making online games playable through SMS. Once proved successful in the gaming, it will set the stage for prime time business applications.

PS: My fav is Mturk though. Yet another less popular yet more potential component.

Cheers.
-Balaji S.

[oli]

unfortunately SQS is quite expensive...

[Bryan Wilhite]

One Queue Service that should really get your point across is a plugin for popular Blog engines (like WordPress) that allows posts to be submitted to the queue and then "batch" loaded later.

Without such a tool, losers have to send their Blog user name and password in clear text from tools like Qumanana or Word 2007 directly to some XML-RPC stuff. The Amazon.com solution would be more secure since this batch process takes place on the server and kicked off from INSIDE the admin console of the Blog tool. Get to it!