On Using AWS in Loosely-Coupled SOA

Jay Ridgeway from ShareThis.com co-presented with me at ETech recently. We were telling a packed room about Amazon SimpleDB, and Jay was specifically telling folks about how ShareThis implemented this Web service.

You should check out their site--it's a great way to share content and allow others to drive traffic to your site. I found that it was extremely easy to integrate one of their buttons in my own site.

Jay has Services Oriented Architecture nailed, as you'll see in the points below. I'm posting this blog entry because I believe that ShareThis.com represents a best practice for others who strive to build a durable architecture using Amazon Web Services. The basic approach to a robust system is to architect with the assumption that everything fails.

• Architecturally he insists on at least 2 instances of every service and does not allow local resource sharing. Jay points out that for a small startup redundancy such as this seems like overkill. On the other hand, I didn't hear a single war story from Jay that involved the use of the word "pager" or the phrase "2 A.M. on a Sunday morning". And it's much easier to grow into a well-designed architecture than it is to fix an architecture that doesn't match the needs of a rapidly-growing organization. Come to think of it, that's one of the core value propositions of Amazon Web Services as a platform.
• AWS services are not accessed directly by the application tier. There are intermediate layers that massage those services into application terms. Jay says “We store files to S3 like everyone else, however there are a lot of other features that need to be integrated into an application storage system such as thumbnail generation, triggers, filters, object introspection and general accounting. As a result, the ShareThis storage service is abstracted from the underlying infrastructural parts. This allows us to present a service component tailored to the needs of the business.”
• Amazon EC2 is used for nearly all services except mail.
• Amazon S3 is used for storage of shared objects, log files, database dumps etc.
• Amazon SimpleDB is in place for publisher reporting, and also for object attribute search and general list management, i.e. white/black lists for various communication channels.
• Amazon Simple Queue Service is the glue that holds it all together. This is by far the most important service, and Jay is surprised that he doesn't hear more about it from the development community. He uses it for job control, synchronization and general impedance matching. I agree with Jay: I frequently refer to Amazon SQS as an impedance transformer, or sometimes as a "transmorgifier", because it eliminates dependencies on the other component in a loosely-coupled system. And in a true loosely-coupled system you never know what the other end point looks like, or if it is even in operation at any given moment.

A more in-depth paper on ShareThis will be available on the Amazon Web Services Resource Center in the near future.

-- Mike

Comments

[Niraj J]

On "Amazon S3 is used for storage of shared objects, log files, database dumps etc."

Are you using this directly from the application to write to S3 OR is it as a backup mechanism.

I would expect that writing to log files at runtime on S3 over remote connection could be a performance issue.

Any comments

[Jay Ridgeway]

The S3 writes are used for backups. I like to think of the EC2 disk as a buffer. :)