Cloudbursting - Hybrid Application Hosting

I get to meet with lots of developers and system architects as part of my job. Talking to them about cloud computing and about the Amazon Web Services is both challenging and rewarding. Cloud computing as a concept is still relatively new. When I explain what it is and what it enables, I can almost literally see the light bulbs lighting up in people's heads as they understand cloud computing and our services, and what they can do with them.

A typical audience contains a nice mix of wild-eyed enthusiasts and more conservative skeptics. The enthusiasts are ready to jump in to cloud computing with both feet, and start to make plans to move corporate assets and processes to the cloud as soon as possible. The conservative folks can appreciate the benefits of cloud computing, but would prefer to take a more careful and measured approach. When the enthusiasts and the skeptics are part of the same organization, they argue back and forth and often come up with an interesting hybrid approach.

The details vary, but a pattern is starting to emerge. The conservative side advocates keeping core business processes inside of the firewall. The enthusiasts want to run on the cloud. They argue back and forth for a while, and eventually settle on a really nice hybrid solution. In a nutshell, they plan to run the steady state business processing on existing systems, and then use the cloud for periodic or overflow processing.

After watching (sometimes in real time in the course of a meeting) this negotiation and ultimate compromise take place time and time again in the last few months, I decided to invent a new word to describe what they are doing. I could have come up with some kind of lifeless and forgettable acronym, but that's not my style. I proposed cloudbursting in a meeting a month or two ago and everyone seemed to like it.

So, here we go. Cloudbursting is an application hosting model which combines existing corporate infrastructure with new, cloud-based infrastructure to create a powerful, highly scalable application hosting environment.

Earlier this week my colleague Deepak Singh pointed me to a blog post written by Thomas Brox Røst. In the post, Thomas talks about how he combined traditional hosting with an EC2-powered, batch mode page regeneration system. His site (Eventseer) contains over 600,000 highly interconnected pages. As traffic and content grew, serving up the pages dynamically became prohibitively expensive. Renerating all of the pages on a single server would have taken an unacceptably long 7 days, and even longer as the site became more complex. Instead, Thomas used a cloudbursting model, regenerating the pages on an array of 25 Amazon EC2 instances in just 5 hours (or, as he notes, "roughly the cost of a pint of beer in Norway."). There's some more information about his approach on the High Scalability blog. Thomas has also written about running Django on EC2 using EBS.

I'd be interesting in hearing about more approaches to creating applications which cloudburst.

-- Jeff;

Comments

[Shane]

Damn, beer is expensive in Norway.

[John Nousis]

There are a lot of ideas for using a hybrid model. What we do in one of our projects (a social network in Greece called Zuni.gr) is to switch all non-realtime processing to EC2 instances that are simply started, perform what we need and then stopped. These are mainly tasks related to databases and reports. If you perform such tasks on the same server as your database they will take long time to complete and also cause a noticeable slowdown to your users. But having the possibility of launching a huge machine or even an array of huge machines and performing these tasks in minutes without anyone noticing is really great. Not to mention the cost which is less than buying a beer in Greece for us:)

[John Gannon]

Good post- I think this hybrid model could also address movement of specific IT infrastructure processes into the cloud as well. I touch on this in the post below:

http://johngannonblog.com/2008/09/01/cloudbursting-and-so-much-more/

[John Qualls]

As companies evaluate Cloud Computing there are trade-offs that must be considered. On the positive side companies which take advantage of the dedicated server hosting or Virtual servers available when they switch to cloud computing enjoy unlimited capacity, bandwidth, increased security, and a solid disaster recovery solution.

The challenge - A the transition to an enterprise co-location strategy often requires a fundamental shift in a company's core operating process.

According to Amazon Web Services evangelist Jeff Barr there is a middle ground, existing corporate infrastructure with new, cloud-based infrastructure to create a powerful, highly scalable application hosting
environment. He defines this hybrid approach as cloudbursting.

As interest continues to grow in the capabilities of the cloud, I expect we will see more discussion on these hybrid approaches.