Amazon Web Services Blog

Gowalla - Location-based iPhone 3G Application

Gowalla is a location-based iPhone game. Currently focused on Austin, Boston, New York, and San Francisco, the application runs on any iPhone 3G with the location-based services turned on. You use Gowalla to collect "virtual souvenirs" by visiting specified locations in each city, neatly combining the virtual and the real worlds.

A product of Alamofire, Gowalla runs on EC2 and makes extensive use of S3, SQS, and SimpleDB. To learn more, you can watch Alamofire founder Scott Raymond's presentation at the Scotland on Rails conference. In the video, Scott talks about the pros and cons of using AWS, how they scaled their MySQL database tier, Facebook interaction, and much more.

-- Jeff;

ScienceResearch Deep Web Search Engine - Now On Amazon EC2

The new version of ScienceResearch.com was officially launched today. Earlier this year I spoke with company president Abe Lederman and learned that this deep web science search engine provides a single point of access to over 400 publicly accessible science and technology collections.

The new version is hosted on EC2 and includes advanced search features such as relevance ranking, clustering (by topic, author, publication, or date), and exporting of search results in popular citation formats. The site indexes content in 15 different topic areas including agricultural sciences, astronomy & space, biology & nature, chemistry, computers & technology, defense technologies, earth & environmental sciences, energy, health & medicine, materials science, physics, and mathematics. It also indexes patents and science news.

 

Abe will be presenting a paper titled Journey to Ten Thousand Sources at the SLA (Special Libraries Association) conference this week. The paper contains a lot of interesting information about the architecture of this highly scalable federated search engine including details of the ways in which queries are partitioned into batches and then spread across multiple crawlers.

-- Jeff;

AWS Links - Wednesday, February 25, 2009

Ok folks, it is Wednesday, and that means it is time for another in my continued series of posts about what's happening in the Amazon Web Services world! With 490,000 developers in our program, there's always plenty to talk about.

I write these posts using source material drawn from emailed hints, clues that I find on Twitter, hallway conversations with colleagues, technical news sources such as Hacker News and Reddit's Programming section, and items that I glean from the hundreds of blogs that I scan each week.

Today I've got information about some new AWS case studies, a help-wanted ad for a system administrator position in New York, law firm automation, news from RightScale, an update on the SimpleDB Explorer, another iPhone interface to EC2, a developer contest from the makers of the Peek, a beta test announcement from CohesiveFT, and a success story from Others Online.

 

New Case Studies

Our marketing department has been working overtime to put together some new case studies. Their newest studies describe how Wowza Media Systems ("The #1 Choice For Media Streaming") and Xignite ("Financial Market Data On-Demand") use AWS.

The Wowza Media Systems case study recounts Wowza's decision to use EC2 to allow startups and small businesses to offer scalable Flash media streaming, live and on-demand video, and other services. They explain that hundreds of customers have signed up for the service via Amazon DevPay and that their revenue has increased by 400% in just 6 months. Their solution can also access media stored in Amazon S3.

Moving right along, the Xignite case study talks about how they used EC2 and S3 to construct a highly scalable platform which serves up financial market data to over 400 corporate clients. Faced with a traffic curve which looked like a Bell Curve, they built a system which incorporates application servers, load balancers, and caches to handle peak loads without the need for standby capacity.

We have a whole bunch of other great AWS case studies if you want to keep on reading.

 

Help Wanted - AWS System Administrator in New York

Tammy from the Filife personal finance community asked me to post a help wanted ad for her and I am happy to do so. Per their job posting they are looking for a versatile system administrator to maintain their EC2-powered virtual server farm. This is a full-time position based in New York.

Although Tammy that their implementation is "a bit too generic to be interesting," I don't believe that to be the case. They use multiple sizes of EC2 instances to run the web front-ends, MySQL, and HA Proxy, with a bunch of EBS volumes for storage. Code deployment is managed via Puppet and Capistrano, allowing them to set up new instances very quickly.

 

LawRD - AWS-Powered Data Tracking for Law Firms

Jorge emailed me from Portugal to make sure that I knew about his company's product, Lawrd. Powered by EC2, this product collects, tracks, and produces reports on all of the data generated by a lawyer or a law firm.

After paying a very modest per-user monthly fee of 14 Euros per month, users can track the facts (what, when, where, and how) for each case, client, and employee. Timesheets, contact lists, invoices, and reports are all readily available. All data is encrypted and of course there's no local software to install.

Jorge also told me that they use Eric Hammond's Unbuntu AMIs and EBS, and that they can update their code with a simple SVN checkout.

Qualified users can get started today with a free trial.

 

RightScale Rocks On

There's a whole lot happening at RightScale ("Cloud Computing. Delivered"). As I write this, the counter on their site indicates that they have launched 324,051 servers!

On March 3rd we'll host an Executive Seminar on Cloud Computing in New York with speakers from Amazon Web Services, RightScale, MySQL, and Starcut. Unfortunately, registration has already been closed due to a very high response. You can still join the waiting list by emailing rsvp-aws@amazon.com.

Earlier this month they announced support for EC2 in Europe, with full support for replication of AMIs and their server templates across the Atlantic. And, just yesterday, they updated their RightScale Ruby Gems.

 

New SimpleDB Explorer

Saurabh informed that there's a new release of SimpleDB Explorer on the loose! The new release supports exporting of a domain's contents to an XML file, sorting, searching, the new SimpleDB Select syntax, and also displays the cost of each query. There's also a new command-line tool. Read more in his blog post.

 

ElasticPod - EC2 Cluster Management on the iPhone

Eugene wrote to tell me that his ElasticPod EC2 interface is now available in the iPhone App Store.

You can launch new EC2 instances, manage your keys, and terminate individual instances or all instances in a reservation group. You check the status of your instances on a color-coded display, and you can even bounce back and forth between multiple AWS accounts. ElasticPod works in vertical and horizontal view, and looks pretty cool (note to boss: Please buy me an iPhone so that I can test out cool apps like this).

 

Peek Developer Contest

Dan from Peek wrote to tell me that they are running a developer contest. They are looking for some cool applications which combine the Peek APIs and the location-based APIs provided by Xtify.

A Peek (since you were just about to ask), is a mobile device dedicated to email. Designed for everyone except hardcore techies (per their site), it is thin, stylish (available in three colors: gray, aqua, and cherry), and very easy to use. You can buy one online or at your local Target. I have to say that it is is really cool to see AWS-powered devices at the Target down the street from my house. Technology has now become far more commonplace and accessible than it was just a few years ago.

Last year, Dan described Peek's use of AWS in a blog post. At the time they were running over 30 EC2 instances and storing about 400 GB worth of data on some EBS volumes.

 

CohesiveFT Looking for VPN-Cubed Beta Testers

Patrick from CohesiveFT dropped me a note to tell me that they are now looking for some beta testers for the EC2 version of their VPN-Cubed ("Customer controlled security for the cloud") product; read about the features and requirements here. Beta testers are promised a dramatic discount on the finished product.

You will be able to use this new product to set up a secure "overlay network" which can span EC2's US and EU regions and even other cloud providers. You will have full control of addressing, topology, protocols, and encryption.

 

Others Online Uses AWS to handle 100,000,000 User Profiles

Old friend and one-time colleague Mike Dierken (we worked together last century during the dot-com boom) wrote to tell me how his company, Others Online, uses AWS to track over 100 million user profiles. This Seattle company helps publishers to learn what their audiences care about. They do this using natural language processing, evolutionary algorithms, data mining, and some AI. This all takes a lot of compute power.

Mike told me that they use a pool of application servers, scaling up and down throughout the day in response to actual load. Data is stored on a set of horizontally partitioned MySQL databases and they spin up more as their user base grows. They also use EC2 for their heavy-duty data processing, starting up a bunch of instances, doing the work, and then shutting them down once the work is done.

You can learn more about Others Online by watching the fun video on their home page.

 

And that will have to do it for today; I've still got lots of other stuff on today's TODO list.

-- Jeff;

AWS Links - Wednesday, February 11, 2009

As always, my inbox is simply overflowing with new and innovative AWS-powered applications. Here's just some of what I've received in the last week or two: cool applications from Direct Thought, Zemanta, Jooners, Ylastic, enStratus; Solaris on EC2 from Sun; a job opening at Family Link; a Java library for SimpleDB; system software from AsterData and DataSynapse; an EC2 AMI for processing large amounts of data, and a remarkable new way to use the Mechanical Turk.

David from Direct Thought wrote to tell me that they're working on an iPhone application to control EC2 resources -- images, instances, volumes, security groups, IP addresses, and keypairs. The application supports multiple EC2 accounts and includes an AMI search feature to make it easier to find what you want in our AMI catalog.

He also told me that the application was written using an Objective-C port of his highly respected Typica library. This Apache-licensed port is called cTypica and is available from Google Code.

 

Andraz from Zemanta dropped me a line to let me know that they released a major upgrade and expansion of their content creation and discovery tool.

The original release of the Zemanta provided a helping hand (almost a research assistant) for bloggers, suggeting relevant links and content in real-time as the post is written. This functionality has been enhanced to allow it to recommend additional types of content, including audio, video, and maps. Zemanta takes care to recommend content that has a Creative Commons license or that has been approved for reuse by the original content provider.

The new release now extends this same helping hand to web-based email in the form of a Firefox extension which supports Gmail and Yahoo! Mail. It also adds support and recommendation for specialized content categories including books, music, technology, blogging, green, health, travel, gaming, shopping, and wine.

You can learn more by watching a pair of short and sweet screencasts. The first one shows how Zemanta works with Yahoo! Mail, simplifying the task of linking to pages on popular social networking sites while composing an email. The second one shows how Zemanta can enhance Gmail, suggesting some relevant pictures, links, and reference articles.

 

If you've ever had to plan and coordinate an event across families, teams, clubs, or other semi-formal groups, you'll want to take a look at Jooners, especially if you need to apportion and collect money for fees, dues, or other costs from the participants.

Jooners uses a clean and straightforward wizard-based planner model to plan one-time or recurring events, coupled with the Amazon Flexible Payments Service to handle all of the payment details. There's a helpful introductory video on the home page and a gallery of popular planners.

 

Rajesh from Sun wrote to tell me that the OpenSolaris AMIs for EC2 are now available for use with the US and European EC2 regions. His blog entry includes all of the important details.

This is the 2008.11 release of OpenSolaris which includes a number of new features such as new releases of the Gnome desktop, OpenOffice, and Firefox, automatic file snapshots to ZFS, the very cool-looking Time Slider, better printing, the Songbird music player, an improved package manager, and lots more.

Read the Getting Started Guide to, uh, get started. You may also want to consult the OpenSolaris AMI catalog to learn more about the available public and private (registration required) AMIs.

 

My friends down at Utah-based FamilyLink.com want to hire a combination Systems Administrator and Architect.

The job posting notes that "You'll be on the front line, making sure that our Amazon AWS-based FamilyLink.com sites always work." In a recent blog post, founder Paul Allen notes that their We're Related on Facebook application is one of the 5 most popular, with 12.4 million daily users. Paul says that they've had to restrain growth in order to make sure that they have their system issues under control.

You'll need experience designing, building, and running large-scale systems with AWS, fluency in a couple of scripting languages, and experience with a relational database and a key/value store like Memcached or SimpleDB.

 

Per a note in the Ylastic blog, the Ylastic dashboard for the Apple iPhone and the Google Android is now in production and available for just $10 per month. You can control EC2 instances in the US and in Europe, manage CloudFront distributions, SQS queues, S3 buckes and objects, and watch the AWS Status Health Dashboard.

Check out the pricing and the full feature list to learn more.

 

Minnesota-based enStratus has launched a web-based console to control the deployment of EC2-based web sites and enterprise applications. The console manages security and availability profiles based on organizational requirements, and can bring up additional load balancers, application servers, and database servers as needed. It also handles multiple levels of cloud and cloud-independent backups.

Company founder George Reese writes the Cloud Blog and is wrapping up work on an O'Reilly book called Cloud Application Architectures, currently available in a Rough Cuts edition.

 

SimpleJPA is an open source implementation of the Java Persistence API (JPA) for SimpleDB.

It supports ManyToOne references and OneToMany collections, both with lazy loading. It supports large objects using S3 and caches results. It also makes concurrent requests to SimpleDB to speed things up, and even supports a subset of the JPA Query language.

SimpleJPA's caching and S3 support comes courtesy of BigCache, a simple Java API which implements an infinite shared cache on top of S3. BigCache includes come cool asynchronous methods.

 

Aster Data Systems just released the Aster nCluster Cloud Edition. This release brings the Aster nCluster analytic database to the EC2 cloud, greatly simplifying the process of creating a data warehouse or BI (business intelligence) project and reducing time to market. The product includes a unique in-database MapReduce processing framework; data analysis and transformation is supported inside of the database itself. There's more info in the data sheet.

Aster Data will be hosting a webinar, next week. Titled "Data Warehousing in the Cloud", the webinar will describe how AWS customer ShareThis was able to create a multi-terabyte data warehouse without making a heavy investment in people, time, or hardware.

 

A note on the InfoChimps blog, and a hallway conversation/reminder with my colleague Deepak Singh, led me to machetEC2. Described as an "AMI configuration for exploring Big Data," machetEC2 is an EC2 AMI pre-configured with a very long list of data processing, analysis, and visualization applications atop a solid base of Python, Ruby, and Erlang tools and libraries.

You can launch this AMI and be chewing through a massive chunk of data pulled from the AWS Public Data Sets in no time flat. The next version of machetEC2 will include commands to simplify the process of creating and mounting EBS volumes containing this data.

This is your basic nuclear-powered chainsaw for heavy duty data processing!

 

DataSynapse has extended their DASM (Dynamic Application Service Management) product, DataSynapse Federator with support for EC2). DataSynapse customers will be able to seamlessly bridge traditional data centers and the EC2 cloud. They will be able to create development and testing grids at a very low cost. Finally, they'll be able to use EC2 for disaster recovery, paying for capacity only as needed

Existing DataSynapse customers can register for the space-limited beta program between now and March 4th.

 

Seattle-based SmartSheet is a work management and collaboration tool. Using their straightforward spreadsheet-based model, you can easily enumerate and capture the tasks needed to complete a project and then assign them to partners and co-workers. Already users of EC2, S3, and CloudFront, they have now added Mechanical Turk support to the product.

In addition to assigning tasks to partners and to co-workers, you can now assign them to qualified Mechanical Turk workers on an individual or volume basis. You can now create and successfully complete projects using a workforce of over 200,000 people from over 100 countries! The new brochure suggests that this can be used for surveying, transcription, web research, image tagging, copywriting, website testing, and keyword research. The crew at SmartSheet have called this new model smartsourcing, which sounds good to me.

SmartSheet handles all of the setup, submittal, and retrieval of Mechanica Turk HITs (Human Intelligence Tasks); no coding is required. You can request multiple answers for each item, and you can monitor the results as they arrive using a unified management dashboard. Images can be attached to requests or even requested as responses. You also have the power to accept or reject answers and can even pay bonuses to the best workers. Read more about all of this in the advanced brochure.

Before you go, head over to this page and click the link labeled "Watch Overview Video" to see Smartsourcing in action. This amazing new feature is available to all paid users of SmartSheet.

 

And that's all for today. If you would like your AWS-powered application to be featured here, send me some email and I'll do my best.

-- Jeff;

AWS Links - Tuesday, January 27, 2009

A whole lot of interesting AWS items have landed in my inbox in the 20 days since my last links post. Here's what I have queued up:

Chris Richardson wrote to tell me that there are still a few seats available in his upcoming half-day class, Running Java and Grails applications on Amazon EC2.

The class will take place on February 17th in Oakland, California. Chris, the author of POJOs in Action, creator of the Cloud Tools, and proprietor of the Cloud Foundry (recently reviewed in eWeek), will cover Amazon-style cloud computing, deep-dive into AWS, talk about the use of Amazon EC2, provide an overview of the Cloud Tools, and then jump into developing and deploying on EC2.

 

Tom Lounibos, CEO of SOASTA, dropped me a note to tell me that they recently used the EC2-powered CloudTest sytem to simulate the effect of 500,000 concurrent users downloading songs from a major media site. In an entry on the SOASTA blog, Tom notes that

Companies such as Hallmark.com, Genentech, and Proctor & Gamble have discovered Cloud Testing as an affordable and scalable alternative to traditional stress testing. For a few thousand dollars huge stress tests can be acheived in as little as one hour.

These tests generate a substantial amount of data, ranging from 100 GB up to 1 TB in certain cases. SOASTA's testing tool also provides real-time analytic processing of this huge amount of data.

 

Prabhakar from Ylastic wrote in to tell me that they have added full support for EC2's new EU region. They provide access to all EC2 resources (AMIs, access keys, security groups, elastic IPs, EBS volumes, and EBS snapshots), along with with a set of filtering and searching toools. Support for mobile (iPhone and Android) versions is almost ready and will be rolled out in the near future. Ylastic also tracks and organizes all EC2 alerts.

 

I met Yuvi Kochar, CTO of The Washington Post, at a DC-area CTO event a year or two ago. We've stayed in touch and I paid a visit to him and his team last year on one of my trips east.

A few months ago Yuvi let me know that he needed a Wiki running in the EC2 cloud for a new project. After getting a good understanding of his needs I pointed him at the Mindtouch Deki product (previously blogged here). Yuvi wrote about his experiences in selecting and setting up the site, noting that "Our dev and prod sites were up and running in days! Incredible!"

Last week, in conjunction with the inauguration day festivities, they launched Whorunsgov.com. On the site you can get an insight into the people behind the scenes in Washington DC. You can see how deals get made and how policy is shaped. Read more on the site's blog.

While I was putting the finishing touches on this post, I saw a Tweet from the folks at Mindtouch. They put together an excellent press release with even more information about the project and the selection criteria.

 

Jamal from Kaavo wrote in to tell me about the recent release of their IMOD product. IMOD stands for Infrastructure on Demand. IMOD uses an application-centric view of lifecycle management, managing anything from a single server to a complex multi-server system using their dynamic template technology. They also support goodies such as AED256 encryption of EBS volumes, monitoring of CPU, I/O, and memory usage, alerts for application service levels, and automatic backup of EBS volumes.

Read more on the IMOD Fact Sheet or watch some of the following videos (each conveniently hosted on Amazon S3):

• IMOD Demo

IMOD Template Overview

• IMOD Quick Start

 

On Wednesday, February 4th, AWS evangelist Jinesh Varia will talk about use of AWS within the biotech community. Co-sponsored by Cirrhus9, the event will take place at Pfizer's lab in La Jolla, California. Attendance is free but you need to register. You'd better hurry, there are just 17 seats left.

 

Also (coincidentally) on February 4th, AWS Evangelist Simone Brunozzi and CohesiveFT co-founder Alexis Richardson will be hosting an AWS Meetup in London.

Simone and Alexis will each speak for 20-25 minutes, and there will be time for some Q & A after that. The meeting will then adjourn to the Crown Clerkenwall Green and Alexis will buy the first couple of rounds! Once again, registration is a must.

 

Bob from Rockstar Apps wrote to tell me about their new Amazon WS tools. The tools plug in to Eclipse and Aptana, where they are available as the "Amazon WebService Perspective."

The Amazon WS Tools provide full support for CloudFront, SimpleDB, S3, SQS, and EC2. Each tool provides one or more views into the corresponding Amazon web service. For example, you can enter SimpleDB queries in the Query Editor, and you can see the results in the SimpleDB View, as you can see here.

The tools can be downloaded here. Bob provided me with the following installation instructions:

1. Go to the [Help -> Software Updates -> Find and Install] Menu Item.
2. Create a New Remote Site. Call is "jsLex Update Site" and reference http://www.rockstarapps.com/update-beta.
3. Keep clicking through the screens until it is installed.

 

Pete from Juice Analytics wrote to tell me about their new Concentrate application. Concentrate is a search analytics tool for SEO and paid search professionals. It allows them to make better decisions, target SEO efforts, understand paid search campaigns, and better understand customer needs. Concentrate discovers and visualizes interesting patterns in the search queries used to locate a site. The word tree shown at right is just one form of output.

Concentrate is available under a number of different pricing plans, ranging from Free all the way up to Max. You can also start out by looking at the online demo.

On the technical side, Concentrate runs on EC2. Here's what Pete told me:

The front end is powered by the DJango framework and the jQuery JavaScript library, all load balanced atop a number of EC2 instances. They run MySQL and store their data on EBS volumes.

The back end consists of an EC2 cluster running Concentrate's text mining algorithms. It accepts requests using a REST API and returns data in JSON format. The back end uses Amazon SQS and S3, and was patterned after the model found in our Grep The Web example. Scaling is handled using a combination of iClassify, Puppet, and Capistrano. The deployment infrastructure was built by HJK Solutions of Seatle.

 

Andrew from Job Bounty Hunter wrote to tell me that they'd launched the site and that every last bit of it is running on top of AWS.

Employers and recruiters can use the site to publish job advertisements, along with a cash bounty on each job. Bloggers and web publishers can post the ads on their site using a variety of widgets. Job seekers apply for the jobs via the widgets. If they are placed in the job and stay there for 90 days, the blogger or web publisher collects the bounty! As you can see from their bounty chart, there's currently $7,800 up for grabs just a few days after their launch. This is a really cool way to monetize web traffic while also providing a very interesting service for the site's readers.

Because the widgets are embedded in other sites, Andrew has no idea when a traffic surge might hit. They must be able to scale to match the traffic of all of the sites that are displaying their widgets. Of course, this need for elastic capacity made it a perfect for for AWS!

They decided to serve up all of the widgets as static HTML. They are created using PHP and then downloaded using Curl into static HTML. The static HTML is then deployed to CloudFront (via S3) and Nginx. The front end of the site was implemented using HTML, CSS, Adobe Flex, JavaScript, and (again) jQuery.

Job Bounty Hunter currently runs on three EC2 instances. The first runs the free version of IBM's DB2 database. The seocnd one runs Apache, Postfix, PHP, and some web service code. The third instance runs Nginx and serves up static content such as widgets, JavaScript, images, HTML, and SWF files.

Andrew also sent along a very nice architecture diagram as an image and as PDF.

 

Long-time AWS user Alex Iskold has written an illuminating post about his use of Amazon SimpleDB for his new Glue product.

In the post, he talks about how Glue is used connect people and things, recognizing books, music, movies, and other everyday topics in hundreds of web sites and connecting people around the topics. He then goes on to discuss the reasons why a relational database won't solve his problems, including scale and automatic partitioning and replication of his data. From there he describes his use of 30 SimpleDB domains to hold information about People, and another 30 about Things. He uses the djb2 hash algorithm to spread the information out across the available domains, and also stores data redundantly to obviate the need for joins.

 

Lasso2GGO uses a number of different services including S3, EC2, and the Mechanical Turk. This service makes it easy to convert "analog" business cards into handy digital data.

You can take a picture of the card with your cell phone web cam, or otehr device and upload it to Lasso. After some image enhancement on EC2, the image is transcribed by a Mechanical Turk worker and deposited in Salesforce, an email, or a spreadsheet. Read all about this in the recent Information Week story, How The Cloud Enables A New Set Of Personal Applications.

 

Finally, John-David from New Zealand's Mindscape wrote in to tell me about their new products: SimpleDB Management Tools for Visual Studio and LightSpeed - The Mindscape .Net O/R Mapper.

The SimpleDB Management Tools integrate directly into Visual Studio 2008 and above. The tools support direct addition, editing, and deletion of SimpleDB data and domains and can also run SimpleDB queries. The tools plug in to the O/R Mapper, with direct dragging of SimpleDB domains from the management tool into the mapper.

 

And that's all that I have time for today. I hope that you've enjoyed this glimpse in to some of the cool stuff that our developer community (now 490,000 members strong) has been up to. Send me information about what you are doing with AWS and I'll do my best to fit you in.

--Jeff;

AWS Links - Wednesday, December 23, 2008

Lots of people responded to the link post that I put together next week. In fact, between the "what about me" emails and the responses to a Tweet that I made earlier today, I now have a plethora of good material. So, here we go again!

 

Information Week has named Amazon CTO Werner Vogels as their Chief of the Year. In a very detailed article they cover the history, current state, and overall philosophy of AWS. There's also a separate interview with Werner.

The article even talks about our customer base, noting that "AWS is a popular platform among startups, Web companies, and software-as-a-service companies. Increasingly, Amazon's customers are household names: Nasdaq, The New York Times, Philips, SanDisk. Eli Lilly is using EC2 to deploy SQL Server/Windows Server instances as needed for research data. The Indianapolis Motor Speedway uses AWS for Web site mirroring, video streaming, and digital image archiving."

 

Werner recently wrote an article about Eventual Consistency for the ACM Queue Magazine. Understanding this important concept (which has its roots in the CAP theorem) is essential to building a world-scale distributed system which is also highly reliable.

Werner sums up the theorem as follows:

A system that is not tolerant to network partitions can achieve data consistency and availability, and often does so by using transaction protocols. To make this work, client and storage systems must be part of the same environment; they fail as a whole under certain scenarios, and as such, clients cannot observe partitions. An important observation is that in larger distributed-scale systems, network partitions are a given; therefore, consistency and availability cannot be achieved at the same time. This means that there are two choices on what to drop: relaxing consistency will allow the system to remain highly available under the partitionable conditions, whereas making consistency a priority means that under certain conditions the system will not be available.

 

Stax makes it easy for Java developers to build, manage, and scale applications on Amazon EC2. On the coding side, there's a complete MVC framework, RIA tools, and complete support for scripting. Development and testing is simplified by a local toolkit and an Ant plugin for easy builds. At deployment time, Stax uses EC2 to provide access to unlimited server resources. It is currently in beta and you can sign up here.

 

John M. Willis wrote to tell me about his Cloud Cafe podcast. He's already recorded episodes with many of the major players in the cloud computing space, including GigaSpaces, RightScale, and Elastra. He's already recorded 28 episodes, so you'd best start listening now before you fall even further behind!

 

The very same John M. Willis asked me to mention that Cloud Camp Atlanta will be taking place on Tuesday, January 20. Cloud Camps are a wonderful way to meet other people who are involved in and interested in cloud computing, and a can also give you a good sense of why people are so excited about it.

 

Adam Kalsey has put together a how-to video for their new Drupal AMI. In the 10 minute video, Adam shows how to use ElasticFox to launch their public Drupal AMI, connect it to an EBS (Elastic Block Store) volume, stop the instance, and then reconnect the storage to another EC2 instance. Run the video full-screen in order to see all of the details. Adam's howto is also a really nice introduction to ElasticFox and to EBS. Although the video shows how how to use fdisk to create a partition table on the EBS volume, I've never bothered with that step. Instead, I simply run mkfs on the entire device (which would be /dev/sdj in the video).

 

Bob and Tony from HelpStream wrote me and asked "How can we get on this list?" That's easy - all you need to do is ask, and I'll do my best! They recently moved the complete running HelpStream ("The World's First Truly Social CRM system"), all 140 clients and 90,000 users, over to Amazon EC2.

ZDNet recently chronicled this migration in a comprehensive and worthwhile story titled Migrating to Amazon Web Services: The Blueprint. In this story you can read about how Helpstream's infrastructure was previously running at just 10% of capacity and how our prices were a selling point for them. The article covers phase 1 (using Amazon S3 for backups, and EC2 for test servers); phase 2 (moving about 85% of their storage over to S3), and phase 3 (moving the production system to EC2). The final step took them 5 hours over a weekend. This was time well spent since they estimate that the move will save them 21% on bandwidth and cage space, 59% on monitoring, server administration and in-cage work, and 100% on servers, switches, VPNs and other infrastructure which was obviated by the move to the cloud.

 

So you now understand Software-As-A-Service, Infrastructure-As-A-Service, and the various other "-As-A-Service" models that have been becoming increasingly popular of late. Great, because I've got another one for you. William from RetailZip wrote me early this month to tell me that their new product is a Format-As-A-Service!

RetailZip files are small, encrypted container file which provide gatewayed access to high-value online content. Once created and posted online, the content represented by a RetailZip file can be purchased online in any of 18 currencies. The content is stored on S3 and is decrypted and downloaded via EC2. Personal ($1.99 / month) and business ($9.99/month) licenses are available.

 

Kingsley from OpenLink Software wrote to tell me that they have released the Cloud Computing Edition of their Virtuoso Universal Server. As the press release notes, "The new product release leverages the solution packaging and deployment prowess of the Amazon EC2 cloud-computing platform by delivering a pre-configured and tightly tuned edition of Virtuoso on a Fedora Linux-based Amazon Machine Image (AMI), ready for immediate use (post initialization)."

Their cloud offerings include the Bio2Rdf bioinformatics database packaged up and available in AMI form, with full directions for instantiation and use here, the Neurocommons database for biological research (full directions here), and the DBPedia ontology of Wikipedia knowledge, with full drections here.

Kingsley is intrigued by the ease with which researchers can now instantiate truly massive databases in the cloud and told me that "In all cases, analysts, researches, and knowledge/information workers in general now have the ability to instantiate knowledgebases in the EC2 Cloud. We are talking 1.5 hrs compared to error prone 16 - 22 hrs knowledgebase commissioning marathons that are inherently error prone."

 

The brand-new Amazon Payments blog looks like it is going to be interesting. They say that "A number of authors will contribute on their respective areas of interest and expertise such as ecommerce, online shopping, merchant technologies, developers integrating payments into their projects, mobile payments and more."

They are also on Twitter.

 

Billy Marshall from rPath wrote to tell me about their new video, Cloud Computing in Plain English. In just 5 minutes, this entertaining video will educate you about SaaS, Cloud Computing, Virtualization, and the relationship between them.

 

Cale Bruckner from Email Center Pro wrote to tell me that they've been using AWS to help companies of all sizes to do a better job of processing customer email. Their application allows companies to centralize emails, assign them to people for followup, and to respond with greater efficiency using templates, internal notes, and other facilities (hmmmm, maybe I need this). Pricing starts at $19.00 per month after a free trial.

Earlier this year they wrote a long story about their architecture. They started out by storing messages in Amazon S3. This worked well, allowing them to keep their customer's data safe and secure. They then moved the application itself over to EC2, building a set of AMIs which allow them to launch additional instances as needed to deal with peak traffic and high volumes of email. They conclude with the statement that "Email Center Pro is an example that you really can build an application entirely on the Amazon Web Services platform with great results."

 

Ok, that should do it. Happy holidays everyone!

-- Jeff;

Twilio - Telephony in the Cloud

Twilio founder Jeff Lawson stopped by Amazon headquarters yesterday for a show and tell session. Twilio provides a simple yet powerful way to build highly scalable telephony applications. Of course, Twilio itself runs on Amazon EC2 and stores data in Amazon S3.

A Twilio application is simply a phone-activated web application. When the application's phone is called, Twilio answers and activates the application. The application then returns an XML document containing TwiML (Twilio Markup) commands. Jeff showed up how Twilio's 5 commands (<Play>, <Gather>, <Record>, <Say>, and <Dial>) can be combined to create applications in minutes.Here's what they do:

<Play> is used to play an audio file for the caller. Twilio will transcode the file in real-time, turning high-quality audio into the required 8 bit 11 kHz format.

<Gather> accepts one or more digits from the caller's keypad and passes them to a specified URL using POST or GET.

<Record> captures the caller's voice and returns a URL which points to the recorded audio. Recording can be terminated using a specificed keypad key or after a specified quiet period.

<Say> invokes a text to speech engine with male and female voices in 4 languages.

<Dial> is used to connect the caller to another phone number.

Pricing is friendly for developers! Developer accounts are free and include 1000 minutes of calls. Full accounts cost $5 per phone number (local or toll free), then 3 (local) or 5 (toll free) cents per minute for incoming calls and 3 cents per minute for outgoing calls.

Jeff showed us an application that he'd built the day before. The application allows the caller to request the status of EC2, S3, or SQS. The application then parse's the AWS status dashboard's HTML and echoes the status of the requested service. You can read all about the application or you can try it out by calling 206-866-5918. 

You can get started here (you'll need to ask for an invite code there first).

Update: Jeff just emailed me a link to a Slideshare presentation with even more info about Twilio. The presentation includes some really interesting information about how they use EC2, S3, and SQS to build Twilio, and how they build and customize their EC2 instances. He also let me know that they have plenty of invite codes available for readers of this blog.

-- Jeff;

VholdR in Action

Earlier this year I blogged about the VholdR, a wearable camera combined with a video sharing system. VholdR is designed for people who participate in action sports such as mountain biking, snowboarding, or skiing.

As noted previously, the hardware component is a small and rugged video camera suitable for use in extreme conditions -- ski slopes, competitive cycling, and so forth. The camera can record 2 full hours of video on an embedded MicroSD card. Once mounted, leveled (using a pair of lasers, no less) and aimed, a single, glove-friendly switch controls the recording process. There's also a microphone built-in.

Marc Barros, CEO of VholdR, stopped by Amazon headquarters last month and brought me up to speed on their progress. They are now in full production and now has users in 57 countries sharing videos on the site.

I am firmly in the "none of the above" category, with respect to action sports, but Marc was kind enough to leave a VholdR for me so I spent some time wandering around my house. If you watch the video embedded in this post you can see my dog, my backyard, my home office, and my bird feeders. You can also go here to see the video on the VholdR site.

Marc told me that the site stores the customer's raw, high-quality video in Amazon S3. They also do flexible, on-demand video transcoding on Amazon EC2 and store the finished files back in S3 for delivery via the website.

Of course, if you want to buy one of these devices for yourself or as a gift, you can get one at your favorite online retailer.

-- Jeff;

PS - They have some job openings too.

US National Voter Guide for Election '08

The EC2-powered Voter Guide is a very powerful and helpful site, as well as a handy illustration of why scalable cloud computing is the perfect solution for short-term sites which have the potential to draw a lot of traffic.

Designed by E-thepeople and produced in conjunction with over 100 local newspapers and TV stations, the site was designed to improve civic participation in US elections using internet technologies.The site makes it easy to find the races in each area, compare the candidates, evaluate other ballot issues, and then record and print choices in ballot form for election day.

After an address is entered, the site creates a customized voter's guide. Based on my home address, my voter's guide contained 23 items -- some candidates, some initiatives, some propositions, and a few amendments. Each page (representing a single item) contains a summary of the item in question and a side-by-side comparison of the candidates or issues in question.

Sites which are relevant for a relatively short amount of time (in this case the weeks and days leading up to the US election). Paul Kahn, the site's operator, told me that peak traffic will run at 100x to 1000x the normal level. This makes EC2 a perfect solution, since they can add more instances as traffic grows, and then remove them immediately after the election after the polls close and traffic subsides.

So, US citizens, give the Voter Guide a shot, and don't forget to vote!

-- Jeff;

New and Cool - VPN-Cubed & Glue

Two of my friends, coincidentally named Alexis and Alex, emailed me yesterday with information about their latest and greatest AWS-powered products.

Alexis works for CohesiveFT.

Their new product, VPN-Cubed, creates encrypted, private VPN (Virtual  Private Network) connections between endpoints in a single cloud, between multiple clouds, and between the cloud and a physical data center.

Corporate customers (such as the ones I met at yesterday's Architecture and Integration Summit in Minneapolis) are taking a serious look at EC2. When they do so, they invariably ask about security. Products like VPN-Cubed are of immense value in this situation, providing customers of this type with important new ways to securely move data from place to place in a secure and totally protected fashion.

Read the VPN-Cubed datasheet to learn more, or flip through the 30-slide overview on SlideShare.

Update: The Rational Security blog has a really nice review of the product: CohesiveFT VPN-Cubed: Not Your Daddy's Encrypted Tunnel.

 

Alex is CEO of AdaptiveBlue. He was delighted to tell me that they just completed their Series B funding round (congratulations) and also mentioned that they have some open developer positions in their New York City office.

They have just released Glue, a new way to share favorites and connect with like-minded friends online as they navigate through popular web sites. You can easily what your friends think about books, movies, music, stocks, restaurants, and more, even if information about the subject is scattered across more than one site.

Learn more by reading the blog post, watching the movie, and then get Glue. Glue was built using EC2, S3, and  SimpleDB.

And with that, time to head to the airport to fly home to Seattle!

-- Jeff;