Amazon Web Services Blog

IntrIdea Rolls out Scalr, Updates MediaPlug, Builds Community

The folks at Intridea have been coding up a storm!

In time for this week's Railsconf, they are rolling out a hosted, on-demand version of their popular Scalr tool, a new release of their MediaPlug media server appliance, and the Acts as Community social network.

Scalr gives enterprise IT professionals the ability to quickly and easily set up and run EC2-powered server farms. Once such a farm has been set up, Scalr monitors and maintains it, with automatic scaling, failover, and redundancy. Scaling is based on load averages, with automatic instantiation of new instances of the proper type once the aggregate load average reaches a configurable threshold. If an instance crashes, Scalr replaces it with a new instance of the proper type. The hosted version is available for $50 per month; the open source version can be downloaded here.

MediaPlug is an EC2-powered media server appliance, packaged and sold as an AMI (Amazon Machine Image). It supports transcoding, uploading, and storage of  images, audio files, and video files in a number of popular formats. There's a MediaPlug web service for direct integration into backend code, along with a JavaScript library for the front-end. MediaPlug is priced at double the cost of a Small EC2 instance, or approximately $150 per month.

Finally, Acts As Community, is a social network for Ruby developers. Of course, the site runs on Amazon EC2 and is managed by Scalr. The site includes forums and blogs for open source projects hosted at GitHub; user group creation and management tools, personal profiles, and a number of community features such as forums, questions and answers, media sharing, and code sharing.

-- Jeff;

More EC2 Power

Amazon EC2 users now have access to a pair of new "High-CPU" instance types. The new instance types have proportionally more CPU power than memory, and are suitable for CPU-intensive applications. Here's what's now available:

The High-CPU Medium Instance is billed at $0.20 (20 cents) per hour. It features 1.7 GB of memory, 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units Each), and 350 GB of instance storage, all on a 32-bit platform.

The High-CPU Extra Large Instance is billed at $0.80 (80 cents) per hour. It features 7 GB of memory, 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each), and 1,690 GB of instance storage, all on a 64-bit platform.

The AWS Simple Monthly Calculator now supports these new instance types.

We've been working with a number of tool vendors to line up early support for this important new feature. I plan to update the blog post several times in the coming days as this support becomes available.

-- Jeff;

Napera Networks - Network Health Solution

Seattle-based Napera Networks has built a really interesting piece of hardware. The Napera N24 Appliance automatically enforces health and identity rules for up to 200 computers, preventing rogue or unpatched machines from accessing a corporate network in an uncontrolled fashion.

The device basically protects access points in conference rooms, sales offices and wireless networks from indiscriminate usage. Napera uses the Microsoft NAP (Network Access Protection) protocol to enforce a fully configurable set of health and safety rules. Logging and monitoring features provide auditing and other historical information.

At this point you are probably thinking "Uh, Jeff, you work for Amazon.com and you usually talk about software on this blog. What's going on here?"

So here's where it gets really cool. The Napera N24 appliance securely stores all of its long-term logging and auditing data in Amazon S3. This way, there's no limit to how much can be logged or how long it is retained, unlike a more traditional design which would retain some logs within the appliance or require local servers. Further, the administrative interface (MyNapera.com) leverages Amazon EC2. Ultimately MyNapera.com will be accessible remotely, making the Napera appliance a great solution for a company with a number of geographically distributed field offices.

Enterprise solutions to this sort of problem are usually very complex and expensive, requiring multiple servers. Using AWS has enabled Napera to build a product for the budget of smaller companies that is incredibly easy to use and quick to install.

Oh yeah, one last thing. The company is based in Mercer Island (near Seattle). Per their jobs page, they are looking for a web developer and some summer interns with CS or IT experience.

-- Jeff;

New York Times TimesMachine

Derek Gottfrid and his colleagues at the New York Times have obviously been having a lot of fun with Amazon EC2.

Their latest offering is the TimesMachine. Print subscribers can access any issue of the New York Times, dating back to Volume 1, Number 1 in 1851. Non-subscribers can take a peek at 6 different (and historically significant) issues, including the inaugural edition, the end of World War I, and the sinking of the Titanic.

As they explained in their blog post, they used EC2, Hadoop, and some of their own code to convert 405,000 large TIFF images, 3.3 million SGML files, and 405,000 XML files to 810,000 PNG images and 405,000 JavaScript files. This didn't take all that long:

"By leveraging the power of AWS and Hadoop, we were able to utilize hundreds of machines concurrently and process all the data in less than 36 hours."

The content itself is really interesting, but I also enjoyed the fact that it was possible to see the articles in the context of the other issues of the day. The advertising is also interesting.

Robert Scoble has more coverage, including a video interview with Derek.

-- Jeff;

Vertica Webinar

I will be participating in a webinar on Thursday, May 22nd at 11 AM PST. Hosted by AWS user Vertica, the webinar will cover Vertica's cloud-based approach to analytic data management. The webinar is free, but you will have to register in advance if you would like to attend.

The Vertica Analytic Database runs on Amazon EC2 and S3 and is hosted completely within the Amazon cloud. Using this approach, they are able to smoothly scale to meet large and complex workloads, while also supporting automatic replication, failover, and recovery.

Unlike a traditional database install where you would have to pay for a data center, hardware, software, and administrators before you can store a single row, the Vertica solution is priced on a per-month, per-node basis. New nodes are available 30 minutes after receipt of order! In true cloud-based fashion, payment is handled through the Amazon Flexible Payments Service.

A traditional relational database takes a row-oriented approach to data storage. A fixed or variable block of contiguous space is allocated to each row. Vertica, by contrast, takes a column-oriented approach (hence the company name). The data is grouped by column instead of by row. This opens the door to many types of optimizations. Processing a single column of a database which has a large number of rows becomes very efficient, as does compression. Benchmarks indicate that approach can be 30 to 200 times faster.

I will be speaking about EC2 and S3 and about some general cloud computing concepts. I hope that you can join in.

-- Jeff;

Lots of Bits

In January of 2008 we announced that the Amazon Web Services now consume more bandwidth than do the entire global network of Amazon.com retail sites.

Amazon.com CEO Jeff Bezos has been showing a chart of the relative bandwidth usage and I just received permission to post it here:

Pretty cool, huh?

-- Jeff;

Cloud Studio

Alexsey and Tatyana from Cloud Services dropped me an email to tell me about the beta release of their new Cloud Studio product.

Cloud Studio is a Java application for the management of Amazon EC2 instances. It features a multi-pane interface with a list of available AMIs, a list of running instances, and access to keypairs, security groups, and  IP addresses. Menu options are provided for image registration and deletion, keypair manipulation, security group editing, and IP address assignment.

The application can be run standalone or it can be run from within Eclipse.

You can see a Flash demo on the home page, or you can simply download it.

-- Jeff;

New FPS Marketplace Widget and More FPS Features

The new Amazon FPS (Flexible Payments Service) Marketplace Widget gives developers the ability to create a widget which can move money between two other parties, with complete control of fees paid to the developer. Money moves from buyer to seller, and the seller pays a fee to the developer. The fee can be a fixed amount and/or a percentage of the transaction.

This new widget is ideal for creating shopping carts, e-commerce platforms, and marketplaces. Prior to the introduction of this widget, developers would need to make multiple calls to FPS and to juggle several tokens to implement the same functionality.

Once you are signed up for FPS, you can use our interactive builder tool to create the HTML for a Marketplace Widget in minutes.You'll need to write a bit of server-side code to insert the proper request signature into the HTML, but that's about it.

We've also added some new features to FPS itself and to the existing Pay Now Widget:

• API-level support for the fee mechanism used by the Marketplace Widget.
• The Pay Now Widget now supports "reserve and settle," allowing funds to be reserved on a credit card without making an actual charge.
• Both widgets now support an Instant Payment Notification (IPN) feature. An HTTP POST call is made to a third-party URL each time that a payment or refund is processed.
• A new Refund call automates the process of generating refunds.
• Additional transaction details, including the buyer's email address, are now available to sellers.

-- Jeff;

Help Wanted: More AWS Job Openings

Earlier today I met with the AWS Platform (AWSP) team.

This team is responsible for the heavy duty infrastructure pieces which are common to our line of web services. These vital pieces of our infrastructure take care of authenticating and authorizing requests for AWS services, capturing usage information, billing customers for usage, analyzing large volumes of data for internal and external reporting, and so forth. Needless to say, these internal components need to be perfectly reliable, highly scalable, and very efficient. The team is also responsible for SQS, the Amazon Simple Queue Service.

The team is in expansion mode and wanted to make sure that the world was aware of their open positions. Here's what they are looking for:

Principal Software Development Engineer - "As a Principal Engineer on the AWS Platform team, you will drive, architect and implement core functionalities in the Billing, Accounts, Products and Payments domains that has stringent Service Level Agreements (SLAs) for Availability, Reliability and Performance. You will play a critical role in shaping the overall structure of Amazon’s web-service offerings."

Software Development Engineer - "As a software developer on the AWS Platform team, you will work independently with software engineers and program/product managers to create distributed applications and services, front-end API’s for developers and website owners to consume and be responsible for the design and development of various aspects of the AWS Platform."

Software Development Engineer, Amazon SQS - "As a software engineer for SQS, you will find some challenging projects from our rapid growth that need creative solutions.  You should be somebody who enjoys working on complex system software, is customer-centric, and feels strongly not only about building good software but about making that software achieve its goals in operational reality."

Technical Program Manager, Authentication and Authorization - "The AWSP Authentication and Authorization team is seeking a dynamic, entrepreneurial Technical Program Manager (TPM) to drive the technical strategy and execution of key security services. As a key member of a small team, you will have the unique opportunity to influence and shape the entire AWSP strategy.  You will have complete end-to-end technical responsibility for the component you own from defining the product roadmap to initiating, defining and executing the projects necessary to make the roadmap a reality."

Technical Program Manager, Business Intelligence - "The AWS Platform (AWSP) team delivers business intelligence to over 100 internal customers, and the technology gets leveraged to deliver reports to a diverse community of external customers. Amazon Web Services has a culture of data-driven decision-making, and demands business intelligence that is timely, accurate, and actionable. If you join the AWSP team your work will have an immediate influence on day-to-day decision making at Amazon Web Services."

You might want to learn more about Amazon, about our values or our benefits.

As someone who has been at Amazon for almost 6 years, I have to say that this is truly a great place to work! The work itself is varied, interesting, and ever-challenging. There's a very deep, and very solid technical and managerial backbone in place. People aren't afraid to dive deep in order to solve thorny problems, and are rewarded for doing so. Our environment is friendly to dogs, and we like to have fun while we change the world.

-- Jeff;

Amazon SimpleDB Case Studies - ShareThis and Alexa

Even though Amazon SimpleDB is still a beta product, progressive developers are already learning about it and building highly scalable applications. In fact, we just released a pair of case studies.

ShareThis has been deployed to over 30,000 web sites. Faced with rapid growth, the team considered three storage options and chose SimpleDB for  its responsiveness, reliability, zero software cost, minimal staff costs, and low barrier to development. They used EC2, SimpleDB, S3, and SQS to build a complete loosely coupled and fault tolerant system in the cloud, with an estimated savings of $200,000. Read all about it!

The Alexa Site Thumbnail  Service uses SimpleDB to store intermediate status and log data, allowing them to store and deliver millions of thumbnails. They store over 12 million objects in SimpleDB and perform over 5 million queries every day. Read all about it!

-- Jeff;