Amazon Web Services Blog

phps3tk - Command Line Scripts for S3 Access

I met dozens of entrepreneurs and developers at last night's Start-Up Seattle event (you can read more about it here, here and here).

Among those I met were David Busby of Edoceo.com . Through his company, David provides small and medium-sized businesses with IT solutions powered by open source technologies.

David told me that he had written an Amazon S3 access toolkit in PHP. I urged him to release it sometime soon, and promised him that I would blog about it. Apparently my arm-twisting was successful -- earlier today he announced that phps3tk was now available!

phps3tk provides an API and a command-line interface to Amazon S3. Once installed, you can use commands like "s3 ls" to list bucket contents, "s3 get" to retrieve data from an object, and "s3 put" to store data in an object.

There is also a programming interface wrapped around the S3Cache, S3Connection, and S3Response objects.

Nice work, David!

-- Jeff;

Amazon S3 Wins a Codie!

Flash! Amazon Simple Storage Wins a Codie! I'm posting this from the Web 2.0 Expo, which can only be described as having incredible energy and way more people than I expected to see. However this piece of news comes from a nearby event.

The Codie Awards are the Oscars of the software business, and are voted on in two rounds: first by industry journalists, and then SIIA member companies. In 1986, the Codie Awards were established by the Software Publishers Association (SPA), now the Software & Information Industry Association (SIIA), so that the software industry could evaluate and honor each other's work.

This evening's award ceremony was held at The Palace Hotel in San Francisco, CA. This year’s winners were chosen from more than 1,197 nominations – the most in the history of the Codies, according to their press release.

Amazon is honored to receive this recognition for a service that now boasts over 5 billion objects stored in it.

--Mike

Amazon Unspun Widgets

Ok, I'll be the first to admit that this is stretching the definition of a web service just a bit, but let's give it a whirl anyway! Next thing you know I'll be posting about dogs or something.

The new Amazon UnSpun widgets make it easy for any blogger or web site owner to host polls on their site by pasting in a single line of HTML. The polls can be based on existing UnSpun topics or from a topic freshly created for the poll.

Here's an instantiation of a widget with the "Best Books about the LAMP Stack" poll inside:

You can read all about the UnSpun widget here.

-- Jeff;

WeoCeo Video

There's a new WeoCEO video available. After watching this video you will be able to understand how the WeoCeo product dynamically starts, monitors, and terminates Amazon EC2 instances in response to actual machine load while maintaining a constant, externally visible domain name.

You'll need to be familiar with the Unix command line in order to appreciate this video! Follow closely (there's no audio track) for best results. If you don't know too much about the command line, rest assured that the cpuburn command doesn't actually start fires; it simply keeps the processor 100% busy! When the WeoCEO detects that the existing set of EC2 instances is too busy (as measured by the load average) it spawns additional instances (up to a predetermined maximum) to alleviate the load.

In just a few minutes you will see how WeoCEO is able to offer "spike insurance", automatic load balancing, and stable IP addresses.

-- Jeff;

Enomalism Amazon EC2 Migration Module

The Enomalism Amazon EC2 Migration Module supports bi-directional migration of virtual machine images to and from Amazon EC2. In other words, you can easily and transparently move running images back and forth between your local Xen-based computing environment and Amazon EC2.

Using this tool you can do all sorts of interesting things including creation of hot spares, backup snapshots, load balancing, and access to additional compute capacity to handle sudden, unanticipated bursts of traffic. The product also converts between EC2 AMI's, VMWare images, and the Enomalism Xen format using command-line tools. There's support for VMcasting, and plans are afoot to support and run Windows, Solaris, and BSD Unix in the future.

Available in standard ($295, 5 EC2 instances), Professional ($395, 15 EC2 instances), and Enterprise ($1495, unlimited EC2 instances).

Check it out!

-- Jeff;

Tracking AWS Releases

We were sitting around the office the other day, munching on some M&Ms and lamenting the fact that we had to actually go and check the various AWS Forums each day in order to stay abreast of the newest releases of each product. We knew that each forum had its own RSS feed, but the idea of having to check 10 feeds each day for announcements certainly seemed sub-optimal.

Minutes later, Jinesh had put together a Yahoo Pipes application to do the heavy lifting. His application pulls in the requisite feeds, finds the announcements, and combines them into a single result feed for easy reading. Just pop that feed into your newsreader and you'll always know what's happening.

If you want to use RSS to do something similar, take a look at Brian and Craig's article, Generating RSS Feeds with the Amazon E-Commerce Service.

Nice work, Jin!

-- Jeff;

Another ECS Release

The Amazon E-Commerce Service has been updated! New features include:

• ItemSearch now returns products from 25 search bins. The former limit was just 10 bins.
• More Variation dimensions, including GemType, HardwarePlatform, OperatingSystem, ScentName, and TotalDiamondWeight, are now returned, as appropriate.
• New search indexes are supported, include Toys in France, Watches in Japan, and Sporting Goods in the UK.
• The ECS Developer Guide has been totally rewritten.
• A new document, the ECS Getting Started Guide, is now available.
• The release notes are now available in Japanese as well as in English.

We also fixed some bugs!

Read the release notice to learn more.

-- Jeff;

PS - It is hard to believe, but we are coming up on the 5 year anniversary of the release of what was first called AWS, which later morphed into ECS. Per the AWS FAQ, the first release took place in July of 2002.

New Video: EC2 + Ruby + Elastic Rails + Capistrano = Scalability

In this new video (Flash or QuickTime), AWS developer Steve Odom shows how to use Elastic Rails and Capistrano to create and deploy Ruby on Rails applications.

 

-- Jeff;

PS: The first commenter asked us to add a Pause button to the video presentation earlier today, and we have done as requested. You won't get anything if you don't ask!

Another Database Solution for EC2 - S3DFS and SQLite

An anonymous poster at the (very entertaining) Nanobeepers blog, reports on the use of Amazon EC2, Amazon S3, and the SD3FS file system from OpenFount to host the SQLite database. As he notes:

So, by adding EC2 instances that are all attached to the same S3 bucket, and using round-robin dns to distribute traffic among them, you can theoretically scale a web service infinitely. Just launch a new EC2 instance and add it to the round-robin dns pool and you're done! Even that could be automated to respond dynamically to traffic.

There's some additional information in the discussion thread.

Thanks to Jin for pointing me at the original blog post!

-- Jeff;

Fastest way to Get Started with EC2 - EC2UI

In this age of speed where Electric Cars go 0 to 60 in 4 seconds, I found the fastest way to get started with Amazon EC2 :

.... With the new Firefox Plugin : EC2UI

Firefox lovers will love it. It takes less than 4 seconds to download and install (unless you are running at stone-age dialup speeds). Yes! its a Firefox plugin for Amazon EC2. Created by one of our ingenious Amazonian developers and the fastest way for anybody who would want to get started with Amazon EC2.

Point your Firefox browser here, Install the Plugin and restart Firefox, Key in your AWS credentials and Start spawning Amazon EC2 Instances.

The plugin currently allows you to Manage AMIs, Manage your Instances, Keypairs, Security Groups and Permissions. View all the available public and private AMIs, launch your AMIs, terminate your instances - all within the powerful Firefox environment.

-- Jin

MySQL Interface to Amazon S3

Independent developer Mark Atwood has been working on a MySQL interface to Amazon S3. Released under the GNU Public License, the code is compatible with version 5.1 of MySQL. Once the interface has been installed and configured with your AWS developer credentials,  you can now create tables using the AWSS3 storage engine like this:

CREATE TABLE atst (s3id VARCHAR(255) NOT NULL PRIMARY KEY, s3val BLOB)) ENGINE='AWSS3' connection='awss3 bucketname aws_id aws_secret'

This is a bleeding-edge, first-cut release and, as is the case with popular open source projects, will undoubtedly evolve and mature rapidly over the coming weeks and months.

Based on Mark's S3 journal entries, the basic functionality is now in place. Each database table row is stored in an S3 object. The object's S3 key corresponds to the table's primary key (which must be of type VARCHAR). Inserts, deletes, and selects are functional.

Mark's code is stored in a Mercurial repository and can be found here.

Congratulations, Mark; this is awesome.

-- Jeff;

Links for Thursday, April 5, 2007

A few things I couldn't wait (waiting being defined as having time to write real blog entries) to share:

• Mitch Garnaat: Monster Muck Mashup - Mass Video Conversion Using AWS - Over the past year or so, Amazon has been expanding it's line of infrastructural web services. Amazon CEO Jeff Bezos likes to call this collection of services muck, meaning these kinds of services are difficult to build in a scalable manner. That's exactly what this article will focus on: combining three of these scalable services from AWS using an architecture that allows us to build robust, reliable, and scalable compute services.
• Jeff Kim: Net::Amazon::EC2 - Perl interface to the Amazon Elastic Compute Cloud (EC2).
• Surj Patel: Racks and Pipes for Pennies - If you have any pointers to good uses of EC2 / S3 / Utility computing or know of any good startups powered by this tech please let others know by posting below.

-- Jeff;

Monday Links

Here are some interesting Monday morning links:

• Smugmug's Don MacAskill has posted the slides from his recent ETech talk. Per the slides, they now have 192TB of photos stored in Amazon S3! Don's deck also documents cost avoidance of $692K to date. After reviewing several different approaches to combining local storage with S3, Don discusses performance and reliability, and also talks about his plans to use Amazon EC2 to handle image processing for 500,000 to 1 million photos per day.
• The newest release of TeamDirection stores project information in S3. Check out the podtech video to learn more, then go for the free trial download.
• SmartSheet is another cool and functional application built on top of S3 and EC2. There's another podtech video (how does Scoble do it?). SmartSheet also won the Best in Show award at the recent Under the Radar conference.

Enjoy,

-- Jeff;