Amazon Web Services Blog

New Amazon Mechanical Turk Features...

The Amazon Mechanical Turk team has been hard at work on a couple of important new features: credit card funding and transaction history. Both of these features are available from the Your Account page.

With the debut of credit card funding, Mechanical Turk Requesters (the people and organizations who create the HITS) can instantly add funds to their accounts using a credit or debit card or from an existing balance in an Amazon Payments account. It is no longer necessary to  fund work using a bank account.

This new feature simplifies and accelerates the process of getting work done. Requesters can pre-fund their account at any level (the minimum is one dollar). Once this is done, the prepaid balance can be used to create and pay for work right away.

You now have access to your detailed transaction history from Your Account as well. You can check on your account balance and you can see the last 18 months of prepayments into your account and payments to workers. You can even download this data in CSV form.

-- Jeff;

Update on the Search for Steve Fossett (Update 3)

Update 3 We know many of you are anxious to learn what happened to your search submissions. Full details on areas screened and passed on to the Steve Fossett Search and Rescue team are at: s3.amazonaws.com/fossett/index.html.

Important Update2! Thanks for the great feedback. We found this great tool for submitting your coordinates manually - http://www.violentskies.com/search-for-steve-fossett/index.html. Users are encouraged to use this tool as it would help us aggregate and organize the search results and most importantly not miss any results because every response is important.

Important Update! Thanks to all the thousands of people making this a full-time job! Please read these guidelines before reporting a find here. (1) If you find a likely location, submit it as a "yes" in Mechanical Turk. Please do not post here! The team is reviewing, and most of these "likely" observations are being reported by mutiple people. Only by using the official process are we able to aggregate feedback on a given tile of imagery. (2) If you wander off in Google Earth looking around, it is easy to wander out of the search dataset and into old images that are not relevant to the search. Another reason to register and use the images in Mechanical Turk per-se, rather than flipping back and forth between Google Earth. (3) However if you still want to post a manual set of coordinates, best to do so at http://turksearch.wordpress.com

Recently I blogged about the use of Amazon Mechanical Turk to scale out the search for Steve Fossett. As a pilot who used to be involved in traditional Search and Rescue efforts, this form of innovation is incredible. Someone's fate no longer depends solely on a relatively small number of aircraft; instead a large number of people can quickly search and cross-check high-resolution images.

If you have not yet helped in the search, please visit www.mturk.com to pitch in.

So it seems appropriate to provide some feedback to all of the tens of thousands of people who have helped in the search. Accordingly, the following is straight from the product team:

Workers: thanks for your continued support!

Since launching the search for Steve Fossett, the support from the Mechanical Turk worker community has been outstanding. To date, tens of thousands of Steve Fossett searchers have completed millions of HITs. Your support in reviewing hundreds of thousands of images, covering thousands of square miles of terrain, has generated a number of highly credible leads. This effort has filtered the images down to less than 0.03% of images received.

After you submit each HIT, Mechanical Turk compares your answer to those received from workers looking at the same image. As highly rated images are identified, experts review them using Google Earth and pass the most promising leads directly to search teams.

Images are made available through the support of providers GeoEye, DigitalGlobe and Sanborn. Teams at Google prepare the raw images into the tiles. Data is then processed using EC2 and stored in S3 before being made available through Mechanical Turk.

Workers have already reviewed hundreds of thousands of images from GeoEye, and are now working on imagery from DigitalGlobe. We've just begun processing Sanborn data and will begin loading new HITs shortly.

Your continued participation is sincerely appreciated. Thanks for all your support.

So what have they found? Here's a couple of screenshots...

Experts reviewing each image have immediate access to the comments provided by workers as well as the percentage of workers who flagged the image.

After workers submits their HITs, responses are reviewed by experts using Google Earth to better examine the surrounding areas and evaluate each lead.

-- Mike

Help Find Steve Fossett

Please help find Steve Fossett! Steve's disappearance has been all over the news, but so far no trace of him has been found. You can help search, by using Amazon Mechanical Turk. Several organizations have pulled together to use recent satellite imagery for this effort. The image on the right-hand side of this page is one of the actual images that I just looked at as part of my contribution to the search. Have to say that some of this terrain is incredibly rugged.

Here are some highlights of the imagery, copied from one of the "HITs" in Amazon Mechanical Turk:

• The images shown below are recent and of the relevant search area.
• Each tile is roughly 256x256 pixels, representing an area of 85m or 278ft square.
• Steve's airplane will appear in the images as a 30pixel wingspan and 21 pixels in length.
• We're asking between 5 and 10 people to review the same images so we can be certain nothing was missed.

A cool feature of this particular set of hits is that you can view the satellite imagery in Google Earth. But a note about that... I am using a new computer, and Google Earth was not installed. I had to restart Internet Explorer before the links functioned properly for Google Earth integration. Think it's a mime type association issue.

This search effort may sound familiar--a few months ago the same sort of effort was launched to look for missing Microsoft Researcher Jim Gray. Unfortunately no trace of Jim was ever found.

The basic idea of Amazon Mechanical Turk is to deliver a highly scalable workforce for tasks. In this case, a large number of satellite images were injected into the system for people to review. Because each image is made available to multiple people for review, there is a de-facto quality check built into the workflow. While the example at hand is unfortunate in its circumstances, its existence does showcase the innovation of Amazon Mechanical Turk. Take a look at the other work in the system to see how others leverage this workforce.

I'm a pilot, and within the General Aviation community there is a long-standing debate around the subject of ELTs, which is short for Emergency Locater Transmitters. These beacons were originally required by law after a couple on members of Congress went missing in Alaska. (They never were found.)

The original beacons were simple analog radio signals that transmit on 121.5 MHz, the universal emergency frequency for aircraft. Satellites in orbit also monitor for signals on this frequency. A number of pilots maintain that any beacon at all is a waste of money; if you're old enough, this is reminiscent of the airbag and seatbelt rhetoric of years ago. The problem with the old technology is false alarms: over 100,000 PER YEAR.

Modern digital technology is now used, often coupled with a GPS receiver to make it much easier to pinpoint a downed aircraft. Problem is that these new beacons are extremely expensive. Anything certified for aircraft is insanely priced--and currently 406 MHz beacons cost over $1,000 per plane. That's a lot in aircraft that are flown occasionally for recreational use.

If you want more info on the two technologies, I've posted an article about the two ELT versions on an aviation website that I run called PopularAviation.com.

Finally, as a pilot who has participated in a number of Search and Rescue missions, the probability of the wreckage looking like an airplane is low. This image shows what the plane would look like if it is resting horizontally after a normal landing. In fact wreckage usually looks more like random junk. Anything that looks out of place should be considered "suspect" in these images.

Mike

Some Cool HITs

There are some really interesting Mechanical Turk HITs available right now. Using a Flash application which is actually embedded in the body of the HIT, workers tag a sequential series of images to denote the presence of certain road features -- yellow lines, manholes, drains, bollards, and pedestrian crossings.

Read more about these HITs at the OpenGeoData blog, or just go do them.

-- Jeff;

New Mechanical Turk Getting Started Guide

A new version of the Mechanical Turk Getting Started Guide is now available.

This all-new document includes code samples in 6 programming languages:

• Java with SOAP
• C# with SOAP
• Perl with REST
• PHP with REST
• Python with REST and SOAP
• Ruby with REST and SOAP

The samples are also available in the downloads section of the Developer Connection.

-- Jeff;

Sheep Market Thesis

Aaron Koblin, developer of The Sheep Market, has published his thesis as a Word document: The Sheep Market: Two Cents Worth. It is fun to read, with some very far-ranging references and some entertaining diversions. It is interesting to read about Aaron's hesitancy to be just a cog in a machine, then to see him use the Mechanical Turk to actually rotoscope a movie frame illustrating Charlie Chaplin as just such a cog, and finally to the use of the Turk to get 10,000 people to draw a sheep. I didn't know that some of the Turkers had actually taken issue with Aaron's decision to offer the sheep for sale.

After having given many, many AWS presentations in the last 4 years, I do have to say that The Sheep Market is one of those applications which always gets people to pay attention. The productivity factor of "11 sheep per hour" always gets a good laugh from the audience.

-- Jeff;

This guy really likes the Mechanical Turk

Ok readers, let's make this guy a star:

Of course, this video was itself the product of a Mechanical Turk HIT that paid $8 to the star. If you would like to star in a video of your own, there are some $10 "Make a 5 Minute Movie" HITs in there right now (that link won't last forever, sorry historians).

Pass around, link, watch lots of times. I happen to think that this is pretty cool.

-- Jeff;

Making Money with the Amazon Mechanical Turk

Salon has published "I Make $1.45 a Week and I Love It", a very interesting article on the Amazon Mechanical Turk. You will need to watch a short ad in order to read the entire article.

The article covers the historic origin of the Mechanical Turk name, Amazon's rollout of the web site, and requesters like Casting Words and Efficient Frontier, then reviews comments that they have collected from several members of the worker community.

The comments from the worker community are very interesting and reveal the motivations behind the people who process the HITs.

-- Jeff;

Mechanical Turk Requester Interface

A few weeks ago the Amazon Mechanical Turk team introduced a new interface to simplify the process of creating HITs (Human Intelligence Tasks). I promised them that I would blog about it, and it has been near the top of my todo list until now. I'm holed up in a hotel near London Heathrow airport, and I'm going to spend the morning catching up on some blogging.

Introduction

The new Amazon Mechanical Turk Requester Site is accessible through this button:

The button leads directly to the requester site, which looks like this:

Before you can start creating your  HITs, you need to create a Requester account and you need to put some money in it (so that you can pay your Workers). Once you have done this, you can create your HITs using one of two HIT wizards: basic or advanced.

The Basic Wizard

The basic wizard lets your create an entire HIT using one fairly simple page. You simply fill in the fields, confirm the cost (conveniently calculated and displayed in a little  box in the corner), and you are ready to go. I created a simple "Worker Demographics" HIT. The HIT simply asked the worker to reply with the full name of the country where they live. Here's what it looks like:

Once the HIT is ready to go, you can preview it and then (when you are happy with how it appears) you go ahead and create it. It is posted to the Mechanical Turk site, and is immediately available to Workers all over the world. You'll get a confirmation box with all of the salient information:

The Advanced Wizard

The advanced wizard (totally unrelated to anything that you would find in a Harry Potter book), gives you a bit more control over the content and appearance of the HIT. Your HIT can have a free text answer or a list of options that the Worker can choose between. As my plane was landing at London Heathrow last night I saw an unusual sight and quickly snapped a picture. So I thought it would be fun to see if people were able to identify what I saw. Here's the HIT:

Here's the actual HIT in action, as a Worker would see it:

That was, by the way, one of those million-in-one shots. I have started to keep my camera primed and ready so that I am fully prepared to capture interesting stuff like this rather than regretting it forever.

Managing HIT Results

The Requester site also lets you manage the HIT results as submitted by the Workers. My demographic survey has been running for a few minutes; here are the answers that I have received so far:

From here I can approve or reject all or some of the results. I can also view the completed results. I will post the (unscientifically collected) worker demographic data as soon as I have it.

-- Jeff;

ACM Queue: AI Gets a Brain

The newest issue of the ACM Queue contains an article that I co-wrote with Felipe Cabrera (VP of Software Development at Amazon). The article, "AI Gets a Brain", covers the fundamental concepts behind the Amazon Mechanical Turk.

We start off with a historic review of the original Mechanical Turk (covered in depth in Tom Standage's book, The Turk). From there we cover the genesis of Amazon's development work, including an in-depth look at the types of internal tasks that we wanted to use it for. We proceed to look at the various Mechanical Turk objects and data structures, and the system workflow. We end by envisioning some possible uses for the system, and wrap up with a look toward the future.

The print and PDF editions of this issue also contain an interview with Werner Vogels, CTO of Amazon. Werner has a great blog, All Things Distributed.

-- Jeff;

A Walk Through Hit-Builder.com

Earlier today I spent some time using Hit-Builder.com and found myself very impressed with the site and with what I was able to accomplish.

The site implements a complete, requester-oriented interface to the Amazon Mechanical Turk. Using Hit Builder, it is possible to create and issue a series of HITs (Human Intelligence Tasks) with just one visit to the Amazon site. There's no need to deal with source code or with raw XML data.

Before getting started, you need to create and fund a Mechanical Turk Requester account. This is done on the Turk site, and would be necessary whether or not the Hit Builder will be used to create and manage HITs. Note that there's a time delay (usually 3 or 4 business days) between the time when you fund the account and when the funds are available within the account.

Next, you need an AWS Subscription Id and the accompanying Secret Key, which you can get by registering here. Once you've done this, you can log in to the AWS Developer Portal, and click on the button labeled "Your Web Services Account":

Click on "View Access Key Identifiers" and copy down the two keys that you see there (or open up another browser tab).

Moving right along, visit the signup page for the Hit Builder and create your account:

Once you've done this, you can sign in and get started. Use the Manage Account tab to enter your Amazon Subscription Id and Secret Keys:

By the way, those aren't my real keys, so don't strain your eyes trying to figure them out! While you are within the Manage Account tab, you should go to the Account Summary page and verify that your account has been funded --  look for the line labeled "Amazon Payments account balance."

Ok, time to build the HIT! Visit the Build HITs tab, and select Build HIT Group:

From here on out, you need to think of the HIT Builder as a classical desktop application, not as a web site. There are 2 layers of popup windows, and much of the navigation is accomplished by pushing buttons rather than by clicking on links. The UI is smooth and easy to use once you have a good grasp of the elements of a HIT.

At this point it is a good idea to step away from the keyboard and to spend a few minutes designing your first HIT. Good HITs are self-contained, unambiguous, and carefully described. We have seen that new Requesters often interate on the HIT design in order to fine tune the model and to weed out any ambiguities.

My HIT is going to be really simple. I am going to ask people about their favorite vacation activity, which they must select from the following list of choices in a "radio group":

• Stay home
• Sit on the beach
• Take a cruise
• Drive somewhere
• Go skiing
• Build a web site
• Other

Start filling out the form by entering a title and a description for your HIT. When it is time to create the overview, click the "Add" link and enter the overview into the popup:

The next step is definitely the most complex. It is time to actually build the HIT; start by clicking the "Add New HIT" link. This will open up the Add/Edit HIT popup. Enter in some text to introduce the question (in my case I typed "What is your favorite vacation activity?"). Now use the "Add Answer Control" option at the bottom to choose a UI element to collect the answer. I chose "Radio Button - Text", and then filled in the popup:

For each item in the radio group, I entered two values. The first value is the "pretty" value for display, and the second is the value to be returned as the result value for the HIT. If you are going to be pulling this data into another application, you can use this feature to supply that application with the exact set of values needed for it to proceed; there's no need for any kind of mapping layer. I also created a free text answer in case the worker chooses "Other."

The last step is really easy. We need to specify how many workers get to work on this HIT, how much they'll get paid, and how long the HIT will be active.

We could also attach Qualifications to the HIT, but that's not necessary in this case. The Hit Builder makes it very easy to build, manage, and score Qualifications.

I'm going to ask 50 workers to do this HIT, and I'll pay each of them 2 cents. I fill in these values and click "Post HITs", and the Workers will get right to work.

At this point switch over to the Manage HIT Results tab, and choose your HIT Group and your Reviewable HITs:

As the HITs come in, the Hit Builder interface can be used to review, approve, and even chart them:

 

Overall, this is a remarkable handy site, with tons of easy to use functionality. Check out the Hit Builder today!

-- Jeff;