Amazon Mechanical Turk Status Update

It's a good time for a quick update on the status of the Amazon Mechanical Turk beta test!

Overall, the beta test is going extremely well. The number, type, and variety of HITs in the system continues to grow. We have already seen the following types of HITs:

• A9 BlockView Image Adjustment - Workers examine a number of candidate images and choose the one which contains a specified organization.
• Confirm Artist Name - Workers examine an album cover and extract the name of the artist (or artists).
• Extract Product Description - Workers summarize a block of text into a series of bullet items.
• Select Japanese Text Direction - Workers read a block of Japanese text and respond with the text direction (left to right or top to bottom).
• Podcast Transcription - Workers listen to a short segment of a podcast and transcribe it to text, basically performing a "speech to text" function.
• Language Translation - Workers translate English to French or vice versa. A related type of HIT allows workers to verify the results of such a translation.
• Web Site Review - Workers examine a web site and answer some survey questions about the site.

We are working with a number of Requesters to get even more, so keep your eyes on the list of HITs . These Requesters are very creative, and they tell us that they have a lot of pent-up work. This work should start to appear later this month and throughout January.

The number of qualified Workers is also growing, many of whom are able to do hundreds or even thousands of certain HITs per day. The Worker community has been efficient and conscientious about doing a good job, taking on an impressive quantity of work and returning it with a very high acceptance rate.

Mechanical Turk is still a beta-level product, and we are learning a lot every day as we watch the activity on the site, the multitude of blog posts, and the traffic on the various discussion forums. We are taking in all of this feedback, and it is feeding directly into our product plans. Our goal is to make the entire Mechanical Turk experience pleasant and profitable for Workers while providing Requesters of all sorts with a highly qualified and motivated work force. Getting all of the parts into balance will take some time, but we are making great progress.

There are a couple of issues that are worth a bit more discussion: zero-valued HITs, "bots," and blocked accounts:

• Some members of the community have expressed concern over HITs that have a zero payoff. These HITs are placed in the system for testing purposes by Requesters while they are building their application. Of course, no Requester expects to get work done for free. There will be a better way to load these "test HITs" in the future. It is important to remember that Workers are free to ignore any HITs that are not to their liking or that don't seem to pay enough to be worth doing. The Requesters have the ability to adjust HIT pricing to manage supply and demand.
• A few Workers are trying to run "bots", which you can think of as mindless HIT robots. While we do recognize that some high performance Workers use a variety of scripts to simplify and increase the efficiency of the Mechanical Turk workflow, we do need to point out that the use of fully automated bots runs counter to the Mechanical Turk Participation Agreement. Furthermore, it hurts the entire community by lowering overall quality as well as the amount of available work on the site.
• In some cases we have blocked accounts when it appeared that a bot, rather than a real live human being, was doing the work. We have subsequently unblocked accounts where we believe the activity has actually been legitimate. If you are not using a bot and you believe that you have been blocked in error, please send email to mechanicalturk-account@amazon.com post a note via the Contact Us Page to get our attention. Be sure to provide us with your account name and an explanation of any special techniques that you use to do your HITs.

--Jeff;

Note: This post was updated on December 15th to indicate that the Contact Us page should be used instead of the email address.

Comments

[Aaron Doucet]

THANK YOU for posting about blocked accounts--it's good that I don't have to be afraid of doing 800 music hits/hour as long as I can justify it in e-mail. I'm trying to get $1000 for an iBook and I don't want to be afraid to make any more than $10/day. It's good that you have a way for legitimate users to get unblocked.

[br0adband]

Thanks for the information, Jeff. I know you're busy, so I want to clarify your statement:

"A few Workers are trying to run "bots", which you can think of as mindless HIT robots. While we do recognize that some high performance Workers use a variety of scripts to simplify and increase the efficiency of the Mechanical Turk workflow, we do need to point out that the use of fully automated bots runs counter to the Mechanical Turk Participation Agreement."

If possible, can you absolutely clarify that statement that script assistance is allowed to increase our efficiency and productivity as long as they do not automate the entire process start to finish?

I hope your answer is yes, because there are a lot of us that are willing to do the work but think we can improve it client-side and get more done.

Thanks for the informative post and for keeping the community up to date.

bb

[Sarah]

"A few Workers are trying to run "bots", which you can think of as mindless HIT robots. While we do recognize that some high performance Workers use a variety of scripts to simplify and increase the efficiency of the Mechanical Turk workflow, we do need to point out that the use of fully automated bots runs counter to the Mechanical Turk Participation Agreement. Furthermore, it hurts the entire community by lowering overall quality as well as the amount of available work on the site."

Thank you for communicating. I hope the scripts vs. bots issue is clarified soon. I see no logical reason why scripts would be verboten, being that they are merely helping the workers work more efficiently. Bots, however, completely defeat the purpose of MTurk.

[Jamey]

Glad to see some comment being made regarding using a script to complete the task better and more efficiently. I agree, that a 'bot' completely bypasses the whole point of Mechanical Turk.

I am concerned when members of the community write to the feedback addy asking about scripts, and get a blanket statement;

"Greetings from Mechanical Turk.

Scripts are not allowed.
[edited for brevity]
Thank you for participating in Mechanical Turk.

Best regards,

Michael
Amazon.com Customer Service
http://www.amazon.com "

Now, I enjoy doing the HITs, but found myself locked out of my account tonite. I have written to the email address you listed, Jeff, and hope to have a reply soon. My bleeding eyeballs can attest to the fact that I am indeed a (the!) human who is doing the work. I think I have learned more about music in the past week than I have in a lifetime.

Yes, there will be people who try to cheat the system. I feel that the community working together with MTurk and the requesters will be able to work out a method to put a stop to that. There are a lot of talented people out here just waiting to work.

Thanks again for this very cool opportunity,

Jamey

[Joey Thompson]

I love Mechanical Turk, especially the music HITs. Keep them coming, and we'll keep doing them.

[Barrett]

I'm really glad that you're being very communicative with the turking community, Jeff. I appreciate it more then I can say.

I have a question though that I'd love to hear from you on.

The number of music hits have DRASTICALLY decreased. There's a real struggle for them right now, and I was wondering what the reason was. Is mturk.com's staff just late on those huge, replenishing dumps, or are we about to see a new type of hit, or maybe a hit already mturk.com, just not as available as the music hits, and that will become main hit type in just a short time?

[Joan Landy]

I have been turking for a while, and also really like the music HITs... However, something has happened today. I'm not sure if it's bots or what, but all of the music hits are gone! I used to be able to do a sizeable number of them in an hour, with good accuracy, but now I can't even manage 100 in an hour, and that makes it not at all worth my time.

It is because there are almost none in the category, even though it gets very slightly replenished all the time. If this is intentional, please reconsider! People won't do it for long.

Please put more up at a time, or else get rid of the bots so we can get back to work!

[Moonho]

Hi Jeff,

Why don't you insert random captcha(http://en.wikipedia.org/wiki/Captcha) questions between every 10-100(random) HITs assigned to someone?

In case he(or bot) fails to answer in 30-60 seconds, you may ask 2 or 3 more questions and if he(or bot) fails again, you may block his account for about 1 hour.

In this way, bots can be neutralized effectively.

[Samuel]

Thanks for keeping us up-to-date. The more info we can get from the people behind this, the better.

A question, though: A9 Image HITs have dried up and left us, and Music HITs are heading down that same route. What I - and I imagine, hundreds of other Turkers - am concerned about is the future of these sort of HITs. That is, HITs we can do lots and lots of, and make a decent amount of money doing so. Can you give us any sort of information about when we'll be seeing more Image or Music HITs? I, for one, miss the days of thousands of image/music HITs. Fighting for HITs is just frustrating.

Thanks for providing this service, and thanks in advance for any information you might share.

[Kristina Kiser]

Jeff,

I am writing in regards to the use of scripts for completing the Mechanical Turk duties. I believe that the auto-accept process is hurting the whole system as this practice seems to block others who are trying to participate. What ever happened to ethics in business? It seems in a way that auto-accepting and scripts are a way of cheating, much like lying of one's abilities on a resume. Really, without using auto-accept(which I don't) the site is totally inaccessable. Boooo to all you scripters.

[Aaron Doucet]

It's not the auto-accepters, it's the botters. All auto-accept does is make sure you get the hit (because otherwise it's nigh impossible to grab them.)

[andy]

Jeff -

I just found out my account was banned, and I have no idea why. I don't use bots, don't complete an inordinate number of HITs, my approval rating was (last time I saw) over 90%, am the only user on my account, and do not have multiple accounts.

The only thing I can think of is that we have a home wireless network with 3 computers, and both my wife and I have accounts. It's an easy way to pass a little time and make a little money toward a book or the like. So, if we're both using our accounts, it would look like multiple accounts at the same IP (which is true, but multiple accounts in use by different people).

In the e-mail response I got it, the answer was basically "your account is closed, it won't be reopened, but why not go open another one?"

Well, because I don't see much point in earning some money only to have it denied to me again when I'm falsely accused of wrongdoing.

If I've been gaming the system, I must be pretty dumb since I've netted a whopping $16 in two months.

Any ideas or help you could provide would be appreciated. Thanks.