FeedbackArmy

Photo Credit: Waxy.org

Earlier I wrote about my impressions using Mechanical Turk to run the Feedback Army Usability Testing Service. Here I write about my favorite part, dealing with the workers and taking care of them.

What to Pay on Mechanical Turk

What you pay will affect the turn around time of responses and to some extent their quality. Some people won’t even bother if the pay is too low. I found people will review websites for $0.25-$0.30 but I changed the pay to $0.45-$0.50 and stuck with it.

The best way to find this number is to create test tasks using Amazon’s HIT Builder. Try a few numbers and see which ones yield turn-around that you’re comfortable with.

I recommend when creating a new task or service that you include a question in each new HIT asking the workers for feedback to improve the HIT interface and what confused them. The workers will follow your instructions literally and you will want to try a few iterations before going live with them. Getting this feedback can greatly speed up the process.

Mechanical Turk Response Turn-Around Time

With the right pay, responses come quickly. With Feedback Army’s current pay level, ten response jobs are usually completed in one to three hours.

Feedback Army requires unique reviews and this greatly affects turn around time.

Mechanical Turk Response Quality

The response quality varies. Some Feedback Army responses are several paragraphs. I have seen users write a page or more. Usually the workers are out to answer the questions as quickly as possible. Spelling and grammar is sometimes poor.

Just like E-Bay has a seller rating–Turkers have a task approval rating. You can set a minimum approval rating for your tasks. 95% or above is near impossible for workers to keep. I tried 85% for a while but eventually lowered it to 70% to open the field to more workers. I have not noticed a change in quality from doing this.

No matter what–you or your customers will need to moderate the responses. Sometimes workers accidentally hit submit and you end up with a blank or incomplete response. I’ve put the power to reject poor or unacceptable responses into the hands of my customers and we have both been happy with the results. Less work for me and more control for them.

Open ended questions work best and draw out longer responses. Also the worker’s interest in the site affects the response. I saw a “how to get women into bed now” site posted to this service and the responses were passionate, useful, and quite entertaining (most Turkers are women).

Know the Workers

As I said in the last paragraph, most Turkers are women. It helps to understand the demographics of the workers. I found an excellent study awhile ago and according to it must Turkers are in their 20s, are from the United States, and have college educations. Mechanical Turk does not have a direct mechanism for restricting tasks to specific demographics.

Several Turkers interact on a bulletin board called Turker Nation. I recommend that you introduce yourself on the Turker Nation board. They will answer questions about what works and doesn’t work for them.

Also, if you are unfair–they’ll add you to the hall of shame and you don’t want that.

Final Thoughts

Pay affects the quality and speed of responses. Remember that Mechanical Turk is an open market place and workers can choose to accept or reject your tasks. Pay fairly, listen to the feedback from your workers, and plan for moderating the responses. If you do these things you’ll be impressed by what you get.