I’ve used Mechanical Turk for several projects including Feedback Army. I’m often asked for my impressions of the service and what worked for me. Here are my thoughts on the API and support behind Mechanical Turk.
I use Amazon’s Mechanical Turk Java SDK to deal with the service. I’m able to do what I need with the API but it is confusing and not well documented. Several times I’ve had to dig in their source code to find certain options.
I’ve also dealt with more XML verification issues than I care to think about. I’ve found the ExternalQuestion mechanism is the way to go. Through this you can direct your workers to an external HTML page that you design and control. This is simple to do and helps avoid these dreaded errors I speak of.
The SDK requires an insane amount of external libraries. Fortunately these libraries are included. A more complex application that uses different versions of these libraries may have issues. If this is you, I recommend creating a light-weight service that runs stand-alone and acts as an interface to the Mechanical Turk service.
Support from Amazon is non-existent. There is a forum for those working with Mechanical Turk but reading it shows cries for help with few answers. One time, a power outage knocked out my server and Feedback Army stopped working. I was at a loss because my code had not changed. I kept receiving an “InternalServiceException – Request has expired.” error. I later learned that my server’s clock was out of sync and correcting this fixed the error.
In the year I’ve run Feedback Army, Mechanical Turk has proven stable for me from a software point-of-view. The SDK is difficult to work with and the support is non-existent. However once you get your solution developed, you’re good to go. It will work fine.