Photo by: PinkMoose

Recently I was asked via email for my methods marketing my online business. I’ve tried a lot of things to promote Feedback Army. Here is a quick list of the methods I’ve tried and how well they worked for me.

Cold Emails

I’ve had a lot of success with cold emailing. To find prospects, I visit buysellads.com and search for tags related to usability testing and startups to look for blogs that might take interest in Feedback Army. I then visit the blog, view the about page, and see what the blogger has written about usability testing. Once I have this information, I contact the blog owner modifying the first sentence or two of a template email to say something about their site. I also include a coupon for the blog owner to try Feedback Army. I usually see a response from 3/10 emails, which isn’t too bad.

When cold emailing, ask for an opinion or feedback. Do not ask for a review. If the blogger wants to, they’ll review your site. If they give you feedback that makes your business better, that’s helpful too.

I usually target small sites, but even popular bloggers have helped me out in unexpected ways. I once contacted Andrew Lock, the producer of the excellent Help My Business Sucks show. He never wrote about Feedback Army but he did mention it at a conference and this led to new customers.

If you plan to use this tactic, I recommend you read Startup Marketing Advice from Balsamiq Studio. Peldi’s advice help put me on the right path here.

Start a Blog

Now that people are aware of Feedback Army, I’m trying to market it through my blog (hey, you’re here). You’ll notice that I post about once a week. My goal is to be consistent and open in my blogging. I find posts where I’m most vulnerable (read: open and honest) generate the most attention. I seed relevant posts on Hacker News. It’s hit or miss but when it hits, it’s worth it. A hit blog posts also leads to a sales spike for the current day and the day after. Sometimes it also leads to other blogs writing about Feedback Army.

Advertising and Analytics

I’ve tried ads on buysellads.com. It took little effort on my part and it made me feel like I was doing something to market my business. However the party ended once I installed google analytics and started tracking “goals”. I noticed my ads weren’t converting to sales. After A|B testing variations of my ads, I decided to abandon buysellads.com and Google AdWords.

Before you buy any advertising, I recommend that you install Google analytics and set it up to track whatever you consider a conversion or goal. It’s easy to do and will help you decide if you’re seeing an ROI from your advertising.

When exploring advertising opportunities, make sure you set a time limit, a budget, and a have a means to measure the effectiveness of the ad.

Cold Calling

I’ve also tried cold calling.  To do this I decided on a target (small web development shops). I chose these smaller businesses as I felt I would be able to build rapport with them easily. I used google to put together a list of names and numbers. I also wrote a script to guide me through the beginning of the conversation. Before my first call I practiced this script with my cell phone held up to my ear until I felt confident and automatic saying it.

When cold calling, it’s important to know what your goal is so you can steer the conversation there. My goal was to get an email address so I could send a Feedback Army coupon worth ten responses. The callee wins and I win because I’m able to track how many coupons were redeemed and how many sales came from it.

The people I spoke with were pretty cool and the conversations weren’t awkward like I initially feared. Despite the favorable chit chats, the conversion rate from this experiment was terrible. For a startup with a low-cost product like Feedback Army, I do not recommend cold calling.

Affiliate Marketing

I tried setting up an affiliate sales program for my mechanical turk worker base. The idea was to give each worker a URL that they could refer people to. My software would pay a bonus to the worker when someone they referred bought feedback. I was very excited when I put this program in place. Unfortunately it ran into a few difficulties:

1) The profit margin on Feedback Army was too low to support an effective affiliate marketing program. I had several complaints that I wasn’t offering enough of a reward.
2) I believe recruiting the workers on Mechanical Turk was a mismatch as the workers didn’t entirely understand the program.

If you’re thinking of going this route, I recommend designing your price packages to accommodate this beforehand.

Search Engine Optimization

I’ve tried some white-hat SEO stuff. I still only get 10% of my traffic from search engines and it’s not a big money-maker for me. Someone once said if people aren’t searching for your business by name then it’s more important to focus on the product. Make sure some people are happy with what you’re doing before you focus on SEO. No organic growth is a bad sign.

That said, you should check your basics to make sure you’re not doing anything that will hurt you. If your business can accommodate it, look into finding a way users can share what they bought or build out the content on your page. For example, I’m planning to create a badge to let clients share their Feedback Army results on their sites. I don’t know how well it will do, but as you can see from this post–marketing is about trying a lot of things.

Meetup.com

Make sure you visit meetup.com and look for events in your area. I have tiny cards I printed via moo.com that I hand out to the folks I meet. In the past I haven’t seen a lot of conversions through this approach (although I’m not very aggressive about going to events to solely market feedback army)–but it is a good way to get your idea in front of people and get instant feedback. This is easy to do and if you’re an extrovert (like me), it’s a lot of fun. If you live in an area with a startup community, look for events where you can pitch your business to a crowd. This is also a great way to get exposure.

Customer Service

Feedback Army’s sales continue to grow each month. To keep this up, I focus on the most important marketing tactic of all: delivering a good product and providing good customer service. When my software detects an error or a customer uses my contact form, it goes straight to my cellphone. This way I’m able to address issues quickly. Good customer service has led to these customers coming back. I believe this is responsible for the growth of Feedback Army.

Final Thoughts

The world of marketing is filled with options. As you can see, it’s also a lot of work. Don’t let yourself get lulled into believing there is one magical thing that will build your business for you. An off-shore team offering to “increase your SERPs” for $30 is not going to make your business. Once you get going your efforts will pay off. Remember that you may have to try a few things and don’t expect every campaign to succeed. Ultimately, your business will be made by delivering a good service and acting like a responsible netizen. Don’t forget that.

Photo by: springhill2008

Recently, I was looking for an accountant using Yelp. I came across a review where the reviewer asked simple questions and the accountant kept avoiding straight answers. Worse, the accountant was non-committal and kept exclaiming “I’m an honest accountant” as a reply to every question. This woman’s behavior didn’t give her customers confidence in her as an accountant.

Part of being in business is dealing with the responsibility of dealing with things that can go wrong. It’s important to realize that fear of these bad outcomes exists and to realize you can deal with it when it happens.

I once bought eye glasses from a nice boutique place in Syracuse, NY. The glasses took an extra few weeks to get to me. Why? It turns out the frames I selected were a poor fit for the prescription I had. While making the final pair of glasses, three pairs of lenses were broken. Someone had to eat that cost. It wasn’t me.

Even though these bad situations are a part of business, they’re rare. When they do happen, they have a cost and that cost is usually something that the overall profit margins can absorb.

I used to work as a consultant. One thing I really disliked is I behaved much like the accountant in the first story. I was always thinking about what could go wrong and working with a fear of what would happen if I couldn’t deliver what I had promised. Fortunately the clients knew my work and put up with my quirks.

From that experience I’ve learned. If you’re in business, it’s ok to plant your flag and exclaim “this is what I’m offering”. Own it and be ready and willing to tackle what goes wrong.

I run a website service that gives customers (some big, some small) paid feedback on their site. I have a control freak’s nightmare. I work with a completely anonymous work force. Imagine the level of control I have in this situation. When I started, I had tested the concept and was satisfied it provided value. Yet, I was still pretty scared of it so I priced it way low to compensate. Fast forward over a year later, this service is growing and my anonymous work force has serviced many happy customers. There is a lot of opportunity for things to go wrong.

Once, I had a worker write a very vulgar review filled with swear words. Imagining this situation in the beginning of my service may have been enough to stop me from pursuing it. The customer brought it to my attention and was happy to accept my apology and a few extra responses as compensation. I also added a feature to let me ban misbehaving workers to my system. I learned that bad can happen but when the bad happens, it’s also a chance to shine.

Part of being in business is having the courage to own some task and tell the world you can do it. Don’t worry about what can go wrong–it will. When it happens, you’ll find customers are happy to work with you so long as you’re fair and treat them with respect. When starting, focus on inspiring confidence and realize the rest will work itself out.

Photo by: Orin Zebest

I receive quite a few direct messages from folks asking me to consider advertising with them on BuySellAds.com.These messages are almost always examples of poor marketing. I feel for you. If you want to get better, read this post. If you follow these tips, you’re bound to generate more good will and find some buyers. Trust me.

Add Value

Before you contact someone (like myself), ask yourself: what kind of extra value can you create for this person when they deal with you directly. Shotgunning an email with your URL and saying “buy from me! buy from me!” adds no extra value to your service. Contacting someone and offering them something they can’t get through the direct channel, that may get their attention.

When I market Feedback Army via email, I usually offer a coupon for 10 free responses. This is me offering something without demanding something from the person I contacted. This immediately creates some good will. If the relationship ever had a chance, it’s going to do a lot better now.

Maybe you’re asking: “hey, great you have a service… I just have a website and there is no way for me to give my ad space away”. I buy that. But there are things you can do.

For example, you can say “hey, I have a newsletter with 5,000 subscribers. If you’re interested, I can add your link to it one day and we’ll see what kind of response you get from my audience. If it benefits your business, you should consider advertising on my site to reach more of this audience every day”.

I’ve also seen webmasters offer to tweet a “welcome our new sponsor” message to their Twitter followers. Things like this add value over raw CPM and ad space on your blog.

There is a goal to this value add. When you contact someone, you want them to feel special because they received the message from you. Offering something they wouldn’t get otherwise is a great way to do this.

Avoid Irrelevant Facts

When contacting a potential advertiser, stick to facts that are relevant to the advertiser. It’s great that your site is PR5 but since the ads are nofollow type links, that does not benefit the person you’re trying to get to hand money over to you.

I sometimes advertise on sites outside my niche. For example, I ran an ad on a site that caters to graphic designers with stock vector art, Photoshop paint brushes, and other stuff I’ve never used. I do this because I’m curious to see how a different audience will respond to my advertising. I got an email from someone touting how many brushes and other things they had on their site and trying to lure me to them based on that alone. This led me to ask, did they even look at my site? This gets me to my final tip:

Personalize Your Message

When contacting someone, always take a minute to look at their site. It’s ok to send a template email. You’re in business and need to save some time. But always leave room to customize the first few sentences for whoever you’re emailing. Your goal is to make sure they know you looked at their site and you’re not just blasting a message to a bunch of different people.

That’s it. The goal is to make sure the person you’re writing to feels special. Personalize your message, tailor it to your audience (they’re reading it, not you), and finally add value. If you do, you’ll do great.

Photo credt: individuo

Last week I wrote about how to stay motivated and keep blogging. Something I didn’t touch upon is the need to make it easier to blog. Writing a blog is tough work. Not only do you have to create good content, you also have to spice it up, and promote it. Sometimes these side tasks add friction to the writing process.

So another secret to writing more–make it easier to write. I like to think of this as making your writing process usable for you. The more usable your writing process is, the more you’ll do it.

So what tools can make your writing process easier?

1. Post Scheduling – Get Ahead and Stay Ahead

In my earlier post I mentioned the post scheduling feature of WordPress. This makes it easy to build up a lead. I recommend you do this.

2. After the Deadline – Feel Good about Your Writing

Make sure you install a tool like After the Deadline to help with your spelling and grammar. My writing is never perfect but after I run it through AtD, I am comfortable that the obvious mistakes are gone. This is enough to make me feel safe clicking publish.

3. Full-Screen Visual Editor – Avoid Distractions

The full-screen mode  of the WordPress visual editor helps me avoid distractions. With this turned on I feel like I’m using a word processor in my browser. It’s very nice.

4. Find Images with iFlickr – Add Spice to Your Posts

I find installing the right plugins like iflickr makes it easier to get those images in place to spice up the content of my blog posts. I used to avoid this step because it took too long. Now I include something in every post and I’m happy with the result.

5. Make an Outline – Know what Kind of Post You’re Writing

While not technically a tool, this is a good tip. You should know what kind of post you’re writing. What do I mean by this? There are different formats of blog posts that seem to attract eye balls and get the message to your readers. Know what these are, know which type you’re going to write, and make an outline. I knew I was going to write five tips when I set down to do this post. By having an outline you’re giving yourself a clear end-state for the post. Doing this will make your process easier.

What tools do you use to make the writing process easier?

Photo credit: kk

I find my relationship with blogging is much like my relationship with running. Once a year I go to the running store. I’m greeted by someone with a slender build, clothes made of Thermax or DryFit, and endless enthusiasm. I try to act like I’m one with him by bringing in my old running shoes, recounting the one time I ran a marathon, and selecting what I think I’ll need for the next year. I almost always end up getting more than I need. On this day, I feel like a runner.

Unfortunately spending money and getting the equipment is a lot easier than actually running. Previously I wrote about my experience setting up the Feedback Army blog and I’m proud to say I’ve beat my low bar for consistency keeping this blog up. I’m now starting to feel more like a blogger and less like someone who spends time and money setting up blogs.

In this post I’ll share interesting tips and ideas for keeping consistent. Some may work for you, some may not.

Self Accountability: Jerry Seinfeld’s Writing Secret

Jerry Seinfeld has an amazing productivity secret. He has a big calendar on his wall. Each day he does a writing task he puts a big red X. Eventually the red Xs pile up and he feels pressure to avoid breaking the chain. This keeps Jerry motivated.

I use a similar process of self-accountability to juggle a full-time job, girlfriend, side business, and other projects. Each week I print out an 8.5×11 sheet of paper with a grid on it. The X axis represents the days of the week. The Y axis represents the hours in the day. I mark what time I wake up, what time I go to bed, and I darken the hours during which I “wasted time”. The hours I accomplished something (even errands, meals, and other stuff I have to do), I write what I did. This process keeps me accountable to myself and helps me enjoy my downtime with no guilt.

Note: this is the converse of “scheduling your time”–a practice my spirit has never cared for. If scheduling works for you, do it. If you find scheduling doesn’t work, try this self accountability technique. They’re two sides of the same coin.

Stop in the Middle – Ernest Hemingway’s Secret

What works for me may not work for you. And what doesn’t work for me, may work for you. In the latter group is Hemingway’s Hack. Ernest Hemingway’s secret to writing was to stop in the middle while the getting was good. This prevented him from having to start the next day with the daunting task of starting from scratch. He always resumed something he was in the middle of.

I’ve heard of scientists using a similar approach. They’ll try to call it quits when they’re almost done with something. Me? I like to finish what I start so I can begin the next day or task with a fresh mind. Context switches are expensive (This is a system programmer’s humor. It means I don’t multitask very well.)

So with that said, how do I keep up on blogging?

Get Ahead and Stay Ahead – Raphael’s Secret

Don’t worry, I’m not putting myself in the same class as Jerry Seinfeld and Ernest Hemingway. I need so much writing help that I had to invent a new kind of proofreading tool to keep ahead of my deficiencies.

Self deprecation aside, I know something about getting things done.

The first step to getting ahead is deciding what this means. I set a goal to publish something on the Feedback Army blog once a week. A good goal is reasonable and measurable.

My problem is the mood to write hits me rarely. When it does, I know it’s a gift and I try to harness it as much as possible. During these periods of inspiration I start writing. Start to finish I take my ideas and write out as many posts as I can. I then use WordPress’s post scheduling feature to add the completed posts to the queue. The idea is that as I write I build up a buffer of posts that will automatically publish once per week. I’m about a month ahead now.

This is very motivating as I don’t feel the pressure of falling behind. When the time comes to write I feel like I’m adding to an existing thing not fighting to build something from scratch.

Of course the hardest part is getting started. To break the running analogy from earlier: runner’s can’t go out and do 100 miles the day they get new shoes. Bloggers can. When you get a new blog and you’re excited, use this time to write a bunch of posts and get ahead.

These tips should give you a running start keeping up on your blog. Try them and let me know how it goes.

Photo credit: respres

According to B.L. Ochman, there are over 15,740 folks on Twitter who list themselves as social media experts. I’m now going to write a post about social media and you should listen to me because I’m not one of those 15,740 people.

What’s your impression of social media? When I think of a guru, I immediately imagine someone signing up for every single service to post their link bait to non-existent followers. I also think of folks experimenting with leading technology like “auto-follow on Twitter” who use that to amass a large list of reciprocal followers. Follower who rarely convert to customers and do not represent a meaningful relationships.

Now that I’ve angered 15,740 experts, I will get to the substance of this post. The secret of social media has nothing to do with signing up for a bunch of services and spamming your links. In fact, you probably don’t need to sign up for every new service. It helps if you already have a following–these services give your followers a way to connect. But if you don’t have anyone connecting with you yet, don’t bother. You should spend your efforts another way.

The secret to social media has everything to do with being social. Here are the steps:

1. Find a community.

Yes, an active community. Ideally you want to be part of a community before it explodes in growth. It’s easier to get a name in a small community and reap the benefits of being known later. However making this judgement is a lot like picking stocks.  Your effort is better spent joining a community related to what you’re passionate about. If you started a business, you’re passionate about that business area. Finding and keeping involved in a community related to that passion shouldn’t be hard. If there isn’t one, start one and nurture it.

Feedback Army was built for technology entrepreneurs. Some with small businesses, others with larger businesses. I’m very passionate about this topic. The two communities I found myself involved in are Hacker News and the Four Hour Work Week community (Dear Tim Ferris–I think you’re great). I’m involved with these communities because I’m interested in technology entrepreneurship. In fact, Feedback Army was born out of a discussion on the Hacker News website.

2. Learn the social rules and mores of that community.

If you’re going to take part in a social group you need to know the rules. Constantly breaking the rules will show you’re only out for your self-interest and you won’t benefit “socially” from this. Learning the rules also means you’ll learn what people respond to and what they ignore. Knowing this is the key to getting return value out of the community vs. getting nothing.

On the Four Hour Work Week automation forum people talk about their muse businesses. Writing a post telling people to “check out my business” or “buy my service” does not fly there. What works is asking for feedback. Writing tips about topics people are interested in and telling your story without crossing the line of selling is a good way to get exposure and potential customers from that community.

The Hacker News community is interesting as it’s big and all things considered, pretty easy to get time in the spotlight if you approach us right. I say us, because I feel like I’m a part of that community. A common series of posts are “Ask HN” where an entrepreneur poses a question for others to answer. Announcements go over so so, but writing “Ask HN–Please review my business” almost always receive due attention. News of first launches are also well received by the community. Brutally honest commentary that sparks discussion is always upvoted too. This blog post contributes to the community but I will probably see a few sales as a result too.

3. Get involved and stay involved.

Of course for people to take you seriously, you need to get involved and stay involved. This just means commenting on things and contributing to the over all discussion. We’re each beautiful snow flakes (mine is covered in army green) and have something to offer. As a bonus tip I’ll offer that you should make yourself vulnerable once in a while. I have trouble with this but I find when I’ve done it I’ve seen a strong response. If it’s honest, useful, but makes you slightly uncomfortable to post it, then it’s probably a good candidate.

I can’t measure how getting involved and staying involved has helped me in a conversion rate or $$$ sense. In fact, when I contribute–I’m not out to promote my business or put money in my pocket. By getting involved in a community and staying involved you get another benefit. You’re a part of a community. People who know you, care about you, and  celebrate your successes and console you during your failures. I’m not saying people from Hacker News will hold a vigil if something bad happens to me. I’m just saying at different times I’ve interacted with HN users in real life and when my successes came, these people were there to celebrate with me. And that has made this entrepreneurship process not so lonely.

And now… The Secret

If you find a community that you’re passionate about and learn what people will respond to then you’ve got an opportunity to promote your business. When you find that community: always remember that there is a much bigger benefit to being a part of that community than your just  businesses. So put that aside and just get involved–the rewards will come later. HN has been very good to Feedback Army.

Or as a friend once said, “I’ve found over time that the more I give, the more I end up getting back”. This my friends is the secret to social media.

Photo credit: Annie Mole

Feedback Army continues to grow and do well. It’s my privilege to serve you. Despite this growth I noticed Feedback Army’s search engine traffic is poor. I looked at some options to improve SEO and settled on a strategy of adding quality content to this site. Step 1 of this meant getting a blog up and running with Feedback Army.

For a long time I avoided a blog on this site for fear it would take too much effort to set it up and make it feel like part of the site. Fortunately my fears were wrong and the overall process wasn’t too bad.

Here is what I did to quickly get Feedback Army up and running with WordPress.

My Custom Theme

The first step to a natural integration is a theme that fits the site. WordPress has many beautiful themes to choose from but I wanted the Feedback Army look and feel. Moreover I didn’t want to learn the loop and the WordPress specific theme hooks.

To create the Feedback Army theme I used ThemesPress. ThemesPress is a tool that generates a WordPress theme using the HTML and CSS you give it. They let you develop the theme, preview it, and when you’re happy cash out for $10.

I can’t recommend this product enough. I started playing with it waiting for a flight and when I landed I couldn’t wait to get home and finish it. All told in a few short hours I had a theme that fit into the Feedback Army design and didn’t break WordPress.

If you have a design already and want to bring WordPress into your site, try this product out.

My Plugins

One of the great strengths of WordPress is the variety of plugins available for every need. Here are the ones I use and tips for setting them up.

After the Deadline – Write Better

After the Deadline is a replacement for the spell checker that comes with WordPress. It checks spelling, misused words, grammar and style. I invented After the Deadline and continue to work on it full-time for Automattic. It’s unlike any proofreading tool you’ve used before. Your browser will not alert you when a word is misused and the style checker is similar to other products that cost over $100. This tool is a bargain at the low price of FREE.

To get the most out of After the Deadline, I recommend you visit your user profile settings and enable automatic proofreading. This will force After the Deadline to run before a post is published and prompt you if there is anything to change. I also recommend you have After the Deadline check for everything except “passive voice” and “hidden verbs”. These last two options are valuable if you’re writing a manual or a proposal. For blogging, they’re overkill.

I’m an enthusiastic user of my product and trust me, this product will make you a better writer.

Akismet – Stop SPAM

Akismet is distributed with WordPress. You have to activate it and I highly recommend you do. Once enabled Akismet provides near-100% protection from comment spam. Being free of spam is a good thing.

All in One SEO Pack – Improve SEO

This is one of the most popular plugins on the WordPress repository. Maybe it’s the allure of the name? I downloaded it because it creates a few meta data fields that WordPress does not bother with. Once setup it has been pretty transparent to me.

CommentLuv – Increase Participation

Since my purpose in setting up a blog was to help with SEO, I figured I’d like to return the favor to those who participate on this blog. Enter CommentLuv. CommentLuv scans the site you provide when you comment and generates a link for your latest post. It’s a way of rewarding comments. I think it’s a great concept and it’s non-intrusive.

Google XML Sitemaps – Be Search Engine Friendly

This plugin generates a search engine friendly sitemap to make life easier for robots crawling your site. WordPress has good out of the box SEO but again, this is something that can’t hurt. Once loaded, this plugin has been transparent to me.

iflickr – Makes Posts More Interesting

I love this plugin. From my Add New Post screen, I’m able to search for creative commons licensed photos on Flickr. This is how I’ve added the nifty images to spice up these posts. While this plugin works fine, it is slightly rough around the edges.

Twitter Updater – Publicize

Good social media integration is as important as good SEO. I’m not too active on Twitter but I’m a fan of having an auto-publicize feature for blog posts. This plugin works for what it does.

What I’d like to see in the future is a publicize plugin that lets me publicize my posts to sites based on what I tag my posts with. For example, most of my business musings post are right for Hacker News. Moreover, I’d like to see such a plugin support an API that lets users contribute code to make submitting to smaller or more niche sites possible. Good promotion isn’t hitting just the big services, it’s reaching the niche ones.  Note–getting this right is a great opportunity for someone.

WordPress.com Stats – Measure Growth

The WordPress.com Stats plugin is easy to use and provides statistics for measuring your blog’s growth. I use it because I’m used to it from WordPress.com. When it works, it works fine. When it doesn’t work, it’s a mystery. For the first few weeks of this blog’s existence I saw zero users according to the WordPress.com stats. If you think it’s demotivating writing for 40 or 50 users, try writing for zero.

I later discovered that the theme created by ThemesPress lacks the <?php wp_footer(); ?> line in their footer.php file. This line is necessary for the stats plugin to work. If you’re using a ThemesPress theme (I still recommend it), you’ll need to update footer.php to get the WordPress.com Stats plugin to work. Just insert the code right before the </body> tag at the end of the document. Once I fixed this, WordPress.com Stats started reporting results and I saw that I do have readers.

Closing Thoughts

As I said earlier, getting up and running with WordPress was pretty easy. ThemesPress made it possible for me to get the look and feel I desired. The plugins help make writing go from a chore to fun. Happy blogging.

Photo Credit: Waxy.org

Earlier I wrote about my impressions using Mechanical Turk to run the Feedback Army Usability Testing Service. Here I write about my favorite part, dealing with the workers and taking care of them.

What to Pay on Mechanical Turk

What you pay will affect the turn around time of responses and to some extent their quality. Some people won’t even bother if the pay is too low. I found people will review websites for $0.25-$0.30 but I changed the pay to $0.45-$0.50 and stuck with it.

The best way to find this number is to create test tasks using Amazon’s HIT Builder. Try a few numbers and see which ones yield turn-around that you’re comfortable with.

I recommend when creating a new task or service that you include a question in each new HIT asking the workers for feedback to improve the HIT interface and what confused them. The workers will follow your instructions literally and you will want to try a few iterations before going live with them. Getting this feedback can greatly speed up the process.

Mechanical Turk Response Turn-Around Time

With the right pay, responses come quickly. With Feedback Army’s current pay level, ten response jobs are usually completed in one to three hours.

Feedback Army requires unique reviews and this greatly affects turn around time.

Mechanical Turk Response Quality

The response quality varies. Some Feedback Army responses are several paragraphs. I have seen users write a page or more. Usually the workers are out to answer the questions as quickly as possible. Spelling and grammar is sometimes poor.

Just like E-Bay has a seller rating–Turkers have a task approval rating. You can set a minimum approval rating for your tasks. 95% or above is near impossible for workers to keep. I tried 85% for a while but eventually lowered it to 70% to open the field to more workers. I have not noticed a change in quality from doing this.

No matter what–you or your customers will need to moderate the responses. Sometimes workers accidentally hit submit and you end up with a blank or incomplete response. I’ve put the power to reject poor or unacceptable responses into the hands of my customers and we have both been happy with the results. Less work for me and more control for them.

Open ended questions work best and draw out longer responses. Also the worker’s interest in the site affects the response. I saw a “how to get women into bed now” site posted to this service and the responses were passionate, useful, and quite entertaining (most Turkers are women).

Know the Workers

As I said in the last paragraph, most Turkers are women. It helps to understand the demographics of the workers. I found an excellent study awhile ago and according to it must Turkers are in their 20s, are from the United States, and have college educations. Mechanical Turk does not have a direct mechanism for restricting tasks to specific demographics.

Several Turkers interact on a bulletin board called Turker Nation. I recommend that you introduce yourself on the Turker Nation board. They will answer questions about what works and doesn’t work for them.

Also, if you are unfair–they’ll add you to the hall of shame and you don’t want that.

Final Thoughts

Pay affects the quality and speed of responses. Remember that Mechanical Turk is an open market place and workers can choose to accept or reject your tasks. Pay fairly, listen to the feedback from your workers, and plan for moderating the responses. If you do these things you’ll be impressed by what you get.

I’ve used Mechanical Turk for several projects including Feedback Army. I’m often asked for my impressions of the service and what worked for me. Here are my thoughts on the API and support behind Mechanical Turk.

Software Integration

I use Amazon’s Mechanical Turk Java SDK to deal with the service. I’m able to do what I need with the API but it is confusing and not well documented. Several times I’ve had to dig in their source code to find certain options.

I’ve also dealt with more XML verification issues than I care to think about. I’ve found the ExternalQuestion mechanism is the way to go. Through this you can direct your workers to an external HTML page that you design and control. This is simple to do and helps avoid these dreaded errors I speak of.

The SDK requires an insane amount of external libraries. Fortunately these libraries are included. A more complex application that uses different versions of these libraries may have issues. If this is you, I recommend creating a light-weight service that runs stand-alone and acts as an interface to the Mechanical Turk service.

Support

Support from Amazon is non-existent. There is a forum for those working with Mechanical Turk but reading it shows cries for help with few answers. One time, a power outage knocked out my server and Feedback Army stopped working. I was at a loss because my code had not changed. I kept receiving an “InternalServiceException – Request has expired.” error. I later learned that my server’s clock was out of sync and correcting this fixed the error.

Overall Impression

In the year I’ve run Feedback Army, Mechanical Turk has proven stable for me from a software point-of-view. The SDK is difficult to work with and the support is non-existent. However once you get your solution developed, you’re good to go. It will work fine.