Markiter is looking to expand into more in-depth market research. We've been getting a lot of feedback and a lot of requests for more detailed surveys and analytics. That's exactly what we're going to do. We've chosen to start with Consumer Electronics specifically because we believe that the Mechanical Turk audience fits very well with this space, and it will allow us to provide our customers with a high level of both quantity and quality of responses. So if you've got a new product coming out, or are interested in learning about the current trends in Consumer Electronics, stop by and subscribe because we've got 100,000 people ready and waiting to share their opinions with you.

http://www.markiter.com/consumer_electronics_market_research/

A few aesthetic principles that you can consider when working on a design.  Inspirational.

via http://www.presentationzen.com/presentationzen/2009/09/exposing-ourselves-to-traditional-japanese-aesthetic-ideas-notions-that-may-seem-quite-foreign-to-most-of-us-is-a-goo.html

Markiter completed our site redesign and improved our messaging.

Check it out at http://www.markiter.com

Let us know what you think!

Markiter believes strongly in the value of A/B testing, and using real data to make decisions.  However - the issue can be much more complicated than at first glance.  Many people will put up two designs and test with Google Weboptimizer and go with the better converting option.  The problem here is that you are only discovering the local maximum for conversions - and have no way of knowing how well that compares against the global maximum.

A quick analogy will help make things more clear.  If you want to climb to the top of a mountain in a snowstorm, you will keep heading up until you are at a peak.  However, you could just be still on the side of the mountain on a small peak and nowhere near the top.

A great example of this in action is illustrated at:

http://www.blastam.com/broadcast/2008/12/google-website-optimizer-test-we-didnt.html  

They ended up taking the losing option from Weboptimizer and tweaking it to get to a far better result.  This case study illustrates the importance of looking beyond the local maximum.

Markiter aims to help you solve this problem.  By providing simple A/B testing with candid feedback, you can perform a study like this on any media.  It could be website mockups for similar results to the example - or it could be posters or demo videos.   Think of the possiblities for using real data to make more educated decisions about media!

Markiter currently uses Amazon's Mechanical Turk as the primary method for engaging voters for testing campaigns. Mechanical Turk is a web service created by Amazon.com based on the belief that there are certain things that humans can do better and faster than a computer. Image recognition is a perfect example; a human can easily tell what products, or items are in a picture much more accurately than a computer can. This is why Amazon's official IPhone application currently uses Mechanical Turk for one of it's key features. An IPhone user can take a picture of an item, send it to Amazon, and Mechanical Turk workers will identify the item, and return the product detail page on Amazon that best resembles the item. Wired magazine has some more information about it here.

There have been numerous studies done to validate our beliefs that Mechanical Turk is the perfect venue to leverage the wisdom of crowds for market research purposes. The main concerns we had before selecting Mechanical Turk on behalf of our customers were the distribution of voters, and the quality and accuracy of voters. Let's take a look at the independent research we were able to uncover for each of these.

Voter Distribution:
A study was recently done to study the distribution and demographics of Mechanical Turk voters. Here is a summary of their findings. You can read the full study and see more graphical breakdowns here. 

• 75% of voters are from the States, and not a third world country
• Almost 50/50 distribution between male/female participants
• Large majority of voters were between 21-40.
• Over 50% of Mechanical Turk voters have a Bachelor's degree
• The income distribution of voters mirrors the income distribution of the US

Take a look at the some real Mechanical Turk workers, and the experiment to find out more about them:

Enrique Andres Ferraro from the University of Liverpool did an extensive study into the characteristics of Mechanical Turk, and concluded the following: 

"We conducted an analysis of worker’s demographic characteristics with a sample of 1428 workers, which we contrasted and compared with the US Census of 2000. Our study reveals that The Amazon Mechanical Turk attracts workers from all segments of a population, largely in proportions matching the population from which they were drawn, and that the system represents a portal through which market research can be conducted easily and cost-effectively without incurring some of the sacrifices in validity inherent in captive panels or indirect online research."

You can see his full dissertation here

Voter Quality:
One fundamental question about Turk that raises concern is this: "How do you control the quality of the responses? 
Aaron Koblin and Takashi Kawashima have answered this question. Exhibit A: Their Ten Thousand Cents project 

They broke a $100 bill into 100,000 segments and asked Mechanical Turk workers to draw their piece of the bill. With no validation on the submitted picture, its clear that a large majority of responders followed through with the requested task. While it is true that a few people that did not do their part, this experiment goes to show the normalization power of crowds. Despite the few exceptions, the sum of the overall community will point you down the right path. Applying this community to market research provides a powerful value proposition. 
This is no different from traditional market research. There are bound to be a few people that don't take the time to answer questions correctly, or randomly fill out surveys, but market research is still extremely valuable because removing the outliers gives you a clear picture on the general populations views and opinions. 

Finally, we put in several controls to minimize the risk of poor responses. We restrict participants to an approval rating of over 95% on Mechanical Turk. That is, participants in your campaign have a strong reputation for completing the task. In addition, we also restrict participants to only vote once, so they cannot skew the results.

In any business, market research is essential to determine what marketing materials, products and general media presented to your audience is successful. Direct snail mailers would split test flyers they send out, even then the costs were outweighed by the benefits. 

The online business environment has made this type of testing substantially cheaper and easier. A/B testing has some great benefits, since it measures actual behavior. You can determine what resonates with the user in the real world, where as traditional surveys try indirectly asking respondents questions in order to try to make some hypothesis about what they would like. 

However, many online A/B tests have a few limitations. There needs to be a clear and measurable action such as a sale to determine success. Many systems like Google WebOptimizer tend to be primarily used by e-commerce sites to test sales pages. However, this is only a piece of the puzzle for most online sites. There are many other goals such as user retention, enjoyment and approval. 

Additionally traditional online A/B testing systems only work for fully implemented and running sites. In order to test a few variations, you must completely implement these different sites as well. It is difficult to expose customers to different experimental designs during the creation process. 

Markiter strives to achieve all the benefits of A/B split testing, at the same time reducing the limitations. By providing the user with a direct opportunity to compare two pieces of media Markiter is not limited by tracking through sales. Additionally, you do not need to have a fully implemented website in order to leverage A/B split testing through Markiter. You could compare anything at any stage of the design process - even a few pictures of sketches you created. 

Markiter has decided to move our blog.

We are now hosting it with our friends at Posterous.

Come check us out at www.markiter.com - we will move articles here over the next few days.

Founders of Markiter

Justin Lai and Troy Simpson