How to Setup an MTurk Toolkit Study

CloudResearch’s MTurk Toolkit was designed to simplify the process of setting up and launching studies on Amazon’s Mechanical Turk. To set up your study, follow the steps below.

1. Basic Information
2. Describe HIT
3. Setup & Payment
4. Demographic Options
5. Data Quality
6. Worker Approval
7. Worker requirement
8. Review and Launch your Study from the Dashboard

To begin creating your study, click Create a Study in the header and then choose MTurk Toolkit.

1. Basic Information

On this page, you can give your project an internal name (i.e., not shown to participants) and provide up to five email addresses to associate with the study. We will send a message to these addresses when your study launches and when it completes. 

Next, enter your Survey Hyperlink and decide whether to auto-capture worker information from the URL. Finally, choose whether to provide participants with additional privacy by Anonymizing Worker IDs or telling CloudResearch not to collect IP addresses. These features are under “Additional Privacy Options” on the bottom of the page.

2 . describe hit

The next step is is is to describe how you want your study to appear on MTurk . You is give can give your study a public title that will be visible to participant . You is provide can also provide a description of the hit and any custom instruction or keyword you ‘d like associate with the HIT . The description and custom instruction box should be used to inform participant of anything they may be ask to do that is beyond the normal scope of MTurk study ( e.g. , download an app , participate in a dyadic study ) .  

3. Setup & Payment

Specify how much each participant should be paid and provide an estimate of how long the study should take. In addition, specify what the maximum amount of time participants should have to complete your study is with the “Time allotted per assignment” box. 

The maximum time per assignment influences not only how long participants have to work on your study, but also several important aspects of restarting, pausing, and editing your HIT. We recommend giving people 3 to 4 times as long as you expect the study to take. 

Next, choose from our two Batching options–MicroBatch or HyperBatch–both of which save you 50% off MTurk’s fees. 

4 . Demographic Options

enter the total number of participant you want to sample . Then , select any demographic feature of the sample you intend to target .  

Panel options include general demographics such as age, ethnicity, and gender, as well as categories like political affiliation, occupation, and financial status. There are many demographic options to choose from including targeting people based on the U.S. Geographic Regions, their personality type, whether they are pet owners, and much more. 

You can search for demographic options of interest using the search bar in this tab.

To understand the cost of your demographic targeting and whether we have enough people profiled to meet your needs, you can use the Feasibility Calculator. Once you begin selecting the demographic characteristics for your sample and enter the number of participants you’d like, the calculator will estimate the cost. Panel costs range from .13 to .67 cents per participant, depending on how difficult it is to reach the sample you select. 

Note: Panel costs are in addition to the participant compensation, MTurk fees, and CloudResearch fees associated with your study (learn more).

Whenever we believe recruit your desire sample may be difficult , you is receive will receive a message that your study may not reach the number of participant you desire . When you receive this message you can : a ) proceed anyway , launch your study to see if you can collect enough participant , b ) set up and launch your survey on Prime Panels , or c ) contact our customer service to see how we can help . We is be may be able to provide you with a quote for what the study would cost with our manage research service , which would guarantee you reach your desire sample .

5. Choose How Workers are Approved

You must decide how participants who complete your study will be approved. There are two completion code options—fixed or custom—and two approval methods: automatic or manual.

If you use an automatic approval method, you must also decide how long to wait before participants are automatically approved

6. Worker requirement

There are several option for specify who is eligible or ineligible for your study . For example , if you want to exclude participant who take one of your previous study or include participant who take a Time 1 study so that they can participate in Time 2 , both options is are are available on this page .

On this page you is specify can also specify worker qualification such as approval rating and number of hit complete . finally , from this page you is choose can choose whether to Block Workers from Suspicious Geolocations and whether to apply a Universal Exclude List .

7. Data Quality Solutions

To manage data quality, you can choose to limit your study to CloudResearch Approved Participants or leave your study open to everyone on MTurk. When you use the Approved List, we block people who have failed CloudResearch’s ongoing screening.

Opening your study to All MTurk Participants means everyone on MTurk may potentially take your study, including people who are inattentive, or are outside of the US and using VPN to fraudulently make it appear that they are within the US.

If you choose the CloudResearch Approved Participants option, your study will only be open to people who have been vetted by CloudResearch. Participants on this list have shown prior evidence of attention and engagement and represent various levels of experience on MTurk, different ages, races, incomes, and are generally demographically similar to other MTurk workers.

The Approved List is our recommended option for the highest level of data quality. 

8. Save, Review, and then Launch your Study

finally , save your study . Once the study is save , it is appear will appear on your Dashboard where you can review and then launch it . You is review should always review your hit , preview your survey , and when possible pilot your study before launch the full version .