We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments

Micro Seminars
Image of the campus woudestein, showcasing the flags of the School's and Institutes

Abel Brodeur

Friday 23 Sep 2022, 15:30 - 16:45


Add to calendar
Image of the campus woudestein, showcasing the flags of the School's and Institutes

Amazon Mechanical Turk is a very widely-used tool in business and economics research, but how trustworthy are results from well-published studies that use it?

Analysing the universe of hypotheses tested on the platform and published in leading journals between 2010 and 2020 we find evidence of widespread p-hacking, publication bias and over-reliance on results from plausibly under-powered studies. Even ignoring questions arising from the characteristics and behaviors of study recruits, the conduct of the research community itself erode substantially the credibility of these studies' conclusions.

The extent of the problems vary across the business, economics, management and marketing research fields (with marketing especially afflicted). The problems are not getting better over time and are much more prevalent than in a comparison set of non-online experiments. We explore correlates of increased credibility.

How to attend the seminar

If you wish to attend the seminar or would like to have a bilateral with the speaker, please contact the seminar organizers Anne Boring (boring@ese.eur.nl) or Robert Dur (dur@ese.eur.nl).

See also

Female labour supply, taxation and intergenerational spillovers

Mareen Bastiaans (Erasmus School of Economics)
Woman sitting at a desk browsing through a laid out newspaper

Setting the course for the future: How do 15-year-olds make education choices?

Sonja Settele (University of Copenhagen)
Image - Schoolkids

Identifying the Effect of Election Closeness on Voter Turnout: Evidence from Swiss Referenda

Patricia Funk (Università della Svizzera italiana)
Image - Voting

6th Erasmus Corporate Governance Conference

A one-day workshop which brings together scholars of Executive Compensation and other fields of Corporate Governance

Compare @count study programme

  • @title

    • Duration: @duration
Compare study programmes