We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell us about p-Hacking and Publication Bias in Online Experiments: a new GLO Discussion Paper by GLO Fellow Abel Brodeur and colleagues.

Discussion Paper, News, Post, Post-22

A new GLO Discussion Paper finds evidence of widespread p-hacking, publication bias and over-reliance on results from plausibly under-powered studies. 

GLO Discussion Paper No. 1157, 2022

We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell us about p-Hacking and Publication Bias in Online Experiments – Download PDF
by Brodeur, Abel & Cook, Nikolai & Heyes, Anthony

GLO Fellow Abel Brodeur

Abel Brodeur

Author Abstract: Amazon’s Mechanical Turk is a very widely-used tool in business and economics research, but how trustworthy are results from well-published studies that use it? Analyzing the universe of hypotheses tested on the platform and published in leading journals between 2010 and 2020 we find evidence of widespread p-hacking, publication bias and over-reliance on results from plausibly under-powered studies. Even ignoring questions arising from the characteristics and behaviors of study recruits, the conduct of the research community itself erodes substantially the credibility of these studies’ conclusions. The extent of the problems vary across the business, economics, management and marketing research fields (with marketing especially afflicted). The problems are not getting better over time and are much more prevalent than in a comparison set of non-online experiments. We explore correlates of increased credibility.

Featured image: Mika-Baumeister-on-Unsplash

JUST PUBLISHED
Vol. 35, Issue 4, October 2022: Journal of Population Economics (JOPE): 15 articles
https://link.springer.com/journal/148/volumes-and-issues/35-4
Just released: CiteScore of JOPE moves up from 3.9 (2020) to 6.5 (2021)! LINK
Similar, its Impact Factor is now 4.7 (2021) after 2.8 (2020)! LINK

GLO Discussion Papers are research and policy papers of the GLO Network which are widely circulated to encourage discussion. Provided in cooperation with EconStor, a service of the ZBW – Leibniz Information Centre for Economics, GLO Discussion Papers are among others listed in RePEc (see IDEAS,  EconPapers). Complete list of all GLO DPs – downloadable for free.

The Global Labor Organization (GLO) is an independent, non-partisan and non-governmental organization that functions as an international network and virtual platform to stimulate global research, debate and collaboration.

Ends;