Prediction Markets Inspire Tool for Gauging Study Reproducibility

Penn State researchers are exploring the possibility of using artificial intelligence (AI) to “bet” on hypothetical replication studies—in other words, estimate confidence that a published, scholarly work could be reproduced. The team hopes such methods could help address the growing lack of reproducibility in research and prevent public mistrust of science.   

Replication of experiments and studies, a critical step in the scientific process, helps provide confidence in the results and indicates whether they can generalize across contexts, says Sarah Rajtmajer, assistant professor in information sciences and technology, Penn State. However, as experiments have become more complex, costly and time consuming, scientists increasingly lack the resources for robust replication efforts—what is often referred to as the "replication crisis."

“As scientists, we want to do work, and we want to know that our work is good,” said Rajtmajer. “Our approach to help address the replication crisis is to use AI to help predict whether a finding would replicate if repeated and why.”

In the team’s approach, bots were trained to recognize key features in academic research papers, such as the authors and institutions, statistics, and linguistic cues; downstream mentions; and similar studies in the literature. Using this information, the bots then make assessments regarding the confidence that the study is robust enough to replicate in future studies.

The AI-powered bot then bids on its level of confidence, just like a human betting on the outcomes of a sporting event. In this way, the bots are similar to crowdsourced prediction markets that forecast real world events, from sports to elections.

“What inspired us was the success of prediction markets in precisely this task—that is, when you place researchers in a market and give them some cash to bet on outcomes of replications, they’re pretty good at it,” says Rajtmajer, who is also a research associate at the Rock Ethics Institute and an Institute for Computational and Data Sciences associate. “But human-run prediction markets are expensive and slow. And ideally, you should run replications in parallel to the market so there is some ground truth on which researchers are betting. It just doesn’t scale.”

According to the researchers, who presented their results at a recent meeting for the Association for the Advancement of Artificial Intelligence, the system provided confidence scores for about 68 of 192 papers—or about 35%—of the papers that were eventually reproduced. On that set of papers, the accuracy was approximately 90%.

The team suggests a combination of humans and bots to deliver a system that would have high accuracy but still be scalable. “Maybe we can train the bots in the presence of human traders every so often, and then deploy them offline when we need a quick result, or when we need replication efforts at scale,” said Rajtmajer. “Moreover, we can create bot markets that also leverage that intangible human wisdom. That is something we are working on right now.”