Study Blames Database Errors for Reported Decline in Scientific Disruption

A widely cited Nature article from 2023 claimed that scientific innovation has significantly declined in recent decades, based on citation analysis of large sets of metadata on scientific papers and patents. Now researchers from Vrije Universiteit Brussel (VUB) and KU Leuven say incomplete reference data account for most of that alleged decline, a conclusion that has already drawn pushback from other researchers in the field. 

Doctoral researcher Vincent Holst, the new study’s first author, said he tried to reproduce the original study using open data shortly after its publication. “I discovered a large peak of papers receiving the maximal disruption score. This was strange, since the histograms published in Nature showed no such peak.” The Belgian team traced the discrepancy to a bug in the plotting library the original authors used, which omitted the largest data points from the published histogram without warning, even though that data remained part of the underlying analysis.

“If the papers with maximum scores were genuinely groundbreaking, the conclusion would still hold,” said Professor Vincent Ginis, who led the study. “But we found that they mainly correspond to dataset artefacts. When the references from a publication are not listed in the database, the method that the authors used wrongly labels that publication as maximally disruptive.” Because older database records are more likely to have incomplete reference data, and record quality improved over time, the pattern created the appearance of declining disruption, when it largely reflected improving metadata quality.

“When we filter out the database errors, the decline largely disappears,” Holst said. “This holds for all of the scientific databases that we examined. In the study’s largest dataset, the artefacts account for 93% of the reported decline between 1945 and 2010.”

The original claim appeared in more than 300 news stories worldwide. The Belgian team submitted their critique to Nature in October 2023, and the journal took 32 months to accept it. During that time, Nature published an editorial and a News Feature drawing on the original conclusion, the paper was cited more than 1,000 additional times, and it was used in at least 16 policy documents. “Whereas the software bug was fixed by the open-source community in months, it took years for the scientific correction to get published,” Ginis said. “And all this time, the journal’s policies prohibited us from speaking to the press while our critique was under submission.”

Notably, not everyone in the field draws the same conclusion from the corrected data. Other researchers have suggested restricting the analysis to papers with at least ten references and citations; when the Belgian team applied that guideline, they found a slight, statistically significant increase in disruption. Even so, that isn’t their takeaway. “Based on the used measure, we simply cannot make conclusive statements about whether science is getting more or less disruptive,” Ginis said. “It has been shown, however, that the methodology and conclusions from 2023 do not provide a sound basis for our science policy.”